134 lines
7.1 KiB
Text
134 lines
7.1 KiB
Text
|
Intel Graphics Device (IGD) assignment with vfio-pci
|
||
|
|
====================================================
|
||
|
|
|
||
|
|
IGD has two different modes for assignment using vfio-pci:
|
||
|
|
|
||
|
|
1) Universal Pass-Through (UPT) mode:
|
||
|
|
|
||
|
|
In this mode the IGD device is added as a *secondary* (ie. non-primary)
|
||
|
|
graphics device in combination with an emulated primary graphics device.
|
||
|
|
This mode *requires* guest driver support to remove the external
|
||
|
|
dependencies generally associated with IGD (see below). Those guest
|
||
|
|
drivers only support this mode for Broadwell and newer IGD, according to
|
||
|
|
Intel. Additionally, this mode by default, and as officially supported
|
||
|
|
by Intel, does not support direct video output. The intention is to use
|
||
|
|
this mode either to provide hardware acceleration to the emulated graphics
|
||
|
|
or to use this mode in combination with guest-based remote access software,
|
||
|
|
for example VNC (see below for optional output support). This mode
|
||
|
|
theoretically has no device specific handling dependencies on vfio-pci or
|
||
|
|
the VM firmware.
|
||
|
|
|
||
|
|
2) "Legacy" mode:
|
||
|
|
|
||
|
|
In this mode the IGD device is intended to be the primary and exclusive
|
||
|
|
graphics device in the VM[1], as such QEMU does not facilitate any sort
|
||
|
|
of remote graphics to the VM in this mode. A connected physical monitor
|
||
|
|
is the intended output device for IGD. This mode includes several
|
||
|
|
requirements and restrictions:
|
||
|
|
|
||
|
|
* IGD must be given address 02.0 on the PCI root bus in the VM
|
||
|
|
* The host kernel must support vfio extensions for IGD (v4.6)
|
||
|
|
* vfio VGA support very likely needs to be enabled in the host kernel
|
||
|
|
* The VM firmware must support specific fw_cfg enablers for IGD
|
||
|
|
* The VM machine type must support a PCI host bridge at 00.0 (standard)
|
||
|
|
* The VM machine type must provide or allow to be created a special
|
||
|
|
ISA/LPC bridge device (vfio-pci-igd-lpc-bridge) on the root bus at
|
||
|
|
PCI address 1f.0.
|
||
|
|
* The IGD device must have a VGA ROM, either provided via the romfile
|
||
|
|
option or loaded automatically through vfio (standard). rombar=0
|
||
|
|
will disable legacy mode support.
|
||
|
|
* Hotplug of the IGD device is not supported.
|
||
|
|
* The IGD device must be a SandyBridge or newer model device.
|
||
|
|
|
||
|
|
For either mode, depending on the host kernel, the i915 driver in the host
|
||
|
|
may generate faults and errors upon re-binding to an IGD device after it
|
||
|
|
has been assigned to a VM. It's therefore generally recommended to prevent
|
||
|
|
such driver binding unless the host driver is known to work well for this.
|
||
|
|
There are numerous ways to do this, i915 can be blacklisted on the host,
|
||
|
|
the driver_override option can be used to ensure that only vfio-pci can bind
|
||
|
|
to the device on the host[2], virsh nodedev-detach can be used to bind the
|
||
|
|
device to vfio drivers and then managed='no' set in the VM xml to prevent
|
||
|
|
re-binding to i915, etc. Also note that IGD is also typically the primary
|
||
|
|
graphics in the host and special options may be required beyond simply
|
||
|
|
blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a
|
||
|
|
PCI class device. Lower level drivers exist that may still claim the device.
|
||
|
|
It may therefore be necessary to use kernel boot options video=vesafb:off or
|
||
|
|
video=efifb:off (depending on host BIOS/UEFI) or these can be combined to
|
||
|
|
a catch-all, video=vesafb:off,efifb:off. Error messages such as:
|
||
|
|
|
||
|
|
Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow
|
||
|
|
|
||
|
|
are a good indicator that such a problem exists. The host files /proc/iomem
|
||
|
|
and /proc/ioports are often useful for identifying drivers consuming ranges
|
||
|
|
of the device to cause such conflicts.
|
||
|
|
|
||
|
|
Additionally, IGD device are known to generate small numbers of DMAR faults
|
||
|
|
when initially assigned. It is believed that this is simply the IGD attempting
|
||
|
|
to access the reserved GTT space after reset, which it no longer has access to
|
||
|
|
when accessed from userspace. So long as the DMAR faults are small in number
|
||
|
|
and most importantly, not ongoing, these are not an indication of an error.
|
||
|
|
|
||
|
|
Additionally++, analog VGA output (as opposed to digital outputs like HDMI,
|
||
|
|
DVI, or DisplayPort) may be unsupported in some use cases. In the author's
|
||
|
|
experience, even DP to VGA adapters can be troublesome while adapters between
|
||
|
|
digital formats work well.
|
||
|
|
|
||
|
|
Usage
|
||
|
|
=====
|
||
|
|
The intention is for IGD assignment to be transparent for users and thus for
|
||
|
|
management tools like libvirt. To make use of legacy mode, simply remove all
|
||
|
|
other graphics options and use "-nographic" and either "-vga none" or
|
||
|
|
"-nodefaults", along with adding the device using vfio-pci:
|
||
|
|
|
||
|
|
-device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2
|
||
|
|
|
||
|
|
For UPT mode, retain the default emulated graphics and simply add the vfio-pci
|
||
|
|
device making use of any other bus address other than 02.0. libvirt will
|
||
|
|
default to assigning the device a UPT compatible address while legacy mode
|
||
|
|
users will need to manually edit the XML if using a tool like virt-manager
|
||
|
|
where the VM device address is not expressly specified.
|
||
|
|
|
||
|
|
An experimental vfio-pci option also exists to enable OpRegion, and thus
|
||
|
|
external monitor support, for UPT mode. This can be enabled by adding
|
||
|
|
"x-igd-opregion=on" to the vfio-pci device options for the IGD device. As
|
||
|
|
with legacy mode, this requires the host to support features introduced in
|
||
|
|
the v4.6 kernel. If Intel chooses to embrace this support, the option may
|
||
|
|
be made non-experimental in the future, opening it to libvirt support.
|
||
|
|
|
||
|
|
Developer ABI
|
||
|
|
=============
|
||
|
|
Legacy mode IGD support imposes two fw_cfg requirements on the VM firmware:
|
||
|
|
|
||
|
|
1) "etc/igd-opregion"
|
||
|
|
|
||
|
|
This fw_cfg file exposes the OpRegion for the IGD device. A reserved
|
||
|
|
region should be created below 4GB (recommended 4KB alignment), sized
|
||
|
|
sufficient for the fw_cfg file size, and the content of this file copied
|
||
|
|
to it. The dword based address of this reserved memory region must also
|
||
|
|
be written to the ASLS register at offset 0xFC on the IGD device. It is
|
||
|
|
recommended that firmware should make use of this fw_cfg entry for any
|
||
|
|
PCI class VGA device with Intel vendor ID. Multiple of such devices
|
||
|
|
within a VM is undefined.
|
||
|
|
|
||
|
|
2) "etc/igd-bdsm-size"
|
||
|
|
|
||
|
|
This fw_cfg file contains an 8-byte, little endian integer indicating
|
||
|
|
the size of the reserved memory region required for IGD stolen memory.
|
||
|
|
Firmware must allocate a reserved memory below 4GB with required 1MB
|
||
|
|
alignment equal to this size. Additionally the base address of this
|
||
|
|
reserved region must be written to the dword BDSM register in PCI config
|
||
|
|
space of the IGD device at offset 0x5C. As this support is related to
|
||
|
|
running the IGD ROM, which has other dependencies on the device appearing
|
||
|
|
at guest address 00:02.0, it's expected that this fw_cfg file is only
|
||
|
|
relevant to a single PCI class VGA device with Intel vendor ID, appearing
|
||
|
|
at PCI bus address 00:02.0.
|
||
|
|
|
||
|
|
Footnotes
|
||
|
|
=========
|
||
|
|
[1] Nothing precludes adding additional emulated or assigned graphics devices
|
||
|
|
as non-primary, other than the combination typically not working. I only
|
||
|
|
intend to set user expectations, others are welcome to find working
|
||
|
|
combinations or fix whatever issues prevent this from working in the common
|
||
|
|
case.
|
||
|
|
[2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override
|