319 lines
15 KiB
Text
319 lines
15 KiB
Text
|
PCI EXPRESS GUIDELINES
|
||
|
|
======================
|
||
|
|
|
||
|
|
1. Introduction
|
||
|
|
================
|
||
|
|
The doc proposes best practices on how to use PCI Express (PCIe) / PCI
|
||
|
|
devices in PCI Express based machines and explains the reasoning behind
|
||
|
|
them.
|
||
|
|
|
||
|
|
Note that the PCIe features are available only when using the 'q35'
|
||
|
|
machine type on x86 architecture and the 'virt' machine type on AArch64.
|
||
|
|
Other machine types do not use PCIe at this time.
|
||
|
|
|
||
|
|
The following presentations accompany this document:
|
||
|
|
(1) Q35 overview.
|
||
|
|
https://wiki.qemu.org/images/4/4e/Q35.pdf
|
||
|
|
(2) A comparison between PCI and PCI Express technologies.
|
||
|
|
https://wiki.qemu.org/images/f/f6/PCIvsPCIe.pdf
|
||
|
|
|
||
|
|
Note: The usage examples are not intended to replace the full
|
||
|
|
documentation, please use QEMU help to retrieve all options.
|
||
|
|
|
||
|
|
2. Device placement strategy
|
||
|
|
============================
|
||
|
|
QEMU does not have a clear socket-device matching mechanism
|
||
|
|
and allows any PCI/PCI Express device to be plugged into any
|
||
|
|
PCI/PCI Express slot.
|
||
|
|
Plugging a PCI device into a PCI Express slot might not always work and
|
||
|
|
is weird anyway since it cannot be done for "bare metal".
|
||
|
|
Plugging a PCI Express device into a PCI slot will hide the Extended
|
||
|
|
Configuration Space thus is also not recommended.
|
||
|
|
|
||
|
|
The recommendation is to separate the PCI Express and PCI hierarchies.
|
||
|
|
PCI Express devices should be plugged only into PCI Express Root Ports and
|
||
|
|
PCI Express Downstream ports.
|
||
|
|
|
||
|
|
2.1 Root Bus (pcie.0)
|
||
|
|
=====================
|
||
|
|
Place only the following kinds of devices directly on the Root Complex:
|
||
|
|
(1) PCI Devices (e.g. network card, graphics card, IDE controller),
|
||
|
|
not controllers. Place only legacy PCI devices on
|
||
|
|
the Root Complex. These will be considered Integrated Endpoints.
|
||
|
|
Note: Integrated Endpoints are not hot-pluggable.
|
||
|
|
|
||
|
|
Although the PCI Express spec does not forbid PCI Express devices as
|
||
|
|
Integrated Endpoints, existing hardware mostly integrates legacy PCI
|
||
|
|
devices with the Root Complex. Guest OSes are suspected to behave
|
||
|
|
strangely when PCI Express devices are integrated
|
||
|
|
with the Root Complex.
|
||
|
|
|
||
|
|
(2) PCI Express Root Ports (ioh3420), for starting exclusively PCI Express
|
||
|
|
hierarchies.
|
||
|
|
|
||
|
|
(3) PCI Express to PCI Bridge (pcie-pci-bridge), for starting legacy PCI
|
||
|
|
hierarchies.
|
||
|
|
|
||
|
|
(4) Extra Root Complexes (pxb-pcie), if multiple PCI Express Root Buses
|
||
|
|
are needed.
|
||
|
|
|
||
|
|
pcie.0 bus
|
||
|
|
----------------------------------------------------------------------------
|
||
|
|
| | | |
|
||
|
|
----------- ------------------ ------------------- --------------
|
||
|
|
| PCI Dev | | PCIe Root Port | | PCIe-PCI Bridge | | pxb-pcie |
|
||
|
|
----------- ------------------ ------------------- --------------
|
||
|
|
|
||
|
|
2.1.1 To plug a device into pcie.0 as a Root Complex Integrated Endpoint use:
|
||
|
|
-device <dev>[,bus=pcie.0]
|
||
|
|
2.1.2 To expose a new PCI Express Root Bus use:
|
||
|
|
-device pxb-pcie,id=pcie.1,bus_nr=x[,numa_node=y][,addr=z]
|
||
|
|
PCI Express Root Ports and PCI Express to PCI bridges can be
|
||
|
|
connected to the pcie.1 bus:
|
||
|
|
-device ioh3420,id=root_port1[,bus=pcie.1][,chassis=x][,slot=y][,addr=z] \
|
||
|
|
-device pcie-pci-bridge,id=pcie_pci_bridge1,bus=pcie.1
|
||
|
|
|
||
|
|
|
||
|
|
2.2 PCI Express only hierarchy
|
||
|
|
==============================
|
||
|
|
Always use PCI Express Root Ports to start PCI Express hierarchies.
|
||
|
|
|
||
|
|
A PCI Express Root bus supports up to 32 devices. Since each
|
||
|
|
PCI Express Root Port is a function and a multi-function
|
||
|
|
device may support up to 8 functions, the maximum possible
|
||
|
|
number of PCI Express Root Ports per PCI Express Root Bus is 256.
|
||
|
|
|
||
|
|
Prefer grouping PCI Express Root Ports into multi-function devices
|
||
|
|
to keep a simple flat hierarchy that is enough for most scenarios.
|
||
|
|
Only use PCI Express Switches (x3130-upstream, xio3130-downstream)
|
||
|
|
if there is no more room for PCI Express Root Ports.
|
||
|
|
Please see section 4. for further justifications.
|
||
|
|
|
||
|
|
Plug only PCI Express devices into PCI Express Ports.
|
||
|
|
|
||
|
|
|
||
|
|
pcie.0 bus
|
||
|
|
----------------------------------------------------------------------------------
|
||
|
|
| | |
|
||
|
|
------------- ------------- -------------
|
||
|
|
| Root Port | | Root Port | | Root Port |
|
||
|
|
------------ ------------- -------------
|
||
|
|
| -------------------------|------------------------
|
||
|
|
------------ | ----------------- |
|
||
|
|
| PCIe Dev | | PCI Express | Upstream Port | |
|
||
|
|
------------ | Switch ----------------- |
|
||
|
|
| | | |
|
||
|
|
| ------------------- ------------------- |
|
||
|
|
| | Downstream Port | | Downstream Port | |
|
||
|
|
| ------------------- ------------------- |
|
||
|
|
-------------|-----------------------|------------
|
||
|
|
------------
|
||
|
|
| PCIe Dev |
|
||
|
|
------------
|
||
|
|
|
||
|
|
2.2.1 Plugging a PCI Express device into a PCI Express Root Port:
|
||
|
|
-device ioh3420,id=root_port1,chassis=x,slot=y[,bus=pcie.0][,addr=z] \
|
||
|
|
-device <dev>,bus=root_port1
|
||
|
|
2.2.2 Using multi-function PCI Express Root Ports:
|
||
|
|
-device ioh3420,id=root_port1,multifunction=on,chassis=x,addr=z.0[,slot=y][,bus=pcie.0] \
|
||
|
|
-device ioh3420,id=root_port2,chassis=x1,addr=z.1[,slot=y1][,bus=pcie.0] \
|
||
|
|
-device ioh3420,id=root_port3,chassis=x2,addr=z.2[,slot=y2][,bus=pcie.0] \
|
||
|
|
2.2.3 Plugging a PCI Express device into a Switch:
|
||
|
|
-device ioh3420,id=root_port1,chassis=x,slot=y[,bus=pcie.0][,addr=z] \
|
||
|
|
-device x3130-upstream,id=upstream_port1,bus=root_port1[,addr=x] \
|
||
|
|
-device xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=x1,slot=y1[,addr=z1]] \
|
||
|
|
-device <dev>,bus=downstream_port1
|
||
|
|
|
||
|
|
Notes:
|
||
|
|
- (slot, chassis) pair is mandatory and must be unique for each
|
||
|
|
PCI Express Root Port. slot defaults to 0 when not specified.
|
||
|
|
- 'addr' parameter can be 0 for all the examples above.
|
||
|
|
|
||
|
|
|
||
|
|
2.3 PCI only hierarchy
|
||
|
|
======================
|
||
|
|
Legacy PCI devices can be plugged into pcie.0 as Integrated Endpoints,
|
||
|
|
but, as mentioned in section 5, doing so means the legacy PCI
|
||
|
|
device in question will be incapable of hot-unplugging.
|
||
|
|
Besides that use PCI Express to PCI Bridges (pcie-pci-bridge) in
|
||
|
|
combination with PCI-PCI Bridges (pci-bridge) to start PCI hierarchies.
|
||
|
|
|
||
|
|
Prefer flat hierarchies. For most scenarios a single PCI Express to PCI Bridge
|
||
|
|
(having 32 slots) and several PCI-PCI Bridges attached to it
|
||
|
|
(each supporting also 32 slots) will support hundreds of legacy devices.
|
||
|
|
The recommendation is to populate one PCI-PCI Bridge under the
|
||
|
|
PCI Express to PCI Bridge until is full and then plug a new PCI-PCI Bridge...
|
||
|
|
|
||
|
|
pcie.0 bus
|
||
|
|
----------------------------------------------
|
||
|
|
| |
|
||
|
|
----------- -------------------
|
||
|
|
| PCI Dev | | PCIe-PCI Bridge |
|
||
|
|
----------- -------------------
|
||
|
|
| |
|
||
|
|
------------------ ------------------
|
||
|
|
| PCI-PCI Bridge | | PCI-PCI Bridge |
|
||
|
|
------------------ ------------------
|
||
|
|
| |
|
||
|
|
----------- -----------
|
||
|
|
| PCI Dev | | PCI Dev |
|
||
|
|
----------- -----------
|
||
|
|
|
||
|
|
2.3.1 To plug a PCI device into pcie.0 as an Integrated Endpoint use:
|
||
|
|
-device <dev>[,bus=pcie.0]
|
||
|
|
2.3.2 Plugging a PCI device into a PCI-PCI Bridge:
|
||
|
|
-device pcie-pci-bridge,id=pcie_pci_bridge1[,bus=pcie.0] \
|
||
|
|
-device pci-bridge,id=pci_bridge1,bus=pcie_pci_bridge1[,chassis_nr=x][,addr=y] \
|
||
|
|
-device <dev>,bus=pci_bridge1[,addr=x]
|
||
|
|
Note that 'addr' cannot be 0 unless shpc=off parameter is passed to
|
||
|
|
the PCI Bridge/PCI Express to PCI Bridge.
|
||
|
|
|
||
|
|
3. IO space issues
|
||
|
|
===================
|
||
|
|
The PCI Express Root Ports and PCI Express Downstream ports are seen by
|
||
|
|
Firmware/Guest OS as PCI-PCI Bridges. As required by the PCI spec, each
|
||
|
|
such Port should be reserved a 4K IO range for, even though only one
|
||
|
|
(multifunction) device can be plugged into each Port. This results in
|
||
|
|
poor IO space utilization.
|
||
|
|
|
||
|
|
The firmware used by QEMU (SeaBIOS/OVMF) may try further optimizations
|
||
|
|
by not allocating IO space for each PCI Express Root / PCI Express
|
||
|
|
Downstream port if:
|
||
|
|
(1) the port is empty, or
|
||
|
|
(2) the device behind the port has no IO BARs.
|
||
|
|
|
||
|
|
The IO space is very limited, to 65536 byte-wide IO ports, and may even be
|
||
|
|
fragmented by fixed IO ports owned by platform devices resulting in at most
|
||
|
|
10 PCI Express Root Ports or PCI Express Downstream Ports per system
|
||
|
|
if devices with IO BARs are used in the PCI Express hierarchy. Using the
|
||
|
|
proposed device placing strategy solves this issue by using only
|
||
|
|
PCI Express devices within PCI Express hierarchy.
|
||
|
|
|
||
|
|
The PCI Express spec requires that PCI Express devices work properly
|
||
|
|
without using IO ports. The PCI hierarchy has no such limitations.
|
||
|
|
|
||
|
|
|
||
|
|
4. Bus numbers issues
|
||
|
|
======================
|
||
|
|
Each PCI domain can have up to only 256 buses and the QEMU PCI Express
|
||
|
|
machines do not support multiple PCI domains even if extra Root
|
||
|
|
Complexes (pxb-pcie) are used.
|
||
|
|
|
||
|
|
Each element of the PCI Express hierarchy (Root Complexes,
|
||
|
|
PCI Express Root Ports, PCI Express Downstream/Upstream ports)
|
||
|
|
uses one bus number. Since only one (multifunction) device
|
||
|
|
can be attached to a PCI Express Root Port or PCI Express Downstream
|
||
|
|
Port it is advised to plan in advance for the expected number of
|
||
|
|
devices to prevent bus number starvation.
|
||
|
|
|
||
|
|
Avoiding PCI Express Switches (and thereby striving for a 'flatter' PCI
|
||
|
|
Express hierarchy) enables the hierarchy to not spend bus numbers on
|
||
|
|
Upstream Ports.
|
||
|
|
|
||
|
|
The bus_nr properties of the pxb-pcie devices partition the 0..255 bus
|
||
|
|
number space. All bus numbers assigned to the buses recursively behind a
|
||
|
|
given pxb-pcie device's root bus must fit between the bus_nr property of
|
||
|
|
that pxb-pcie device, and the lowest of the higher bus_nr properties
|
||
|
|
that the command line sets for other pxb-pcie devices.
|
||
|
|
|
||
|
|
|
||
|
|
5. Hot-plug
|
||
|
|
============
|
||
|
|
The PCI Express root buses (pcie.0 and the buses exposed by pxb-pcie devices)
|
||
|
|
do not support hot-plug, so any devices plugged into Root Complexes
|
||
|
|
cannot be hot-plugged/hot-unplugged:
|
||
|
|
(1) PCI Express Integrated Endpoints
|
||
|
|
(2) PCI Express Root Ports
|
||
|
|
(3) PCI Express to PCI Bridges
|
||
|
|
(4) pxb-pcie
|
||
|
|
|
||
|
|
Be aware that PCI Express Downstream Ports can't be hot-plugged into
|
||
|
|
an existing PCI Express Upstream Port.
|
||
|
|
|
||
|
|
PCI devices can be hot-plugged into PCI Express to PCI and PCI-PCI Bridges.
|
||
|
|
The PCI hot-plug into PCI-PCI bridge is ACPI based, whereas hot-plug into
|
||
|
|
PCI Express to PCI bridges is SHPC-based. They both can work side by side with
|
||
|
|
the PCI Express native hot-plug.
|
||
|
|
|
||
|
|
PCI Express devices can be natively hot-plugged/hot-unplugged into/from
|
||
|
|
PCI Express Root Ports (and PCI Express Downstream Ports).
|
||
|
|
|
||
|
|
5.1 Planning for hot-plug:
|
||
|
|
(1) PCI hierarchy
|
||
|
|
Leave enough PCI-PCI Bridge slots empty or add one
|
||
|
|
or more empty PCI-PCI Bridges to the PCI Express to PCI Bridge.
|
||
|
|
|
||
|
|
For each such PCI-PCI Bridge the Guest Firmware is expected to reserve
|
||
|
|
4K IO space and 2M MMIO range to be used for all devices behind it.
|
||
|
|
Appropriate PCI capability is designed, see pcie_pci_bridge.txt.
|
||
|
|
|
||
|
|
Because of the hard IO limit of around 10 PCI Bridges (~ 40K space)
|
||
|
|
per system don't use more than 9 PCI-PCI Bridges, leaving 4K for the
|
||
|
|
Integrated Endpoints. (The PCI Express Hierarchy needs no IO space).
|
||
|
|
|
||
|
|
(2) PCI Express hierarchy:
|
||
|
|
Leave enough PCI Express Root Ports empty. Use multifunction
|
||
|
|
PCI Express Root Ports (up to 8 ports per pcie.0 slot)
|
||
|
|
on the Root Complex(es), for keeping the
|
||
|
|
hierarchy as flat as possible, thereby saving PCI bus numbers.
|
||
|
|
Don't use PCI Express Switches if you don't have
|
||
|
|
to, each one of those uses an extra PCI bus (for its Upstream Port)
|
||
|
|
that could be put to better use with another Root Port or Downstream
|
||
|
|
Port, which may come handy for hot-plugging another device.
|
||
|
|
|
||
|
|
|
||
|
|
5.3 Hot-plug example:
|
||
|
|
Using HMP: (add -monitor stdio to QEMU command line)
|
||
|
|
device_add <dev>,id=<id>,bus=<PCI Express Root Port Id/PCI Express Downstream Port Id/PCI-PCI Bridge Id/>
|
||
|
|
|
||
|
|
|
||
|
|
6. Device assignment
|
||
|
|
====================
|
||
|
|
Host devices are mostly PCI Express and should be plugged only into
|
||
|
|
PCI Express Root Ports or PCI Express Downstream Ports.
|
||
|
|
PCI-PCI Bridge slots can be used for legacy PCI host devices.
|
||
|
|
|
||
|
|
6.1 How to detect if a device is PCI Express:
|
||
|
|
> lspci -s 03:00.0 -v (as root)
|
||
|
|
|
||
|
|
03:00.0 Network controller: Intel Corporation Wireless 7260 (rev 83)
|
||
|
|
Subsystem: Intel Corporation Dual Band Wireless-AC 7260
|
||
|
|
Flags: bus master, fast devsel, latency 0, IRQ 50
|
||
|
|
Memory at f0400000 (64-bit, non-prefetchable) [size=8K]
|
||
|
|
Capabilities: [c8] Power Management version 3
|
||
|
|
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
|
||
|
|
Capabilities: [40] Express Endpoint, MSI 00
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
Capabilities: [100] Advanced Error Reporting
|
||
|
|
Capabilities: [140] Device Serial Number 7c-7a-91-ff-ff-90-db-20
|
||
|
|
Capabilities: [14c] Latency Tolerance Reporting
|
||
|
|
Capabilities: [154] Vendor Specific Information: ID=cafe Rev=1 Len=014
|
||
|
|
|
||
|
|
If you can see the "Express Endpoint" capability in the
|
||
|
|
output, then the device is indeed PCI Express.
|
||
|
|
|
||
|
|
|
||
|
|
7. Virtio devices
|
||
|
|
=================
|
||
|
|
Virtio devices plugged into the PCI hierarchy or as Integrated Endpoints
|
||
|
|
will remain PCI and have transitional behaviour as default.
|
||
|
|
Transitional virtio devices work in both IO and MMIO modes depending on
|
||
|
|
the guest support. The Guest firmware will assign both IO and MMIO resources
|
||
|
|
to transitional virtio devices.
|
||
|
|
|
||
|
|
Virtio devices plugged into PCI Express ports are PCI Express devices and
|
||
|
|
have "1.0" behavior by default without IO support.
|
||
|
|
In both cases disable-legacy and disable-modern properties can be used
|
||
|
|
to override the behaviour.
|
||
|
|
|
||
|
|
Note that setting disable-legacy=off will enable legacy mode (enabling
|
||
|
|
legacy behavior) for PCI Express virtio devices causing them to
|
||
|
|
require IO space, which, given the limited available IO space, may quickly
|
||
|
|
lead to resource exhaustion, and is therefore strongly discouraged.
|
||
|
|
|
||
|
|
|
||
|
|
8. Conclusion
|
||
|
|
==============
|
||
|
|
The proposal offers a usage model that is easy to understand and follow
|
||
|
|
and at the same time overcomes the PCI Express architecture limitations.
|