PCIe for Systems Engineers

PCIe is the bus. Every NVMe drive, every GPU, every NIC faster than gigabit, every serious accelerator, and every modern chipset talks to the CPU through it. Most sysadmins have a working relationship with PCIe that starts and ends at lspci. That’s fine for a while. The day your GPU trains at half speed, your NVMe drive benchmarks at a third of its rated IOPS, or your bifurcation card doesn’t split the slot the way you expected, you find yourself looking at lspci -vvv output and realizing the letters and numbers you’ve been scrolling past for years actually mean things.

This post is a practical reference for what PCIe is doing underneath your drivers — lanes and generations, how the root complex and switches are organized, bifurcation, hotplug, ASPM, what all those fields in lspci -vvv mean, and how to diagnose the “why am I not getting full performance” questions that come up when you’re running close to the metal.

The topology, from CPU outward

PCIe is a tree, not a bus. The terminology is hierarchical:

• Root Complex (RC) — lives in the CPU (or chipset, historically). Owns the top of the tree. On a modern x86 CPU, each processor package has one RC; a dual-socket system has two. The RC is the “start” of the PCIe address space.
• Root Port — a PCIe port emerging from the RC. Each root port is its own subtree. A modern Intel or AMD CPU typically has 4-8 root ports directly, plus additional lanes routed through the chipset (PCH on Intel, FCH on AMD).
• Switch — a PCIe multiplexer. A switch has one “upstream” port facing the RC and multiple “downstream” ports facing endpoints. Switches appear on add-in cards like U.2 backplanes, GPU risers, and NVMe HBAs.
• Endpoint (EP) — the actual device. A GPU, NVMe, NIC, etc.
• Bridge — a device that spans two bus segments. Every root port and every downstream switch port is a bridge.

The hierarchy shows up in lspci as bus:device.function (BDF) numbers:

$ lspci -t
-[0000:00]-+-00.0  Intel Corporation Device
           +-01.0-[01]----00.0  NVIDIA Corporation GA102 [RTX 3090]
           +-02.0  Intel Corporation UHD Graphics
           +-06.0-[02]----00.0  Samsung Electronics Co Ltd NVMe SSD 980 PRO
           +-1c.0-[03]--
           +-1c.5-[04]----00.0  Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE
           \-1f.0  Intel Corporation Chipset LPC

Read that as: 00.0 is the CPU root complex. 01.0 is a root port bridging to bus 01, which has the NVIDIA GPU at 01:00.0. 06.0 is another root port bridging to bus 02, which has the NVMe SSD. The chipset sits at 1f.0 and lanes branching from 1c.0 through 1c.5 are chipset-routed ports.

The numbers matter. 00:01.0 is a bridge; 01:00.0 is what’s behind it. That’s why you’ll see advice like “the GPU is at 01:00.0” even though it’s “in the first PCIe slot” — the first slot is its own bus segment, and the first device on that segment is 00.0.

Lanes and generations: the bandwidth math

A PCIe link is a collection of lanes. Each lane is two differential pairs (a TX pair and an RX pair — it’s full duplex). Links are quoted as xN — x1, x4, x8, x16.

A PCIe generation defines per-lane bandwidth and encoding:

Gen 6 is the transition point — lower gens used NRZ encoding (one bit per symbol), Gen 6 uses PAM-4 (two bits per symbol) to double throughput at the same symbol rate. The consequence is tighter signal integrity requirements; Gen 6+ boards are noticeably harder to design and more expensive.

Back-of-envelope performance:

• A Gen4 x4 NVMe drive: 4 × 1.97 GB/s = ~7.9 GB/s theoretical max. Real-world 6-7 GB/s sequential read is the top end.
• A Gen3 x16 GPU: 16 × 0.985 GB/s = ~15.75 GB/s. Gen4 x16 doubles that to ~31.5 GB/s; Gen5 x16 doubles again to ~63 GB/s.
• A Gen5 x16 accelerator: ~63 GB/s. This is why modern AI accelerators are on Gen5 — Gen4 x16 is a real bottleneck for H100-class chips.

A link auto-negotiates to the lowest gen both ends support and the widest width both ends can do. Plug a Gen5 NVMe into a Gen3 slot and you get Gen3 speeds. Plug a x16 card into a x8-wired slot and you get x8.

Checking trained speed vs capability

This is the first thing I check when performance is wrong:

$ sudo lspci -vvs 01:00.0 | grep -E 'LnkCap|LnkSta'
        LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency ...
        LnkSta: Speed 8GT/s (downgraded), Width x16, TrErr- Train- SlotClk+

LnkCap — what the device is capable of (here, Gen4 x16). LnkSta — what it trained at (here, Gen3 x16, marked “downgraded”).

Reasons a link can train below capability:

1. The slot doesn’t support the speed. Check the board manual — a board with Gen4 CPU might route some slots through a Gen3-only chipset.
2. The slot is half-populated. On many boards, x16 slots electrically become x8 when a second slot is populated — the lanes are shared.
3. Signal integrity. Long cables (PCIe risers, U.2 extenders, external enclosures) can force a downshift. LnkCtl2 and retry counters in lspci -vvv tell the story.
4. Firmware setting. BIOS sometimes lets you lock a slot to a specific gen for compatibility. Set to Auto.
5. A bad cable or reseat. Surprisingly common. Powered-off reseat fixes many Gen4/Gen5 training issues.

A link that trains at Gen4 x16 when both ends support Gen5 x16 is a meaningful performance cliff. For ML training at the top end, this can halve your inter-GPU communication throughput.

Bifurcation: splitting a slot

A physical x16 slot is 16 lanes wired to the board. Whether those 16 lanes present as one x16 link, two x8s, four x4s, or eight x2s is a bifurcation decision made in firmware. The CPU’s PCIe controller and the BIOS have to agree on the split.

Why it matters: consumer chipsets and prosumer boards give you a fixed number of lanes, but you often want to attach multiple devices (several NVMe drives via an M.2 bifurcation card, two GPUs, or a GPU plus a NIC). Bifurcation lets one physical x16 slot fan out into four x4 M.2 slots on a carrier board. No switch chip needed — just clever wiring.

Finding the BIOS option: depends on vendor. Look for something like “PCIe Bifurcation,” “PCIe Slot Configuration,” or under “NB/Chipset.” Options typically include x16, x8/x8, x4/x4/x4/x4, and sometimes x8/x4/x4. Not every chipset supports every split.

Which CPUs can bifurcate what:

• Intel Core (mainstream) — usually 1×x16 direct CPU lanes, bifurcation to x8/x8 or x8/x4/x4 is sometimes enabled by the board vendor. Not guaranteed.
• Intel Xeon Scalable / AMD EPYC — full bifurcation control, any power-of-two split.
• AMD Ryzen — x8/x8 and x4/x4/x4/x4 generally work. Board/BIOS must expose the option.
• AMD Threadripper — full control, multiple x16 slots all bifurcatable.

Bifurcation cards vs switch cards:

• Bifurcation card — passive PCB routing lanes. Needs host bifurcation support. Cheap (~$30), no driver, no latency penalty.
• PCIe switch card — active chip (PLX/Broadcom, Microchip). Presents one uplink to the host, fans out to many devices. Does NOT need bifurcation. Adds ~100ns latency per hop. More expensive ($300+).

For a cheap NVMe expansion, bifurcation cards are great if your board supports the split. If not, a switch card solves the problem at cost.

Hot-plug

PCIe supports hot-plug — physically inserting and removing devices while the system runs. Required on any serious server: hot-swap NVMe backplanes, hot-plug NICs in some blade chassis, FPGA expansion via ExpressCard-descendants, Thunderbolt.

For hot-plug to work end-to-end:

1. Slot must be hot-plug-capable. Hardware design issue — there must be circuitry to ramp power up/down cleanly.
2. Firmware must flag the slot as hot-plug. Expressed via the SltCap (Slot Capabilities) register — HotPlug+ Surprise- means “can be removed via software eject” and HotPlug+ Surprise+ means “can be yanked without warning.”
3. OS must enable the pciehp driver. Modern Linux kernels have this on by default.

Eject a device on Linux:

1

echo 1 | sudo tee /sys/bus/pci/devices/0000:02:00.0/remove

Rescan (after plugging in a new one):

1

echo 1 | sudo tee /sys/bus/pci/rescan

Thunderbolt / USB4 is PCIe hot-plug over a cable (plus DisplayPort tunneling). The thunderbolt subsystem handles authorization (whether to let a new device get PCIe access — a security consideration, since PCIe devices can DMA) and hands off device enumeration to pciehp.

ASPM — Active State Power Management

PCIe links consume power. ASPM is the mechanism for powering down idle links:

• L0 — full speed, full power
• L0s — low-power standby, sub-microsecond wake
• L1 — deeper standby, microsecond-to-millisecond wake
• L1.1, L1.2 — even deeper substates introduced for laptops/embedded

ASPM is the most common source of “my NIC randomly stops responding” stories on Linux. An aggressive ASPM transition plus a firmware bug in a device can drop packets or fail to wake in time. Historical culprits: some Intel NICs, early Broadcom Wi-Fi cards, a handful of NVMe drives.

Current state of a link:

$ sudo lspci -vvs 02:00.0 | grep -i ASPM
        LnkCap: ... ASPM L1
        LnkCtl: ASPM Disabled; ClockPM+ Surprise- LLActRep+ BwNot+

Disable ASPM platform-wide on the kernel cmdline:

pcie_aspm=off

Or policy-based:

pcie_aspm.policy=performance   # always L0
pcie_aspm.policy=powersave     # aggressive
pcie_aspm.policy=default       # firmware decides

For laptops, ASPM enabled saves 2-5 W of idle power — worth having on by default. For a server where you care about consistent latency and have plenty of power, pcie_aspm=off avoids a class of bugs.

Max Payload Size, Max Read Request Size

Every PCIe transaction is a packet. The root complex and each endpoint have a Max Payload Size (MPS) capability (128, 256, 512 bytes, rarely larger). The actual MPS on a link is the minimum of all devices in the path — a 128-byte endpoint attached to a 512-byte root complex runs at 128.

Smaller packets = more header overhead = lower effective bandwidth. Going from MPS 128 to MPS 512 can improve throughput by 10-20% on some workloads.

Max Read Request Size (MRRS) controls how much data an endpoint requests in one read — independent of MPS. Larger MRRS reduces round-trips to memory. Drivers tune this per-device.

View:

$ sudo lspci -vvs 01:00.0 | grep -E 'Max(Payload|ReadReq)'
                DevCtl: ... MaxPayload 128 bytes, MaxReadReq 512 bytes

Forcing MPS is possible with pci=pcie_bus_perf on the kernel cmdline (the Linux kernel walks the tree and picks the max both ends support per link). This is generally safe but can expose firmware bugs in rarely-used paths.

IOMMU groups, ACS, and isolation

The IOMMU (VT-d / AMD-Vi) remaps DMA addresses so a device can only write to memory the host intends. For virtualization and bare-metal safety, this is essential.

ACS (Access Control Services) is a PCIe capability that lets the root complex enforce isolation between sibling devices behind the same switch. If two devices share a switch without ACS, they can bypass the IOMMU and talk peer-to-peer. For security, the IOMMU subsystem groups devices with no ACS isolation into a single IOMMU group — and you can only pass an entire group to a VM, not individual devices in it.

List groups:

1
2
3
4

for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/}; n=${n%%/*}
  printf 'Group %s: %s\n' "$n" "$(basename $d)"
done | sort -V

On server boards, groups are typically fine-grained — each device in its own group. On consumer boards, groups can be coarse because the root port doesn’t implement ACS — everything behind that port is lumped together.

The pcie_acs_override kernel cmdline (kernels with the patch applied — many distros ship it) lets you tell the kernel to pretend ACS is there:

pcie_acs_override=downstream,multifunction

This “fixes” IOMMU groups on consumer boards at the cost of an isolation guarantee. In a homelab, fine. In production where attackers could exploit one VF to attack another behind the same switch, don’t.

Peer-to-peer DMA

PCIe devices can DMA to each other’s memory without going through system RAM. The canonical use case: GPU-direct storage — NVMe drives DMA’ing directly into GPU memory, bypassing the CPU and system RAM. NVIDIA’s GPUDirect Storage, AMD’s ROCm RDMA, and the generic Linux p2pdma framework all implement this.

Requirements:

• Devices must be on the same PCIe fabric (under the same root complex). P2P between sockets goes through the inter-socket link and performance suffers.
• ACS on intermediate switches must permit peer-to-peer (ACS redirect bits).
• The root complex must support “peer-to-peer memory access” — some server chipsets do, most consumer chipsets don’t.

When it works, P2P DMA saves memory bandwidth and reduces latency. When it doesn’t, you get opaque errors and drivers fall back to the slow path through host memory. The lspci -vvs ACS info and the root complex’s documentation tell you what’s supported.

Reading lspci -vvv like a pro

A heavily-annotated excerpt for a GPU:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GA102 [GeForce RTX 3090]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 198
        IOMMU group: 14
        Region 0: Memory at 80000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at 1800000000 (64-bit, prefetchable) [size=8G]
        Region 3: Memory at 1a00000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at 4000 [size=128]
        Expansion ROM at 81000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                DevCtl: ... MaxPayload 256 bytes, MaxReadReq 512 bytes
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <64us
                LnkSta: Speed 16GT/s, Width x16
                DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100 v1] Virtual Channel
        Capabilities: [250 v1] Latency Tolerance Reporting
        Capabilities: [258 v1] L1 PM Substates
        Capabilities: [128 v1] Power Budgeting <?>
        Capabilities: [420 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: nvidia
        Kernel modules: nvidia

Key things to read:

• Region 0-5 — BARs (Base Address Registers). Memory and I/O regions mapped into CPU address space. prefetchable means the CPU can reorder/combine accesses — typical for GPU framebuffers.
• Capabilities: Express — this is the PCIe-specific capability. All the link speed/width, payload, ASPM, and substrate info is here.
• LnkCap vs LnkSta — capability vs status. Already covered.
• Advanced Error Reporting (AER) — UESta, CESta — uncorrectable and correctable error counters. If CESta shows non-zero BadTLP/BadDLLP, you have signal-integrity issues. Non-zero uncorrectable counters are a warning of imminent hardware failure.
• L1 PM Substates — deeper power states for laptops.
• Virtual Channel — PCIe supports multiple logical traffic classes on one link. Rarely used on commodity hardware but present for QoS-sensitive applications.

PCIe errors and what to do about them

PCIe has a hierarchy of errors:

• Correctable — retried by the hardware. You don’t notice unless the count is high. AER registers track them.
• Uncorrectable non-fatal — the transaction failed; driver gets notified. Usually logged as AER events.
• Uncorrectable fatal — the link is dead. System may hang, reboot, or reset the slot depending on configuration.

On Linux, dmesg will show AER events:

pcieport 0000:00:01.0: AER: Corrected error received: 0000:01:00.0
pcieport 0000:00:01.0:   device [8086:1905] error status/mask=00000001/00002000
pcieport 0000:00:01.0:    [ 0] RxErr

A trickle of corrected errors is not dire. A burst usually means bad signal integrity — reseat, swap cable, or lower the link speed.

lspci -vvv | grep -A 10 AER shows the current state. sudo setpci -s <BDF> ECAP_AER+0x10.l=0xffffffff clears counters.

For a server, route AER errors to rasdaemon or the kernel’s edac subsystem for fleet-level monitoring.

Reset: the forgotten feature

Every PCIe function has a Function-Level Reset (FLR) capability. Write a bit, the function resets without affecting its siblings. Crucial for virtualization: when a VM starts or shuts down, the GPU/NIC/whatever being passed through should reset cleanly so the next user sees a fresh device.

Not all devices implement FLR correctly. This is the bane of GPU passthrough, as discussed in the GPU post — the “reset bug” on some AMD cards is an FLR implementation defect.

Check for FLR:

sudo lspci -vvs <BDF> | grep FLReset
    DevCap: ... FLReset+ ...

Workarounds when FLR is missing or buggy:

• Slot-level reset — via the pcie root port: echo 1 > /sys/bus/pci/devices/<BDF>/reset. Goes through the parent bridge’s secondary-bus-reset. Less graceful but usually works.
• Power cycling via the slot’s hot-plug control — if the slot supports it.
• A full host power cycle — when nothing else works.

NUMA and PCIe

On a multi-socket system, each root complex is physically attached to one CPU socket. A GPU plugged into socket 0’s root port is “local” to socket 0. Accessing it from socket 1 means crossing the inter-socket link (UPI on Intel, xGMI on AMD), adding latency and consuming cross-socket bandwidth.

Check which NUMA node a device is on:

$ cat /sys/bus/pci/devices/0000:01:00.0/numa_node
0

For latency-sensitive workloads, pin the process to the same NUMA node as the device:

numactl --cpunodebind=0 --membind=0 myapp

Or for a DB process accessing an NVMe, make sure the NVMe is on the same node as the CPU cores doing I/O. Mixing node allocation with PCIe device placement is a common cause of mysterious performance variance on multi-socket systems.

A few practical troubleshooting recipes

“My NVMe is slower than spec.”

1. lspci -vv | grep -E 'LnkCap|LnkSta' — is it Gen4 x4 or did it downshift?
2. nvme id-ctrl /dev/nvme0 — any error counters rising?
3. cat /sys/class/nvme/nvme0/device/numa_node and run your benchmark pinned to that node.
4. Check MaxPayload / MaxReadReq. Low values limit throughput.
5. Does the slot share lanes with another slot? Remove the other card and retest.

“My GPU trains but gaming performance is half.”

1. Link speed/width (LnkSta).
2. Which PCIe root port is the GPU on? Consumer boards sometimes route slot 1 directly and slot 2 through the chipset with more latency.
3. Is the slot electrically the full x16 the connector suggests? Board manual.
4. ASPM on a latency-sensitive GPU — try pcie_aspm=off.

“My multi-GPU setup’s inter-GPU bandwidth is terrible.”

1. Are the GPUs on the same root complex / NUMA node? If not, traffic crosses the inter-socket link.
2. If they’re behind a switch, verify the switch chip’s generation — an older PLX 87xx switch is Gen3 even if the GPUs are Gen4.
3. Check if NVLink / Infinity Fabric Link is working — if it is, most inter-GPU traffic should bypass PCIe entirely.

“PCIe device disappeared after reboot.”

1. dmesg for AER fatal errors.
2. Reseat.
3. BIOS slot settings — some boards disable a slot when nothing’s detected at POST.
4. Power supply capacity — some cards don’t train until they have adequate power, and a marginal PSU under load can cause intermittent disappearance.

Where PCIe is going

Developments worth tracking:

• CXL (Compute Express Link) — layered on the PCIe 5.0/6.0 physical layer. Adds cache-coherent memory (CXL.mem), coherent device access (CXL.cache), and accelerator attach (CXL.io). Memory expansion modules that slot into a PCIe slot and appear as additional system memory. Expected to reshape how large-memory systems are built.
• Gen 6 and Gen 7 — Gen 6 ships on enterprise platforms 2025-26. PAM-4 signaling and FLIT mode add complexity; the ecosystem is still stabilizing.
• OCP Accelerator Module (OAM) — a standardized mezzanine form factor that uses PCIe + custom host interconnect. AMD MI300, NVIDIA’s HGX baseboards are implementations of this idea.
• NVMe-oF and disaggregated storage — PCIe becoming optional for NVMe as the wire protocol moves over Ethernet/RDMA.

Closing

PCIe feels like one of those pieces of infrastructure you’re supposed to take for granted — the wires that connect the good parts. It’s not. The lane count, the generation, the tree topology, the IOMMU grouping, the ASPM state, the error counters — each of those is a knob or a failure mode that shapes real system behavior. Once you can read lspci -vvv and understand what it’s telling you, you stop guessing when performance is wrong and start diagnosing. You know whether a slot is Gen4 or Gen3, whether it’s x16 or x8 electrically, whether your GPU is on the “right” NUMA node, and whether ASPM is interacting with your NIC.

These are small, concrete facts. Together they’re the difference between “the system is slow” and “the system’s slot 3 is Gen3-limited by the chipset, move the NVMe to slot 2 and it’s fine.” The datasheet is in your system. You just have to read it.