GPU Passthrough End-to-End

GPU passthrough is one of those features that, on paper, is simple: take a PCIe device, hand it exclusively to a VM, let the VM’s driver talk to it as if it owned the hardware. In practice, it’s a maze of kernel modules, firmware quirks, vendor-specific countermeasures, and Windows error codes that seem designed to make you give up. The good news is that once you understand the four or five layers that have to line up, you can troubleshoot anything from “the host crashes when the VM starts” to the infamous NVIDIA Error 43.

This post walks the end-to-end flow for giving a VM full PCIe access to a GPU: IOMMU groups and why they matter, VFIO driver binding, libvirt XML plumbing, SR-IOV for datacenter cards, vGPU licensing, and the most common failure modes with their real causes. It assumes you have a working KVM/libvirt host (the previous post in this series covers that). If you want the one-line “does my system support this,” run dmesg | grep -i -e DMAR -e IOMMU and lspci -nnk and see whether your BIOS exposes IOMMU and your hardware has identifiable devices. If those work, read on.

What GPU passthrough actually is

Modern CPUs have two hardware virtualization extensions relevant here:

1. VT-x / AMD-V — for executing guest CPU instructions efficiently. Every modern CPU has this.
2. VT-d / AMD-Vi (IOMMU) — for remapping DMA, so a device owned by a VM can only write to memory pages the VM is allowed to touch. Without this, a passed-through device could DMA over the host’s kernel memory and you’d have no isolation at all.

Passthrough relies on the IOMMU. The VM thinks it owns a GPU; QEMU, via VFIO and the IOMMU, makes sure that the GPU’s DMA requests are translated through the guest’s page tables and cannot escape the guest’s address space. Without an IOMMU, modern passthrough simply won’t start — vfio-pci refuses to bind.

The stack of layers, from metal up:

• BIOS/UEFI — VT-d / AMD-Vi enabled, SR-IOV enabled if applicable, ACS / “PCIe ACS Override” if your board groups too coarsely.
• Host kernel — intel_iommu=on or amd_iommu=on on the cmdline, vfio-pci loaded and bound to the target devices, nvidia/amdgpu blacklisted for passthrough devices.
• QEMU/libvirt — <hostdev> entries in domain XML, UEFI OVMF firmware (Secure Boot off for most GPU guests unless you’re doing signed-driver work), optional ROM BIOS file for problematic cards.
• Guest OS — vendor drivers installed; for NVIDIA consumer cards, the driver checks whether it’s in a hypervisor and (historically) refuses to load. That’s Error 43.

IOMMU groups: where most first attempts die

An IOMMU group is the smallest unit you can pass to a VM. If your GPU is in a group with the audio controller (common on modern cards), you pass both. If it’s in a group with the PCIe root port and every NIC on the motherboard, you can’t pass the GPU without passing the rest — and you can’t pass the rest because the host needs the NIC.

Check your groups:

1
2
3
4
5

for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done | sort -V

Output looks like:

IOMMU Group 14 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204]
IOMMU Group 14 01:00.1 Audio device [0403]: NVIDIA Corporation GA102 HDMI Audio [10de:1aef]
IOMMU Group 15 02:00.0 Ethernet controller [0200]: Intel I225 [8086:15f3]

If the GPU is in a group with only its HDMI audio, you’re in good shape — pass both. If it’s grouped with a root port, a USB controller, and the NVMe drive holding your root filesystem, you have a problem.

The ACS Override patch. Some motherboards lump devices together because the PCIe root port doesn’t implement ACS (Access Control Services) properly, which is what the IOMMU uses to know that two devices can be isolated. The ACS override patch (in kernels with the patch applied, or by adding pcie_acs_override=downstream,multifunction to the cmdline on patched kernels) tells the kernel “trust me, these devices are isolated.” It’s a security compromise — you’re asserting something the hardware couldn’t prove — but in a homelab this is usually fine. In production, replace the motherboard with one that has proper ACS (server boards almost always do; consumer boards are hit-or-miss).

Workstation boards are the sweet spot. ASRock Rack, Supermicro X11/X12, and most AMD Threadripper TRX40/WRX80 boards have fine-grained IOMMU groups. Consumer Z-series and B-series boards are mixed. Before buying a board for passthrough, search IOMMU groups <board model> — someone has tested it.

Enabling IOMMU on the host

Add to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"
# or for AMD
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on iommu=pt"

iommu=pt sets passthrough mode for devices owned by the host — they bypass translation, which avoids a small performance cost. Devices bound to vfio-pci always get translation regardless.

Regenerate grub and reboot:

sudo update-grub       # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg   # Fedora/RHEL

After reboot, dmesg | grep -i iommu should show enabled:

DMAR: IOMMU enabled
DMAR: Intel(R) Virtualization Technology for Directed I/O

Binding the GPU to vfio-pci

By default the nouveau, nvidia, or amdgpu driver will grab the GPU during boot. For passthrough, we need vfio-pci to claim it first.

Find the device IDs:

$ lspci -nn | grep -i nvidia
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 HDMI Audio [10de:1aef] (rev a1)

The bracketed pairs are vendor:device IDs. Create /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:2204,10de:1aef
softdep nvidia pre: vfio-pci
softdep nouveau pre: vfio-pci

The softdep lines make the kernel load vfio-pci before the NVIDIA or nouveau driver tries to bind. For AMD, softdep amdgpu pre: vfio-pci (and radeon on older cards).

Ensure the modules are in the initramfs:

# /etc/modules-load.d/vfio.conf
vfio
vfio_iommu_type1
vfio_pci

On Debian/Ubuntu:

echo "vfio_pci" | sudo tee -a /etc/initramfs-tools/modules
sudo update-initramfs -u

On Fedora:

sudo dracut --force --kver $(uname -r)

Reboot. Verify:

$ lspci -nnk -s 01:00.0
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia

If it says Kernel driver in use: nvidia, the driver claimed it first. Either nvidia loaded from initramfs (add it to the initramfs blacklist) or the softdep is not applying (check module load order in dmesg).

The single-GPU problem

If the passthrough GPU is also your host display, there is no way to have both a console on the host and pass the GPU. You have three options:

1. Boot headless. Use serial console, SSH, or IPMI for host access. Easy, but you lose the host desktop.
2. Second GPU. An iGPU (Intel UHD, Ryzen APU) or a cheap second GPU handles the host display; the passthrough GPU goes to the VM.
3. Dynamic binding. Boot normally, then rebind the GPU to vfio-pci at VM start. Possible but finicky: the open-source drivers release the GPU reliably, the NVIDIA driver sometimes refuses. Scripts for this exist (the “single-GPU passthrough” setups in the VFIO community) and involve stopping the display manager, rebinding, starting the VM, then doing the reverse on VM shutdown.

The two-GPU approach is by far the simplest.

The libvirt XML

Once vfio-pci owns the GPU, the domain needs <hostdev> entries for each function you’re passing. Edit the guest XML with virsh edit gamer-vm:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<devices>
  ...
  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </source>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0' multifunction='on'/>
  </hostdev>
  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </source>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
  </hostdev>
</devices>

Key details:

• managed='yes' — libvirt detaches the device from the host driver at VM start and reattaches at shutdown. With managed='no', you handle binding yourself.
• multifunction='on' on the first hostdev — so the VM presents a single PCIe device with two functions (GPU + audio), mirroring how the host sees it. Some drivers care.
• Source address is the host PCI address (01:00.0). Target address is where the device appears inside the VM (02:00.0). Keep both functions on the same bus/slot in the guest so they stay grouped.

UEFI, Q35, and Secure Boot

Use q35 and UEFI. Many drivers (especially Windows 10+) assume UEFI:

1
2
3
4
5
6

<os>
  <type arch='x86_64' machine='pc-q35-9.0'>hvm</type>
  <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
  <nvram template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/gamer_VARS.fd</nvram>
  <boot dev='hd'/>
</os>

Hiding the hypervisor from the guest (NVIDIA Error 43)

Historically, NVIDIA’s consumer driver checked whether it was running under a hypervisor and refused with Error 43. In 2021, NVIDIA relaxed this for most cards and it’s no longer required on current drivers. But if you do hit it, the workaround is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

<features>
  <acpi/>
  <apic/>
  <hyperv mode='custom'>
    <relaxed state='on'/>
    <vapic state='on'/>
    <spinlocks state='on' retries='8191'/>
    <vendor_id state='on' value='randomid'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
  <vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='on'>
  <topology sockets='1' dies='1' cores='6' threads='2'/>
  <feature policy='disable' name='hypervisor'/>
</cpu>

<hidden state='on'/> in the KVM block hides the KVMKVMKVM CPU signature. <vendor_id> masks the Hyper-V vendor. feature policy='disable' name='hypervisor' clears the hypervisor CPUID bit. With all three, the guest driver cannot tell it’s virtualized through ordinary means.

Even with the post-2021 driver, adding these stanzas doesn’t hurt and makes older guests (including many Linux distros with older NVIDIA drivers) happier.

CPU pinning for gaming performance

If you’re passing through a GPU for a gaming VM, pin the vCPUs to real cores — don’t let the scheduler move them around. Example for a 6-core/12-thread partition on a 16-core CPU:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

<vcpu placement='static'>12</vcpu>
<cputune>
  <vcpupin vcpu='0'  cpuset='4'/>
  <vcpupin vcpu='1'  cpuset='20'/>   <!-- sibling HT -->
  <vcpupin vcpu='2'  cpuset='5'/>
  <vcpupin vcpu='3'  cpuset='21'/>
  <vcpupin vcpu='4'  cpuset='6'/>
  <vcpupin vcpu='5'  cpuset='22'/>
  <vcpupin vcpu='6'  cpuset='7'/>
  <vcpupin vcpu='7'  cpuset='23'/>
  <vcpupin vcpu='8'  cpuset='8'/>
  <vcpupin vcpu='9'  cpuset='24'/>
  <vcpupin vcpu='10' cpuset='9'/>
  <vcpupin vcpu='11' cpuset='25'/>
  <emulatorpin cpuset='0-3,16-19'/>
  <iothreadpin iothread='1' cpuset='0-3,16-19'/>
</cputune>
<cpu mode='host-passthrough' check='none'>
  <topology sockets='1' dies='1' cores='6' threads='2'/>
</cpu>

The key idea: dedicate cores 4-9 (and their hyperthread siblings 20-25) to the guest. Keep cores 0-3 (and 16-19) for the host and emulator thread. isolcpus=4-9,20-25 on the host kernel cmdline prevents the Linux scheduler from placing tasks on those cores.

Hugepages

For 16 GB of VM RAM with 1 GB hugepages:

# Host cmdline
default_hugepagesz=1G hugepagesz=1G hugepages=16

1
2
3
4
5
6
7

<memoryBacking>
  <hugepages>
    <page size='1' unit='G' nodeset='0'/>
  </hugepages>
  <nosharepages/>
  <locked/>
</memoryBacking>

<locked/> is important with passthrough — the device does DMA directly into guest RAM, so the pages cannot be swapped.

The ROM BIOS trick

Some cards, especially those used in a host’s primary display slot before being passed, have their vBIOS shadow region clobbered and the card will not reinitialize cleanly inside a VM. The symptom: the first VM boot works, subsequent boots hang. Or the GPU never produces output in the VM.

Workaround: capture a clean vBIOS and load it from a file. Download the ROM for your exact card+rev from a reputable source (e.g., TechPowerUp’s VGA BIOS Collection), place at /var/lib/libvirt/vbios/gpu.rom, and:

1
2
3
4
5
6
7

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
  </source>
  <rom file='/var/lib/libvirt/vbios/gpu.rom'/>
  ...
</hostdev>

Modern NVIDIA cards often need a tweaked vBIOS with the NVIDIA “Video BIOS” header stripped. The rom-parser tool plus a small header-strip script (the first 0x200 bytes are often the problematic part) is the standard fix. If you don’t know whether you need this, boot the VM without it first — if passthrough works, leave it alone.

The reset bug

Some AMD GPUs (notoriously the RX 480/580, Vega, early Navi) have a hardware reset bug: the card cannot be reset cleanly via PCIe FLR (Function-Level Reset). First VM boot works; when you shut down the VM, the card gets stuck in a weird state and won’t initialize for the next boot. Usually the host hangs too when it tries to probe the card.

Workarounds:

• vendor-reset — a kernel module that implements vendor-specific reset sequences. Covers most affected AMD cards.
• Never stop the VM. Works around the bug by never triggering it. Practical if the VM is long-running.
• PCIe reset at the root port — echo 1 > /sys/bus/pci/devices/<root-port>/reset. Sometimes works.
• Buy a different card. NVIDIA and Intel ARC don’t have this specific bug. Newer AMD cards (RDNA 2+) have it mostly fixed.

If you’re choosing hardware for passthrough, the reset behavior should weigh in. The vfio-users mailing list and the level1techs forum are authoritative on which card+firmware combinations work.

SR-IOV: splitting one card into many

Single Root I/O Virtualization lets one physical PCIe device present itself as multiple virtual functions (VFs), each independently assignable to a VM. Datacenter NICs have had SR-IOV for years. Some datacenter GPUs do too:

• NVIDIA datacenter cards (Tesla T4, A10, A40, A100, H100, L40) support SR-IOV in vGPU mode with a licensed vGPU driver.
• AMD MI-series (MI25, MI50, MI250, MI300) support SR-IOV on Linux.
• Intel Flex and Data Center GPU Max support SR-IOV natively without licensing.

Enable VFs on an SR-IOV capable GPU:

echo 4 > /sys/class/mdev_bus/0000:01:00.0/sriov_numvfs

Or the older path:

echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs

New PCIe functions appear (01:00.1, 01:00.2, etc.). Each is passable to a VM as a separate device. The parent device stays on the host, managed by its normal driver.

SR-IOV is how cloud providers slice one A100 across many tenants. Each VF has its own PCIe config space, its own doorbell, its own interrupt context — isolation is hardware-enforced.

Mediated devices (vGPU)

Before datacenter GPUs supported SR-IOV cleanly, the approach was mediated devices (mdev) — a Linux kernel framework where the driver exposes a set of profiles (“I can create virtual GPUs of type A with 4GB, type B with 8GB, etc.”) and userspace creates instances. NVIDIA’s nvidia-vgpu-mgr is the production example, implementing time-slicing virtualization of a physical GPU.

Workflow:

1. Install NVIDIA vGPU Manager on the host (licensed software, NVIDIA enterprise subscription).
2. Check available mdev types: ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types.
3. Create an mdev instance: echo <uuid> > /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-257/create.
4. Assign to VM with <hostdev mode='subsystem' type='mdev' model='vfio-pci'> and the uuid.
5. Inside the guest, a licensed NVIDIA vGPU driver calls home to a license server to get scheduling permission.

vGPU is powerful (it’s what powers VDI deployments and shared GPU clusters) but expensive: both the manager licenses and the per-guest licenses add up. For homelab, stick with full-card passthrough or use a card that supports free SR-IOV (Intel GPUs, older AMD MxGPU).

The unlicensed vGPU saga

There is an open-source project called vgpu_unlock (and its successor vgpu_unlock-rs) that patches the NVIDIA vGPU Manager to run on consumer cards by impersonating a Tesla card. It’s remarkable work but in a legal gray area — NVIDIA’s license forbids it, and they patch their drivers to defeat it. If you’re going down this path, understand that you’re playing against NVIDIA’s release schedule.

CUDA inside a passed-through GPU

A passed-through GPU behaves like a normal GPU inside the VM. Install the driver stack as usual:

# Inside the Ubuntu VM
sudo apt install nvidia-driver-550
sudo reboot
nvidia-smi

nvidia-smi should show the card with its full VRAM, temperatures, clocks, etc. CUDA workloads run at essentially native speed — you pay a small PCIe DMA remapping cost (1-2%) and no arithmetic overhead.

One thing that trips people up: don’t mix host and guest drivers. On the host, the NVIDIA driver must not be loaded at all (it’s blacklisted/softdep’d behind vfio-pci). If the host loads the NVIDIA driver, it races with vfio-pci and passthrough fails intermittently.

The troubleshooting matrix

Symptoms you’ll see and their usual causes:

Two tools for diagnosing:

• dmesg on the host — VFIO logs enter/exit reasons, IOMMU faults, unsupported access attempts. If the host logs “IOMMU fault addr 0x…” after VM start, the guest driver is DMA-ing outside its allowed range — often a sign of a broken vBIOS or a card that needs a specific quirk.
• journalctl -u libvirtd and the per-domain log in /var/log/libvirt/qemu/<name>.log — the full QEMU command line libvirt built, plus device init messages.

A pragmatic homelab setup

If you’re doing this for a gaming VM or ML dev workstation, an opinionated recipe that works:

1. AMD Ryzen or Threadripper + ASRock Rack or ASUS ProArt board — good IOMMU groups.
2. Intel iGPU (via dGPU-disabled Ryzen) or a cheap GT 710 for host video.
3. NVIDIA RTX 30/40/50 series for passthrough. They work well and have no reset bug.
4. intel_iommu=on iommu=pt isolcpus=<guest-cores> nohz_full=<guest-cores> rcu_nocbs=<guest-cores> on the host cmdline.
5. 1 GB hugepages, 16-32 GB reserved.
6. virtio everything else — virtio-net, virtio-blk, virtio-scsi, virtio-fs (for shared folders from host).
7. Looking Glass — an open-source KVMFR shared-memory bridge that lets you see the VM’s framebuffer in a window on the host. Zero-copy, very low latency. Combines with passed-through USB keyboard/mouse for a seamless experience.
8. Snapshotting and rollback of the guest disk for experimenting with drivers without fear.

For a small homelab “shared GPU server,” the Intel Flex 170 or an AMD Instinct MI25 with SR-IOV lets you slice one GPU across multiple lightweight VMs with hardware isolation.

Closing

GPU passthrough is a perfectly reasonable thing to run for years without incident once you get past the initial setup. Every layer has gotten better: IOMMU groups on server boards are clean, NVIDIA has stopped actively fighting consumer-card passthrough, AMD’s reset bugs are mostly fixed on RDNA2+, OVMF is stable, QEMU’s q35 machine type handles PCIe properly, and the VFIO subsystem in the kernel has matured. The days of patching kernels and hex-editing vBIOS files are mostly gone.

What’s left is understanding the pieces and matching them. IOMMU isolates. VFIO binds. libvirt’s XML expresses the topology. The guest driver sees real hardware. When it all lines up, you have a VM with 99% of the performance of a bare-metal install of the same GPU, fully isolated from the host, snapshottable, migratable (with some caveats), and disposable. That is a kind of capability you simply can’t get from containerization or software GPU sharing, and it’s worth the one-time setup cost.