Device passthrough in KVM

Chao Cheng

Last updated on Dec 10, 2021 8 min read KVM

Introduction

This note is about device passthrough in KVM. Idealy, device passthrough will give you near-native performance in the VMs.

Basic idea

Devices in the same IOMMU group should be passed through to the VM together.
Devices on the same physical board should (better) be passed through together.
- This is especially true for NVIDIA graphics card where there are video and audio (and even usb controller for newer models) on the same board. And their driver will refuse to work if it detects the card in the VM does not meet its physical design.

Useful tools and commands

ls-iommu.sh script for checking your IOMMU information:

 #!/bin/bash
 shopt -s nullglob
 for d in /sys/kernel/iommu_groups/*/devices/*; do
   n=${d#*/iommu_groups/*}; n=${n%%/*}
   printf 'IOMMU Group %s \n' "$n"
   lspci -vmms "${d##*/}" |grep -E "^Slot|^Class|^Vendor|^Device"
   printf 'IDs:    '
   lspci -ns "${d##*/}" | awk {print\ \$3}
   lspci -vmms "${d##*/}" |grep -E "^Rev"
   printf '\n'
 done

This ls-iommu script comes from FurryJackman at Level1Techs.

A command to list iommu groups and if devices support reset

 for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

This command comes from SpaceInvader One at Dropbox.

A command to list all usb controllers buses and devices

 for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

This command comes from SpaceInvader One at Dropbox.

Some useful commands

 lspci    # use -vv and -nn to show more information
          # use -h more more help
 lspci | grep USB
 lsusb
 lsusb -t

USB device passthrough

USB redirection

This is an easier way but lacks some features (e.g. hot-pulg, reset, .etc). The performance may depend on the USB redirector of your KVM installation.

Use virt-manager -> add hardware -> USB host device to identify the device you want to redirect to the guest system.
Choose that hardware and press Finish to add it to your guest VM.
- It might took the system secondes or even sometimes minutes to finish this redirection.
- If your host machine is a laptop, NEVER redirect your built-in keyboard nor mouse to the guest system since you will loss control of your host unless you have additional keyboard and mouse plug in to it.

If you prefer the command line tool, you can use lsusb to identify the USB device. For example in my laptop it reads

 Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 004: ID 06cb:00bd Synaptics, Inc. 
 Bus 001 Device 003: ID 13d3:56bc IMC Networks Integrated Camera
 Bus 001 Device 006: ID 0930:6544 Toshiba Corp. TransMemory-Mini / Kingston DataTraveler 2.0 Stick
 Bus 001 Device 002: ID 046d:c534 Logitech, Inc. Unifying Receiver
 Bus 001 Device 005: ID 8087:0aaa Intel Corp. 
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The Toshiba USB stick, with device ID 0930:6544 is what I want to pass to the VM.

Add or modify the <hostdev></hostdev> section inside your XML’s <devices></devices> section
```
 <hostdev mode="subsystem" type="usb" managed="yes">
   <source>
     <vendor id="0x0930"/>
     <product id="0x6544"/>
   </source>
   <address type="usb" bus="0" port="4"/>
 </hostdev>
```
You should change the vender id and product id after the 0x part to your device’s information. Also you might have to adjust the bus and port setting. In my VM settings, port 0~3 of bus 0 have already been occupied.
- I recommend using the virt-manager since it can fill in the correct bus and port settings for you.

USB controller passthrough

If you’re looking for something like hot-swap mouse and keyboard, or hot-plug USB drive in your guest system. Then you may consider passing through a USB controller to it.

If your host machine is a laptop, NEVER passthrough the USB controller responsible for your built-in keyboard and trackpad to the guest system. You will loss control to your host unless you have additional mouse and keyboard plug-in to it through different controller.

Identify your USB controller
```
lspci | grep USB
```
In my laptop, there are two of them
```
00:14.0 USB controller: Intel Corporation Cannon Point-LP USB 3.1 xHCI Controller (rev 30)
3a:00.0 USB controller: Intel Corporation JHL6240 Thunderbolt 3 USB 3.1 Controller (Low Power) [Alpine Ridge LP 2016] (rev 01)
```
These two controllers are at PCI path 00:14.0 and 3a:00.0. From previous section, I know that the Toshiba USB stick is plugged in the first Bus, therefore I have to identify the relationship between these controller and the internal buses.

Use the 3rd command provided in section Usefull tools and commands to identify the controller-bus-device relationship. In my laptop. it outputs

 Bus 1 --> 0000:00:14.0 (IOMMU group 5)
 Bus 001 Device 004: ID 06cb:00bd Synaptics, Inc. 
 Bus 001 Device 003: ID 13d3:56bc IMC Networks Integrated Camera
 Bus 001 Device 006: ID 0930:6544 Toshiba Corp. TransMemory-Mini / Kingston DataTraveler 2.0 Stick
 Bus 001 Device 002: ID 046d:c534 Logitech, Inc. Unifying Receiver
 Bus 001 Device 005: ID 8087:0aaa Intel Corp. 
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    
 Bus 2 --> 0000:00:14.0 (IOMMU group 5)
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    
 Bus 3 --> 0000:3a:00.0 (IOMMU group 18)
 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    
 Bus 4 --> 0000:3a:00.0 (IOMMU group 18)
 Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

So Bus 1 and 2 come from PCI path 00:14.0(which means the Cannon Polint-LP controller in the previous step) and both in the same IOMMU group 5 while Bus 3 and 4 come from PCI path 3a:00.0(which means the JHL6240 controller in the previous step) and both in the same IOMMU group 18. Unfortunately, my Toshiba stick is using the same controller as that of my laptop’s built-in keyboard and mouse(the Synaptics) therefore I can not pass this controller.

Let’s say that I want to pass the other JHL6240 controller. Then I’ll have to check the IOMMU information, especially for group 18 since that’s where it belongs to. Use the 2nd command provided in section Usefull tools and commands, it outputs(trimmed)
```
 IOMMU group 18
         03:02.0 PCI bridge [0604]: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] [8086:15c0] (rev 01)
 [RESET]	3a:00.0 USB controller [0c03]: Intel Corporation JHL6240 Thunderbolt 3 USB 3.1 Controller (Low Power) [Alpine Ridge LP 2016] [8086:15c1] (rev 01)
 IOMMU group 5
         00:14.0 USB controller [0c03]: Intel Corporation Cannon Point-LP USB 3.1 xHCI Controller [8086:9ded] (rev 30)
         00:14.2 RAM memory [0500]: Intel Corporation Cannon Point-LP Shared SRAM [8086:9def] (rev 30)
```
So there are two devices in IOMMU group 18, the USB controller and the Thunderbolt bridge. Therefore I have to passthrough both of them to the guest system.

Also there is a [RESET] indicator of that USB controller which we want to passthrough. This is a good sign since some USB devices require this to work properly, e.g Elgato USB capture HD60S.
Once you’ve managed to identify the devices, go to virt-manager -> Add hardware -> PCI host device. Choose all the necesary device(s) and press finish.

GPU passthrough

This is a bit more complex than USB passhtrough, but you can find many useful guide over the internet, like Arch wiki about PCI passthrough via OVMF, A blog post about Fighting Error 43 and the Unraid Forum, especially those guides from Spaceinvader One.

The basic idea stills holds. Devices in the same IOMMU group should all be passed through and Devices on the same physical card should all be passed through.

NOTE: NEVER passthrough the graphic card which you’re using for your host system’s display.

Ideally, this should be as simple as clicking several times virt-manager ->Add hardware -> Host PCI devices. But should you encounter any problem, there are some possible solutioins:

Check agian for the IOMMU information, make sure it is not at an unisolated CPU-based PCIe slot

Load the vfio-pci driver into the device. For example this device

  IOMMU Group 13:
       06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
       06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}

with device ID 10de:13c2 and 10de:0fbb. You can

either set kernel parameter vfio-pci.ids in GRUB_CMDLINE_LINUX_DEFAULT of file /etc/default/grub like below
```
 GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt kvm.ignore_msrs=1 vfio-pci.ids=10de:13c2,10de:0fbb"
```
then run
```
 sudo update-grub
```
then reboot the system
or use modprob by creating a /etc/modprobe.d/vfio.conf and add the following line
```
 options vfio-pci ids=10de:13c2,10de:0fbb
```
then run
```
 sudo update-initramfs -u -k all
```
and reboot the system

You can verify the loaded driver by

  lspci -nnk -d 10de:13c2
  lspci -nnk -d 10de:0fbb

Fighting error code 43.

Edit the XML in <domain> -> <features>, add or edit the <hyperv> and <kvm> sections in order to make sure there are

<hyperv>
    <vendor_id state='on' value='0123456789ab'/>    # value could be any 12 character
</hyperv>
<kvm>
    <hidden state='on'/>
</kvm>

If you’re using QEMU 4.0(or higher) and Q35 chip, the flag ioapic driver='kvm needs to be added in the <features> section

 <features>
     <hyperv>
         <vendor_id state='on' value='0123456789ab'/>
     </hyperv>
     <kvm>
         <hidden state='on'/>
     </kvm>
     <ioapic driver='kvm'/>
 </features>

load the vbios, check Spaceinvader One’s unraid GPU passthrough guides for more details

GPU passthrough in a laptop

If I may quote kdkdkdk1 at reddit post:

This is the end boss of the last stage of nightmare mode in the game of GPU Passthrough.

One reason is that the dedicated graphic card in a laptop works differently from a PCIE graphics card in a desktop tower. In a desktop card, the graphical render computation are done and output through the graphics' own video output port. But in laptop, the renered images will be passed back to the internal GPU for displaying.

Some more details can be found in this blog post. NOTE: even though in this post it gets dGPU to work in the VM, there are still many works to do to ensure it functions properly.

passthrough KVM QEMU libvirt