VFIO/GPU Passthrough

Issues & Assistance
1

Has anyone successfully achieved GPU passthrough to work in Garuda? I've tried several different ways of doing it and have encountered a few issues. Notably, that the binding to the pci ids does not work with kernel parameters. I've only gotten it to work when I bind them manually.

Even with that working, I was still unable to actually get the passthrough to work. Got some help from Reddit'ers that know more about this than I do and they were unable to get it to work.

I'd be curious to know if there's something about the Garuda distro that requires modification of the Arch wiki entry on the subject.

Also tried looking glass, fwiw.

2

I have been working at this also with no success. I can get a VM with winblows 10 to recognize and install my Nvidia driver for an RTX 2060. Buy when I reboot the VM no driver is found.

I am on my phone so posting links is difficult. I will reply with some of my references tomorrow when I get a chance.

3

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

and

The second link is more of a step by step.

2 Likes
4

Here are some commands to go from the vfio to nvidia driver that work for me. It would be nice to not have to logout while switching drivers. I just created a script that I call. I have not been able to find the commands to go from Nvidia to vfio. Also I read but can't find the link that you need to release the vfio driver in between different VMs.

sudo modprobe -r vfio-pci
sudo modprobe nvidia_drm
qdbus org.kde.ksmserver /KSMServer logout 0 0 0
2 Likes
5

  

3 Likes
6

Yes, I saw that post @jonathon. Thanks for reposting. The goal here is to see if anyone else has successfully done it in Garuda, thereby negating the long and winding road of trying a completely different distro. I was also hoping someone with intimate knowledge of the distro would be able to answer the question directly. It’s still a good idea though, and one I can fallback to if others’ suggestions do not work.

That second link you posted @waeking looks promising. It has steps that the wiki does not. I’ll give it a shot this evening. Thanks for posting! [thanks @SGS for adding the link]

8

Hey, i got it to work on Arch but not on Garuda. Everytime i start the VM the screen goes black for 1 second and then puts me back into Garuda login screen.
Hope someone know how to fix it.

2 Likes
9

Not supported.

And welcome to the forums.

3 Likes
10

I think running a VM on Garuda is still supported, though?

3 Likes
11

I am pretty sure that running Garuda in a VM is not supported. However running Garuda on Hardware with VM’s running should be supported.

1 Like
12

Yeah, it's hard to tell from the Garuda install screen what "VM is not recommended" means but from what I've been able to gather, you should certainly be able to run a VM in Garuda just fine (otherwise, this distro would die a quick death). I believe they mean that Garuda is not meant to be installed in a VM.

1 Like
13

After several days I still do not have a working passthrough. In hopes of getting more eyes on the problem, I've published all the outputs I have so far. If anyone can provide some insight or forward this to someone who can, I'd be grateful.

If this is not helpful, I'm happy to provide more helpful information, even do another video walkthrough of what I've done in these steps.

One possibly important note is that I tried to get passthrough to work on the same machine with PopOS and was also unable to get it working. I double checked my mobo and card and I think it should all be fine, though. So maybe it's just human error?

HARDWARE

Mobo: ASUS Tuf Gaming B550M Plus
CPU: AMD Ryzen 5 3600
GPU, primary (used for Garuda): Radeon Pro WX 3100
GPU, secondary (for use w/ VM): Radeon Pro WX 3200

Primary GPU -> HDMI KVM -> monitor
Secondary GPU -> monitor's second input, but not displayed

OUTPUTS/ SETTINGS

Kernel params:

rd.driver.pre=vfio_pci amd_iommu=on iommu=pt vfio-pci.ids=1002:6981 quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3

(^ set using the Gardua Boot Options GUI, but I also tried doing it in the GRUB file at /etc/default/grub)

dmesg | grep -i -e DMAR -e IOMMU

[    0.000000] Command line: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=c167903f-e144-4c5c-b565-2f3158f5ef30 rw rootflags=subvol=@ rd.driver.pre=vfio_pci amd_iommu=on iomm =pt quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3
[    0.053635] Kernel command line: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=c167903f-e144-4c5c-b565-2f3158f5ef30 rw rootflags=subvol=@ rd.driver.pre=vfio_pci amd_iommu=on iommu=pt quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3
[    0.283203] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.476616] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.476676] pci 0000:00:01.0: Adding to iommu group 0
[    0.476686] pci 0000:00:01.2: Adding to iommu group 1
...
[    1.984681] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x1001f2680]
[    1.984684] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x1001f26a0]
[    1.985019] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    2.480867] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x1001f26e0]
[    3.482718] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x1001f2720]
[    4.484637] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x1001f2760]
[    4.490212] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <[email protected]>

^ NOTE there is no mention of DMAR

IOMMU groups script output. Group 13 is where my primary GPU is, Group 14 for passthrough):

IOMMU Group 13:
	01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
	01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
	01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
	02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3100] [1002:6985]
	03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
	04:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
	05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
IOMMU Group 14:
	06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] [1002:6981] (rev 10)
	06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:6981

/etc/mkinitcpio.conf (Github gist)

(no dracut b/c dracut command does not exist in Garuda)

dmesg | grep -i vfio

[    0.000000] Command line: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=c167903f-e144-4c5c-b565-2f3158f5ef30 rw rootflags=subvol=@ rd.driver.pre=vfio_pci amd_iommu=on iommu=pt quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3
[    0.000000] Kernel command line: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=c167903f-e144-4c5c-b565-2f3158f5ef30 rw rootflags=subvol=@ rd.driver.pre=vfio_pci amd_iommu=on iommu=pt quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3
[    7.730980] VFIO - User Level meta-driver version: 0.3
[    7.734151] vfio_pci: add [1002:6981[ffffffff:ffffffff]] class 0x000000/00000000
[    7.758021] vfio_pci: add [1002:aae0[ffffffff:ffffffff]] class 0x000000/00000000

lspci -nnk -d 1002:6981 (2nd gpu for passthrough, WX3200)

06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] [1002:6981] (rev 10)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0b0d]
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

^ NOTE that the "Kernel driver in use is NOT vfio-pci like we'd expect at this point"

So I attempt to manually bind the devices by running the following commands as root:
echo 0000:06:00.0 > /sys/bus/pci/drivers/amdgpu/unbind
echo 0000:06:00.0 > /sys/bus/pci/drivers/vfio-pci/bind

And then run lspci -nnk -d 1002:6981 again:

6:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] [1002:6981] (rev 10)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0b0d]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu

^ Hallelujah.

Now onto configuring libvirt...

virsh network file: (Github gist)

Guest VM XML (Github gist)

I fire up the VM, install the install the AMD drivers from the WX3200, install virtio drivers, update the PCI RAM drivers, and install the Looking Glass application for the guest. Shutdown.

Then as instructed in the Looking Glass instructions, I turn the video model to "none" and start up the VM, then start the Looking Glass host application. And I get nothing.

NOTES

I have not done any of the "tuning" steps such as CPU pinning

I also tried binding the devices with QEMU hooks, using this.

14

Can you try without looking glass to see if the drivers take on your VM after a reboot. It seems that you and I are right about at the same place. I have not installed looking glass.

15

I can try that. Although I think I did already, I can't remember after rebooting so many times :slight_smile: One more can't hurt. BTW how would you confirm that the passthrough is working in the guest OS? In device manager? I've been using a tool inside Fusion360 that tells you how much video memory is available but I'm sure there's an OS level tool for that, I just don't know it since I never use Windows (migrating to Linux from Mac).

16

I installed the Nvidia Drivers on my windows VM. Then looked at the device manager and saw that the driver was installed correctly for my video card. However after I reboot the VM, the Video Card is not recognized and therefore no driver has been installed.

17

No dice. I'm giving up for now. I have to move on with my (dual boot) life :confused:

Keep up the good fight, my friends.

If, however, there is someone reading this who thinks they can help me and needs motivation in the form of payment (PayPal, Bitcoin, silk and spices, whatever), that is something I would be interested in. I would like to learn how it works, too and why it wasn't working for me. Name your price.

1 Like
18

@CaptainMangar would you be able to give a little walk through of what you did in Arch? Perhaps there are some steps that might help others in the future.

1 Like
19

I found another link Arch GPU passthrough · GitHub this one has newer information and a whole bunch more commands. Any Takers?

I don't know what edk2 is needed for.

20

I have a working Windows 10 VM. Followed Bryan Steiner's guide and Arch wiki. It was pretty easy considering my newbieness in Linux. It was super easy the 2nd time around.

As I recall I also managed to try Garuda in a VM with passthrough. That was when I decided to switch to Garuda.

21

Following this guide you got GPU passthrough to work to a windows 10 VM with garuda running on the bare metal?