KVM, QEMU, and SOG

[Blu]

Hello Garuda!

Before saying anything, I'll do the traditional dump of my inxi -Faz as shown below:

System:    Kernel: 5.14.14-zen1-1-zen x86_64 bits: 64 compiler: gcc v: 11.1.0
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=f6f45d85-b66c-48c7-ae2e-becddd3d4d46
rw rootflags=subvol=@ splash amd_iommu=on rd.udev.log_priority=3 vt.global_cursor_default=0
systemd.unified_cgroup_hierarchy=1 loglevel=3
Desktop: KDE Plasma 5.23.2 tk: Qt 5.15.2 info: latte-dock wm: kwin_x11 vt: 1 dm: SDDM
Distro: Garuda Linux base: Arch Linux
Machine:   Type: Desktop Mobo: ASUSTeK model: PRIME X570-PRO v: Rev X.0x serial: <filter>
UEFI: American Megatrends v: 1407 date: 04/02/2020
CPU:       Info: 16-Core (2-Die) model: AMD Ryzen 9 3950X bits: 64 type: MT MCP MCM arch: Zen 2
family: 17 (23) model-id: 71 (113) stepping: 0 microcode: 8701013 cache: L2: 8 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 230395
Speed: 3601 MHz min/max: 2200/3600 MHz boost: enabled Core speeds (MHz): 1: 3601 2: 3599
3: 3620 4: 3595 5: 3602 6: 3600 7: 3612 8: 3599 9: 3602 10: 3615 11: 3625 12: 3611 13: 3599
14: 3593 15: 3598 16: 3614 17: 3607 18: 3617 19: 3608 20: 3624 21: 3599 22: 3599 23: 3599
24: 3594 25: 3602 26: 3599 27: 3601 28: 3592 29: 3598 30: 3603 31: 3600 32: 3612
Vulnerabilities: Type: itlb_multihit status: Not affected
Type: l1tf status: Not affected
Type: mds status: Not affected
Type: meltdown status: Not affected
Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via prctl and seccomp
Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2
mitigation: Full AMD retpoline, IBPB: conditional, STIBP: conditional, RSB filling
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Graphics:  Device-1: NVIDIA GM204 [GeForce GTX 970] vendor: eVga.com. driver: nvidia v: 470.74
alternate: nouveau,nvidia_drm bus-ID: 08:00.0 chip-ID: 10de:13c2 class-ID: 0300
Display: x11 server: X.Org 1.20.13 compositor: kwin_x11 driver: loaded: nvidia display-ID: :0
screens: 1
Screen-1: 0 s-res: 1920x1080 s-dpi: 92 s-size: 530x301mm (20.9x11.9") s-diag: 610mm (24")
Monitor-1: DVI-D-0 res: 1920x1080 hz: 60 dpi: 92 size: 531x299mm (20.9x11.8") diag: 609mm (24")
OpenGL: renderer: NVIDIA GeForce GTX 970/PCIe/SSE2 v: 4.6.0 NVIDIA 470.74 direct render: Yes
Audio:     Device-1: NVIDIA GM204 High Definition Audio vendor: eVga.com. driver: snd_hda_intel v: kernel
bus-ID: 08:00.1 chip-ID: 10de:0fbb class-ID: 0403
Device-2: AMD Starship/Matisse HD Audio vendor: ASUSTeK driver: snd_hda_intel v: kernel
bus-ID: 0a:00.4 chip-ID: 1022:1487 class-ID: 0403
Device-3: SteelSeries ApS SteelSeries Arctis 5 type: USB
driver: hid-generic,snd-usb-audio,usbhid bus-ID: 1-2:2 chip-ID: 1038:12aa class-ID: 0300
serial: <filter>
Device-4: Licensed by Sony Entertainment America Rocksmith Guitar Adapter type: USB
driver: hid-generic,snd-usb-audio,usbhid bus-ID: 3-2:2 chip-ID: 12ba:00ff class-ID: 0300
Sound Server-1: ALSA v: k5.14.14-zen1-1-zen running: yes
Sound Server-2: JACK v: 1.9.19 running: no
Sound Server-3: PulseAudio v: 15.0 running: no
Sound Server-4: PipeWire v: 0.3.39 running: yes
Network:   Device-1: Intel I211 Gigabit Network vendor: ASUSTeK driver: igb v: kernel port: f000
bus-ID: 04:00.0 chip-ID: 8086:1539 class-ID: 0200
IF: enp4s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
IF-ID-1: virbr0 state: down mac: <filter>
Drives:    Local Storage: total: 5.91 TiB used: 3.38 TiB (57.2%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Samsung model: SSD 970 EVO Plus 250GB
size: 232.89 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD
serial: <filter> rev: 2B2QEXM7 temp: 50.9 C scheme: GPT
ID-2: /dev/sda maj-min: 8:0 vendor: Samsung model: SSD 860 EVO 250GB size: 232.89 GiB
block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s type: SSD serial: <filter> rev: 4B6Q
scheme: MBR
ID-3: /dev/sdb maj-min: 8:16 vendor: Seagate model: ST6000VN0033-2EE110 size: 5.46 TiB
block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s type: HDD rpm: 7200
serial: <filter> rev: SC60 scheme: GPT
Partition: ID-1: / raw-size: 232.63 GiB size: 232.63 GiB (100.00%) used: 33.5 GiB (14.4%) fs: btrfs
dev: /dev/nvme0n1p2 maj-min: 259:2
ID-2: /boot/efi raw-size: 260 MiB size: 256 MiB (98.46%) used: 562 KiB (0.2%) fs: vfat
dev: /dev/nvme0n1p1 maj-min: 259:1
ID-3: /home raw-size: 232.63 GiB size: 232.63 GiB (100.00%) used: 33.5 GiB (14.4%) fs: btrfs
dev: /dev/nvme0n1p2 maj-min: 259:2
ID-4: /var/log raw-size: 232.63 GiB size: 232.63 GiB (100.00%) used: 33.5 GiB (14.4%) fs: btrfs
dev: /dev/nvme0n1p2 maj-min: 259:2
ID-5: /var/tmp raw-size: 232.63 GiB size: 232.63 GiB (100.00%) used: 33.5 GiB (14.4%) fs: btrfs
dev: /dev/nvme0n1p2 maj-min: 259:2
Swap:      Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
ID-1: swap-1 type: zram size: 31.27 GiB used: 0 KiB (0.0%) priority: 100 dev: /dev/zram0
Sensors:   System Temperatures: cpu: 56.9 C mobo: N/A gpu: nvidia temp: 63 C
Fan Speeds (RPM): N/A gpu: nvidia fan: 9%
Info:      Processes: 520 Uptime: 11h 34m wakeups: 0 Memory: 31.27 GiB used: 4.95 GiB (15.8%)
Init: systemd v: 249 tool: systemctl Compilers: gcc: 11.1.0 Packages: pacman: 1220 lib: 303
Shell: fish v: 3.3.1 default: Bash v: 5.1.8 running-in: konsole inxi: 3.3.08

Now, cutting to the chase.

I am new to Garuda and have a little bit of Linux experience but I'm not crusty at it. I'm running Garuda now as my main operating system and I quite like it but wanted to do some virtualized gaming, so I decided to follow the guide of my favorite "fat indian" of the internet as he refers to himself SomeOrdinaryGamers. Mutahar went through the process in the video here:
Mutahar's GPU Passthrough/Hijacking Tutorial

I went through the process and did everything needed but now I've hit a bit of a snag. Near the middle bit before I decide to muck about with the VM the scripts required to hijack the GPU seem to be working a little bit funky in comparison to what he's got there, though he does state that it's to your taste pertaining to which distro you're using. I've narrowed down that the service that I'm supposed to yank is sddm-plymouth.service with by writing in the shell script: "systemctl stop sddm-plymouth.service", where as in his script he's just got the basic sddm.service he yanks.

The problem I'm facing is when I go ahead and stop the plymouth service and proceed with the script, it hangs with the messages:

:: running early hook [udev]
:: running early hook [plymouth]

I have to then force reboot but the screen doesn't go black like I'm expecting it to. Especially when running the VM for ■■■■■■■, it's not passing through my GPU and instead just hanging on this screen and nothing else on it. To put in context what the script is doing so nobody has to go sifting through the video to see what I've followed, here are the two scripts I've got copied over:

start.sh

# debugging
set -x

# load variables we defined
source "/etc/libvirt/hooks/kvm.conf"

# stop display manager
systemctl stop sddm-plymouth.service

# unbind vtconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind

# unbind EFI-Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# avoid race condition - can be whatever above 5
sleep 10

# unload nvidia drivers
modprobe -r nvidia_drm
modprobe -r nvidia_modeset
modprobe -r drm_kms_helper
modprobe -r nvidia
modprobe -r ic2_nvidia_gpu
modprobe -r drm
modprobe -r nvidia_uvm

# unbind gpu
virsh nodedev-detach $VIRSH_GPU_VIDEO
virsh nodedev-detach $VIRSH_GPU_AUDIO

# load vfio
modprobe vfio
modprobe vfio_pci
modprobe vfio_iommu_type1

revert.sh

# debugging
set -x

# unload vfio-pci
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

# rebind gpu
virsh nodedev-reattach $VIRSH_GPU_VIDEO
virsh nodedev-reattach $VIRSH_GPU_AUDIO

# rebind vtconsoles
echo 1 > /sys/class/vtconsole/vtcon0/bind

# read nvidia x config
nvidia-xconfig --query-gpu-info > /dev/null 2>&1

# bind efi-framebuffer
echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind

# load nvidia
modprobe nvidia_drm
modprobe nvidia_modeset
modprobe drm_kms_helper
modprobe nvidia
modprobe drm
modprobe nvidia_uvm

# restart display service
systemctl start sddm-plymouth.service

The kvm.conf file referenced is just simply:

VIRSH_GPU_VIDEO=pci_0000_08_00_0
VIRSH_GPU_AUDIO=pci_0000_08_00_1

I then use a patched gpu rom in the further movement through setting of this up, but I haven't gotten to get to that stage yet due to this snag seemingly getting in the way.

I imagine this is a bit beyond the scope of what people might offer but I thought I'd try and see if there might be anyone who knows what I'm doing wrong here and can help. If not, no worries, I'll just figure something else out, but so far it's been fun tinkering.

Thanks for the read!

[Naman]

Hi there, welcome to the community.

We clearly don't have time enough to watch a complete 40mins+ video, follow the guide and tell where it went wrong.

But, I have a shortcut for you, if all you want to do is to setup VM. Just go to Garuda setup assistant and check you favorite vm there.
Or are you trying to accomplish something else?

[tbg]

Several users have reported bugs with their VM’s. Changing a boot param could help:

KVM problem to start virtual machines

This thread is 6 months old and is probably totally unrelated: However, it does have some ideas not proposed before. One is to add this line to the kernel boot parameters: systemd.unified_cgroup_hierarchy=0 Then: grub-mkconfig -o /boot/grub/grub.cfg or: update-grub Then reboot. The Garuda default is: systemd.unified_cgroup_hierarchy=1 Just thought I'd mention this post as it was preventing the startup of the VM's. VM's are not my thing, so I'm just throwing this out there on the remo…

[Blu]

From what I've seen for the VM tutorials, it's typically not through the process of single GPU passthrough. What I'm wanting to do here is unload everything Garuda has captured for my GPU and then passthrough the loaded drivers for my GPU to the VM so I'm able to make windows the base application that's utilising my GPU, not sharing it with Garuda and the like as the framerates will tank through the floor if I try to double run it.

Unless I missed something, from what I've read and seen, they're about running two GPUs. One for Garuda, one of the VM.

As for what went wrong with the guide, my screen doesn't go black, it doesn't wait for the VM to hijack my GPU, and then I don't full screen Windows like I'm supposed to. When running the start.sh script, I should see nothing but instead I see the message I've stated above about it hanging.

[Blu]

Interesting. I'll give it a whirl and see where it takes me. Thanks for the heads-up! Hopefully that's my holy grail.

[His_Turdness]

I recommend you check out Bryan Steiner's GPU passthrough guide.

I've gotten my setup to work with his guide.

Note that I'm using an iGPU and the dGPU which I'm passing through and I'm also using optimus manager to enable/disable the nvidia drivers under Garuda.

[Blu]

I'll take a look at it. So far the source guide from SOG's tutorial is essentially this and it's seemingly not working out of the box. Garuda seems to have some fancy stuff that's making it not want to play nice, so I'm thinking I'll try that, then see if I can shoot this person a message and figure out what is wrong. Thanks for the tip!

[Blu]

Alright, update on what's happening and what's wrong now. I went to reddit to see if I could figure something out with some extra eyes and I'm unsure what to do next and this one's strange. I figure it might be the way Garuda's packages are handling Nvidia's drivers?

The script is set up like this:

#!/bin/bash
set -x

# stop display manager
systemctl stop display-manager

# unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

# Unbind EFI-framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# avoid race condition
sleep 10

# unload nvidia
rmmod -f nvidia_drm # 1
rmmod -f nvidia_uvm # 2
rmmod -f nvidia_modeset # used by nvidia_drm
rmmod -f drm_kms_helper # used by nvidia_drm
rmmod -f nvidia # used by nvidia_uvm, nvidia_modeset
rmmod -f drm # used by drm_kms_helper, nvidia, nvidia_drm

# unbind gpu
virsh nodedev-detach pci_0000_08_00_0
virsh nodedev-detach pci_0000_08_00_1

# load vfio
modprobe vfio
modprobe vfio_pci
modprobe vfio_iommu_type1

Now the issue of what's happened, and what's going on is the unloading of nvidia drivers. I tried initially running modprobe -r for the modules but it kept giving me "Module nvidia_drm is in use." and likewise for every other module down the list. The only way I could unload them definitively is by doing rmmod -f as it very much makes sure they're unloaded, but when it gets down to rmmod -f nvidia, it hangs. Doesn't do anything anymore from there on. Now, this is through Termux on my phone running openSSH, but even running the VM results in not much.

I don't believe I'm missing anything to unload and especially in any particular sequence as it seems to be all accounted for concerning an lsmod | grep nvidia output like this:

nvidia_uvm           1187840  0
nvidia_drm             73728  9
nvidia_modeset       1155072  21 nvidia_drm
nvidia              36966400  1398 nvidia_uvm,nvidia_modeset
drm_kms_helper        331776  1 nvidia_drm
drm                   634880  13 drm_kms_helper,nvidia,nvidia_drm

Not sure what to do from here. Any help in this progress would be greatly appreciated.

[Pheidologeton]

Plymouth is very, very buggy. I had such a problem. Look at this Plymouth boot screen bug , there will be necessary to edit Mkinitcpio and rebuild the kernel image.

[tbg]

My home is very buggy. It is loaded with spiders and ants. Could someone please come to my house and stomp out these bugs for me for free?

Failing that, does anyone know where I should report this problem upstream?

[Blu]

I didn't know you could run sddm like that, so I've done so and it's changed the process a little bit, but I'm still facing the nvidia issue of the modules not allowing me to unload them. It's changed a little bit in the output, but I still can't run a modprobe -r to unload them. I need to use rmmod -f for any luck with it, and only some of them I can do that with. I get an output like this which still poses a problem for me:

+ rmmod -f nvidia_urm
rmmod: ERROR: could not remove 'nvidia_urm': No such file or directory
rmmod: ERROR: could not remove module nvidia_urm: no such file or directory
+ rmmod -f nvidia_drm
+ rmmod -f nvidia_modeset
+ rmmod -f nvidia
rmmod: ERROR: could not remove 'nvidia': Resource temporarily unavailable
rmmod: ERROR: could not remove module nvidia: Resource temporarily unavailable
+ rmmod -f drm_kms_helper
+ rmmod -f drm
rmmod: ERROR: could not remove 'drm': Resource temporarily unavailable
rmmod: ERROR: could not remove module drm: Resource temporarily unavailable
+ virsh nodedev-detach pci_0000_08_00_0
...

And from there it hangs due to not being able to unload the modules that are using the hardware at that address which is my GPU in IOMMU group 23. It hasn't gotten far enough to make it to the audio hardware in the same grouping.

I feel like I'm doing something wrong here but I don't know what.

[His_Turdness]

I've had problems with plymouth as well. Another version fixes it, most times.

[Blu]

Success! Finally!

Okay, so I finally managed to get Nvidia to work with me here. When running an lsmod | grep nvidia, it shows all the modules that are dependant but they don't list any processes that are also working with it. I discovered nvidia has a nice little tool called nvidia-smi which lists all processes running that are accessing the GPU. After running the initial script that kills off the modules and getting stuck at the GPU unload, I broke out of the script and decided to see if it would tell me anything and that functionality still existed. The culpret? Plasmashell.

When you run a systemctl stop display-mananger, it unloads everything as expected that would operate for both plymouth and just generic sddm, but what it doesn't do is unload Plasmashell's operation on it which needs to be killed manually. The fix is to apply pkill plasmashell to the script after stopping display-manager, and that frees everything up enough to unload all the nvidia modules and allow for the pass-over into the vfio modules which can be used by the VM.

tl;dr: this whole setup works with Garuda if you pkill plasmashell in part of the script as it's still processing whatever regardless of unloading display-manager and the like.

Now I'm hype.

Thing to note, it looks like when you start display-manager, it runs plasmashell and all the particular goodies in the background, but when you stop display-manager, plasmashell isn't included in the processes to kill for some reason. I don't know if it was missed in the configuration, but it doesn't seem to have that effect, though I haven't looked into it to particularly verify, but that's what I get from how it seems to be functioning.

[His_Turdness]

So basically you need to restart your desktop session/reload the display manager, correct?

That's why I like otpimus manager, it does just that as you switch between dGPU and iGPU.

[Blu]

More or less. Stop the display manager, plasmashell, unbind vtconsoles, efi-framebuffer, nvidia modules, Video and Audio pci, and then load vfio. I donno but I quite like this setup now that I can get it working. I'm unfimilar with Optimus Manager, though I don't have integrated graphics with my CPU. It's all dedicated, single gpu passthrough. Lets go!

[His_Turdness]

Everything is much harder with just a single GPU. Glad you got it working.