Cat
28 March 2022 22:01
1
Hello.
I have been struggling with setting up GPU passthrough with QEMU/KVM for the past few days now, with no end in sight. Everything works, except the GPU passthrough, I get the classic Code 43 code there, being stuck in 800x600.
For context, I have followed SOG's "tutorial" on how to do his scripts and passthroughs, with additional help from the Arch wiki.
It seems to me that whatever I do, I'm not able to get my GPU to passthrough without getting code 43 on Windows 10, hopefully someone more versed can help me with this.
I use a R5 1600 and Zotac 1070ti mini.
I have attempted the following:
This KVM yoke in the VM XML conf.
<hyperv mode="custom">
...
<vendor_id state="on" value="garfield cat"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
Gpu reservation errors, and after trying the commands, I get the following.
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
...
echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove
echo 1 > /sys/bus/pci/rescan
...
""/""
Most interestingly, the pre-dumped vBIOS for my GPU was about 4 times as big my own vBIOS dumped from my GPU, not to mention that the beginnings of their roms were different.
For some reason, my graphic card's vBIOS did not want to display anything on the screen, and the pre-dumped worked just fine.
My 1070ti's rom:
U.s.K7400.L.w.VIDEO
Pre-dumped 1070ti's rom:
U.y.K7400.L.w.VIDEO
Any help is greatly appreciated, I am at a loss for knowledge and ideas.
Boot parameters:
splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 pcie_acs_override=downstream amd_iommu=on iommu=pt pci=realloc video=efifb:off
Please follow the template and show your garuda-inxi
5 Likes
Cat
28 March 2022 22:11
3
Sorry, here it is.
ystem:
Kernel: 5.16.16-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 11.2.0
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
root=UUID=aa8e138c-42cc-42e0-bdfd-8790fdeccf77 rw [email protected]
splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
pcie_acs_override=downstream amd_iommu=on pci=realloc
Desktop: Xfce v: 4.16.0 tk: Gtk v: 3.24.29 info: xfce4-panel wm: xfwm
v: 4.16.1 vt: 7 dm: LightDM v: 1.30.0 Distro: Garuda Linux base: Arch Linux
Machine:
Type: Desktop Mobo: ASRock model: AB350M Pro4 serial: <filter>
UEFI-[Legacy]: American Megatrends v: P2.30 date: 04/10/2017
CPU:
Info: model: AMD Ryzen 5 1600 socket: AM4 bits: 64 type: MT MCP arch: Zen
family: 0x17 (23) model-id: 1 stepping: 1 microcode: 0x800111C
Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
L1: 576 KiB desc: d-6x32 KiB; i-6x64 KiB L2: 3 MiB desc: 6x512 KiB
L3: 16 MiB desc: 2x8 MiB
Speed (MHz): avg: 1895 high: 3200 min/max: 1550/3200 boost: enabled
base/boost: 3200/3700 scaling: driver: acpi-cpufreq governor: schedutil
volts: 1.2 V ext-clock: 100 MHz cores: 1: 2800 2: 1550 3: 1550 4: 1550
5: 3200 6: 1550 7: 1550 8: 1550 9: 1550 10: 1550 11: 2800 12: 1550
bogomips: 76650
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Vulnerabilities:
Type: itlb_multihit status: Not affected
Type: l1tf status: Not affected
Type: mds status: Not affected
Type: meltdown status: Not affected
Type: spec_store_bypass
mitigation: Speculative Store Bypass disabled via prctl
Type: spectre_v1
mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2 mitigation: Retpolines, STIBP: disabled, RSB filling
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Graphics:
Device-1: NVIDIA GP104 [GeForce GTX 1070 Ti] vendor: ZOTAC driver: nvidia
v: 510.54 alternate: nouveau,nvidia_drm pcie: gen: 3 speed: 8 GT/s
lanes: 16 bus-ID: 09:00.0 chip-ID: 10de:1b82 class-ID: 0300
Device-2: Oculus VR Rift CV1 Sensor type: USB driver: uvcvideo
bus-ID: 4-1:2 chip-ID: 2833:0211 class-ID: ef00 serial: <filter>
Device-3: Oculus VR Rift CV1 Sensor type: USB driver: uvcvideo
bus-ID: 4-2:3 chip-ID: 2833:0211 class-ID: ef00 serial: <filter>
Display: x11 server: X.Org v: 1.21.1.3 compositor: xfwm v: 4.16.1 driver:
X: loaded: nvidia gpu: nvidia display-ID: :0.0 screens: 1
Screen-1: 0 s-res: 3840x1080 s-dpi: 92 s-size: 1060x301mm (41.73x11.85")
s-diag: 1102mm (43.38")
Monitor-1: DVI-D-0 pos: primary,left res: 1920x1080 dpi: 92
size: 531x298mm (20.91x11.73") diag: 609mm (23.97") modes: N/A
Monitor-2: HDMI-0 pos: right res: 1920x1080 hz: 60 dpi: 102
size: 477x268mm (18.78x10.55") diag: 547mm (21.54") modes: N/A
Message: Unable to show GL data. Required tool glxinfo missing.
Audio:
Device-1: NVIDIA GP104 High Definition Audio vendor: ZOTAC
driver: snd_hda_intel v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16
bus-ID: 09:00.1 chip-ID: 10de:10f0 class-ID: 0403
Device-2: AMD Family 17h HD Audio vendor: ASRock driver: snd_hda_intel
v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16 bus-ID: 12:00.3
chip-ID: 1022:1457 class-ID: 0403
Device-3: C-Media Blue Snowball type: USB
driver: hid-generic,snd-usb-audio,usbhid bus-ID: 3-4:2 chip-ID: 0d8c:0005
class-ID: 0300 serial: <filter>
Sound Server-1: ALSA v: k5.16.16-zen1-1-zen running: yes
Sound Server-2: PulseAudio v: 15.0 running: no
Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
vendor: ASRock driver: r8169 v: kernel pcie: gen: 1 speed: 2.5 GT/s
lanes: 1 port: f000 bus-ID: 06:00.0 chip-ID: 10ec:8168 class-ID: 0200
IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
IF-ID-1: virbr0 state: down mac: <filter>
Drives:
Local Storage: total: 1.36 TiB used: 53.69 GiB (3.8%)
SMART Message: Required tool smartctl not installed. Check --recommends
ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Kingston model: SA2000M8500G
size: 465.76 GiB block-size: physical: 512 B logical: 512 B
speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: S5Z42105
temp: 34.9 C scheme: MBR
ID-2: /dev/sda maj-min: 8:0 vendor: Seagate model: ST1000DM010-2DM162
size: 931.51 GiB block-size: physical: 4096 B logical: 512 B
speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter> rev: CC46
scheme: MBR
Partition:
ID-1: / raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
ID-2: /home raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
ID-3: /var/log raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
ID-4: /var/tmp raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
Swap:
Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
ID-1: swap-1 type: zram size: 15.64 GiB used: 0 KiB (0.0%) priority: 100
dev: /dev/zram0
Sensors:
System Temperatures: cpu: N/A mobo: N/A gpu: nvidia temp: 40 C
Fan Speeds (RPM): N/A gpu: nvidia fan: 39%
Info:
Processes: 339 Uptime: 49m wakeups: 0 Memory: 15.64 GiB
used: 2.32 GiB (14.8%) Init: systemd v: 250 tool: systemctl Compilers:
gcc: 11.2.0 Packages: pacman: 1426 lib: 440 Shell: Bash v: 5.1.16
running-in: xfce4-terminal inxi: 3.3.14
Garuda (2.5.6-2):
System install date: 2022-03-25
Last full system update: 2022-03-28
Is partially upgraded: No
Relevant software: NetworkManager
Windows dual boot: No/Undetected
Snapshots: Snapper
Failed units: bluetooth-autoconnect.service
Cat:
UEFI-[Legacy]
Did you install in legacy boot mode on purpose?
Edit: Disregard, now that I look into it I think it should be fine with or without EFI.
Some BIOS menus have a VT-d/IOMMU option you have to turn on, did you already check?
Edit 2: You are also several BIOS updates behind. This could be a factor. ASRock > AB350M Pro4
4 Likes
Cat
29 March 2022 08:58
5
All virtualization and iommu options are enabled.
I will attempt the BIOS Flash now.
Cat
29 March 2022 10:39
6
Updated BIOS, ver 2.3 > 3.4 > 5.1
Booted into VM after adjusting options to the new ids
I still get error 43
garuda-inxi output:
System:
Kernel: 5.16.16-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 11.2.0
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
root=UUID=aa8e138c-42cc-42e0-bdfd-8790fdeccf77 rw [email protected]
splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
pcie_acs_override=downstream amd_iommu=on iommu=pt pci=realloc
video=efifb:off
Desktop: Xfce v: 4.16.0 tk: Gtk v: 3.24.29 info: xfce4-panel wm: xfwm
v: 4.16.1 dm: LightDM v: 1.30.0 Distro: Garuda Linux base: Arch Linux
Machine:
Type: Desktop Mobo: ASRock model: AB350M Pro4 serial: <filter>
UEFI-[Legacy]: American Megatrends v: P5.10 date: 10/05/2018
CPU:
Info: model: AMD Ryzen 5 1600 socket: AM4 bits: 64 type: MT MCP arch: Zen
family: 0x17 (23) model-id: 1 stepping: 1 microcode: 0x8001137
Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
L1: 576 KiB desc: d-6x32 KiB; i-6x64 KiB L2: 3 MiB desc: 6x512 KiB
L3: 16 MiB desc: 2x8 MiB
Speed (MHz): avg: 1687 high: 3200 min/max: 1550/3200 boost: enabled
base/boost: 3200/3700 scaling: driver: acpi-cpufreq governor: schedutil
volts: 1.2 V ext-clock: 100 MHz cores: 1: 1550 2: 1550 3: 1550 4: 1550
5: 1550 6: 1550 7: 3200 8: 1550 9: 1550 10: 1550 11: 1550 12: 1550
bogomips: 76647
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Vulnerabilities:
Type: itlb_multihit status: Not affected
Type: l1tf status: Not affected
Type: mds status: Not affected
Type: meltdown status: Not affected
Type: spec_store_bypass
mitigation: Speculative Store Bypass disabled via prctl
Type: spectre_v1
mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2
mitigation: Retpolines, IBPB: conditional, STIBP: disabled, RSB filling
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Graphics:
Device-1: NVIDIA GP104 [GeForce GTX 1070 Ti] vendor: ZOTAC driver: N/A
alternate: nouveau, nvidia_drm, nvidia pcie: gen: 3 speed: 8 GT/s lanes: 16
bus-ID: 23:00.0 chip-ID: 10de:1b82 class-ID: 0300
Device-2: Oculus VR Rift CV1 Sensor type: USB driver: uvcvideo
bus-ID: 4-2:2 chip-ID: 2833:0211 class-ID: ef00 serial: <filter>
Display: server: X.Org v: 1.21.1.3 compositor: xfwm v: 4.16.1 driver: X:
loaded: nvidia gpu: N/A display-ID: :0.0 screens: 1
Screen-1: 0 s-res: 3840x1080 s-dpi: 92 s-size: 1060x301mm (41.73x11.85")
s-diag: 1102mm (43.38")
Monitor-1: DVI-D-0 pos: primary,left res: 1920x1080 dpi: 92
size: 531x298mm (20.91x11.73") diag: 609mm (23.97") modes: N/A
Monitor-2: HDMI-0 pos: right res: 1920x1080 hz: 60 dpi: 102
size: 477x268mm (18.78x10.55") diag: 547mm (21.54") modes: N/A
Message: Unable to show GL data. Required tool glxinfo missing.
Audio:
Device-1: NVIDIA GP104 High Definition Audio vendor: ZOTAC driver: N/A
alternate: snd_hda_intel pcie: gen: 3 speed: 8 GT/s lanes: 16
bus-ID: 23:00.1 chip-ID: 10de:10f0 class-ID: 0403
Device-2: AMD Family 17h HD Audio vendor: ASRock driver: snd_hda_intel
v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16 bus-ID: 25:00.3
chip-ID: 1022:1457 class-ID: 0403
Device-3: C-Media Blue Snowball type: USB
driver: hid-generic,snd-usb-audio,usbhid bus-ID: 3-4:2 chip-ID: 0d8c:0005
class-ID: 0300 serial: <filter>
Sound Server-1: ALSA v: k5.16.16-zen1-1-zen running: yes
Sound Server-2: PulseAudio v: 15.0 running: no
Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
vendor: ASRock driver: r8169 v: kernel pcie: gen: 1 speed: 2.5 GT/s
lanes: 1 port: f000 bus-ID: 1f:00.0 chip-ID: 10ec:8168 class-ID: 0200
IF: enp31s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
IF-ID-1: virbr0 state: down mac: <filter>
Drives:
Local Storage: total: 1.36 TiB used: 61.46 GiB (4.4%)
SMART Message: Required tool smartctl not installed. Check --recommends
ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Kingston model: SA2000M8500G
size: 465.76 GiB block-size: physical: 512 B logical: 512 B
speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: S5Z42105
temp: 35.9 C scheme: MBR
ID-2: /dev/sda maj-min: 8:0 vendor: Seagate model: ST1000DM010-2DM162
size: 931.51 GiB block-size: physical: 4096 B logical: 512 B
speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter> rev: CC46
scheme: MBR
Partition:
ID-1: / raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
ID-2: /home raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
ID-3: /var/log raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
ID-4: /var/tmp raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
maj-min: 259:1
Swap:
Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
ID-1: swap-1 type: zram size: 15.56 GiB used: 1024 KiB (0.0%)
priority: 100 dev: /dev/zram0
Sensors:
System Temperatures: cpu: N/A mobo: N/A gpu: nvidia temp: 39 C
Fan Speeds (RPM): N/A gpu: nvidia fan: 39%
Info:
Processes: 354 Uptime: 51m wakeups: 0 Memory: 15.56 GiB
used: 2.45 GiB (15.8%) Init: systemd v: 250 tool: systemctl Compilers:
gcc: 11.2.0 Packages: pacman: 1426 lib: 440 Shell: garuda-inxi (sudo)
default: Bash v: 5.1.16 running-in: xfce4-terminal inxi: 3.3.14
Garuda (2.5.6-2):
System install date: 2022-03-25
Last full system update: 2022-03-28
Is partially upgraded: No
Relevant software: NetworkManager
Windows dual boot: No/Undetected
Snapshots: Snapper
Failed units: bluetooth-autoconnect.service
dmesg | grep -i vfio output:
[ 61.945184] VFIO - User Level meta-driver version: 0.3
[ 1583.697820] vfio-pci 0000:23:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem:owns=io+mem
[ 1585.181051] vfio-pci 0000:23:00.0: vfio_ecap_init: hiding ecap [email protected]
[ 1585.182543] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1585.183007] vfio-pci 0000:23:00.0: No more image in the PCI ROM
[ 1607.439170] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439201] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439335] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439349] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439365] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439374] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908387] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908439] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908606] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908663] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908697] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908714] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1842.316598] vfio-pci 0000:23:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem