The fabled Code 43 issue

Issues & Assistance Newbies
1

Hello.
I have been struggling with setting up GPU passthrough with QEMU/KVM for the past few days now, with no end in sight. Everything works, except the GPU passthrough, I get the classic Code 43 code there, being stuck in 800x600.
For context, I have followed SOG's "tutorial" on how to do his scripts and passthroughs, with additional help from the Arch wiki.

It seems to me that whatever I do, I'm not able to get my GPU to passthrough without getting code 43 on Windows 10, hopefully someone more versed can help me with this.
I use a R5 1600 and Zotac 1070ti mini.

I have attempted the following:
This KVM yoke in the VM XML conf.

<hyperv mode="custom">
      ...
      <vendor_id state="on" value="garfield cat"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>

Gpu reservation errors, and after trying the commands, I get the following.

vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
...
echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove
echo 1 > /sys/bus/pci/rescan
...
""/""

Most interestingly, the pre-dumped vBIOS for my GPU was about 4 times as big my own vBIOS dumped from my GPU, not to mention that the beginnings of their roms were different.
For some reason, my graphic card's vBIOS did not want to display anything on the screen, and the pre-dumped worked just fine.
My 1070ti's rom:

U.s.K7400.L.w.VIDEO

Pre-dumped 1070ti's rom:

U.y.K7400.L.w.VIDEO

Any help is greatly appreciated, I am at a loss for knowledge and ideas.

Boot parameters:

splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 pcie_acs_override=downstream amd_iommu=on iommu=pt pci=realloc video=efifb:off
2

Please follow the template and show your garuda-inxi

5 Likes
3

Sorry, here it is.

ystem:
  Kernel: 5.16.16-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 11.2.0
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
    root=UUID=aa8e138c-42cc-42e0-bdfd-8790fdeccf77 rw rootflags=subvol=@
    splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
    pcie_acs_override=downstream amd_iommu=on pci=realloc
  Desktop: Xfce v: 4.16.0 tk: Gtk v: 3.24.29 info: xfce4-panel wm: xfwm
    v: 4.16.1 vt: 7 dm: LightDM v: 1.30.0 Distro: Garuda Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: ASRock model: AB350M Pro4 serial: <filter>
    UEFI-[Legacy]: American Megatrends v: P2.30 date: 04/10/2017
CPU:
  Info: model: AMD Ryzen 5 1600 socket: AM4 bits: 64 type: MT MCP arch: Zen
    family: 0x17 (23) model-id: 1 stepping: 1 microcode: 0x800111C
  Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
    L1: 576 KiB desc: d-6x32 KiB; i-6x64 KiB L2: 3 MiB desc: 6x512 KiB
    L3: 16 MiB desc: 2x8 MiB
  Speed (MHz): avg: 1895 high: 3200 min/max: 1550/3200 boost: enabled
    base/boost: 3200/3700 scaling: driver: acpi-cpufreq governor: schedutil
    volts: 1.2 V ext-clock: 100 MHz cores: 1: 2800 2: 1550 3: 1550 4: 1550
    5: 3200 6: 1550 7: 1550 8: 1550 9: 1550 10: 1550 11: 2800 12: 1550
    bogomips: 76650
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: spec_store_bypass
    mitigation: Speculative Store Bypass disabled via prctl
  Type: spectre_v1
    mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Retpolines, STIBP: disabled, RSB filling
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: NVIDIA GP104 [GeForce GTX 1070 Ti] vendor: ZOTAC driver: nvidia
    v: 510.54 alternate: nouveau,nvidia_drm pcie: gen: 3 speed: 8 GT/s
    lanes: 16 bus-ID: 09:00.0 chip-ID: 10de:1b82 class-ID: 0300
  Device-2: Oculus VR Rift CV1 Sensor type: USB driver: uvcvideo
    bus-ID: 4-1:2 chip-ID: 2833:0211 class-ID: ef00 serial: <filter>
  Device-3: Oculus VR Rift CV1 Sensor type: USB driver: uvcvideo
    bus-ID: 4-2:3 chip-ID: 2833:0211 class-ID: ef00 serial: <filter>
  Display: x11 server: X.Org v: 1.21.1.3 compositor: xfwm v: 4.16.1 driver:
    X: loaded: nvidia gpu: nvidia display-ID: :0.0 screens: 1
  Screen-1: 0 s-res: 3840x1080 s-dpi: 92 s-size: 1060x301mm (41.73x11.85")
    s-diag: 1102mm (43.38")
  Monitor-1: DVI-D-0 pos: primary,left res: 1920x1080 dpi: 92
    size: 531x298mm (20.91x11.73") diag: 609mm (23.97") modes: N/A
  Monitor-2: HDMI-0 pos: right res: 1920x1080 hz: 60 dpi: 102
    size: 477x268mm (18.78x10.55") diag: 547mm (21.54") modes: N/A
  Message: Unable to show GL data. Required tool glxinfo missing.
Audio:
  Device-1: NVIDIA GP104 High Definition Audio vendor: ZOTAC
    driver: snd_hda_intel v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16
    bus-ID: 09:00.1 chip-ID: 10de:10f0 class-ID: 0403
  Device-2: AMD Family 17h HD Audio vendor: ASRock driver: snd_hda_intel
    v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16 bus-ID: 12:00.3
    chip-ID: 1022:1457 class-ID: 0403
  Device-3: C-Media Blue Snowball type: USB
    driver: hid-generic,snd-usb-audio,usbhid bus-ID: 3-4:2 chip-ID: 0d8c:0005
    class-ID: 0300 serial: <filter>
  Sound Server-1: ALSA v: k5.16.16-zen1-1-zen running: yes
  Sound Server-2: PulseAudio v: 15.0 running: no
  Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    vendor: ASRock driver: r8169 v: kernel pcie: gen: 1 speed: 2.5 GT/s
    lanes: 1 port: f000 bus-ID: 06:00.0 chip-ID: 10ec:8168 class-ID: 0200
  IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IF-ID-1: virbr0 state: down mac: <filter>
Drives:
  Local Storage: total: 1.36 TiB used: 53.69 GiB (3.8%)
  SMART Message: Required tool smartctl not installed. Check --recommends
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Kingston model: SA2000M8500G
    size: 465.76 GiB block-size: physical: 512 B logical: 512 B
    speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: S5Z42105
    temp: 34.9 C scheme: MBR
  ID-2: /dev/sda maj-min: 8:0 vendor: Seagate model: ST1000DM010-2DM162
    size: 931.51 GiB block-size: physical: 4096 B logical: 512 B
    speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter> rev: CC46
    scheme: MBR
Partition:
  ID-1: / raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
  ID-2: /home raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
  ID-3: /var/log raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
  ID-4: /var/tmp raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 53.69 GiB (11.5%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
Swap:
  Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
  ID-1: swap-1 type: zram size: 15.64 GiB used: 0 KiB (0.0%) priority: 100
    dev: /dev/zram0
Sensors:
  System Temperatures: cpu: N/A mobo: N/A gpu: nvidia temp: 40 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 39%
Info:
  Processes: 339 Uptime: 49m wakeups: 0 Memory: 15.64 GiB
  used: 2.32 GiB (14.8%) Init: systemd v: 250 tool: systemctl Compilers:
  gcc: 11.2.0 Packages: pacman: 1426 lib: 440 Shell: Bash v: 5.1.16
  running-in: xfce4-terminal inxi: 3.3.14
Garuda (2.5.6-2):
  System install date:     2022-03-25
  Last full system update: 2022-03-28
  Is partially upgraded:   No
  Relevant software:       NetworkManager
  Windows dual boot:       No/Undetected
  Snapshots:               Snapper
  Failed units:            bluetooth-autoconnect.service
4

Did you install in legacy boot mode on purpose?

Edit: Disregard, now that I look into it I think it should be fine with or without EFI.

Some BIOS menus have a VT-d/IOMMU option you have to turn on, did you already check?

Edit 2: You are also several BIOS updates behind. This could be a factor. https://www.asrock.com/MB/AMD/AB350M%20Pro4/index.asp#BIOS

4 Likes
5

All virtualization and iommu options are enabled.
I will attempt the BIOS Flash now.

6

Updated BIOS, ver 2.3 > 3.4 > 5.1
Booted into VM after adjusting options to the new ids
I still get error 43
garuda-inxi output:

System:
  Kernel: 5.16.16-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 11.2.0
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
    root=UUID=aa8e138c-42cc-42e0-bdfd-8790fdeccf77 rw rootflags=subvol=@
    splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
    pcie_acs_override=downstream amd_iommu=on iommu=pt pci=realloc
    video=efifb:off
  Desktop: Xfce v: 4.16.0 tk: Gtk v: 3.24.29 info: xfce4-panel wm: xfwm
    v: 4.16.1 dm: LightDM v: 1.30.0 Distro: Garuda Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: ASRock model: AB350M Pro4 serial: <filter>
    UEFI-[Legacy]: American Megatrends v: P5.10 date: 10/05/2018
CPU:
  Info: model: AMD Ryzen 5 1600 socket: AM4 bits: 64 type: MT MCP arch: Zen
    family: 0x17 (23) model-id: 1 stepping: 1 microcode: 0x8001137
  Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
    L1: 576 KiB desc: d-6x32 KiB; i-6x64 KiB L2: 3 MiB desc: 6x512 KiB
    L3: 16 MiB desc: 2x8 MiB
  Speed (MHz): avg: 1687 high: 3200 min/max: 1550/3200 boost: enabled
    base/boost: 3200/3700 scaling: driver: acpi-cpufreq governor: schedutil
    volts: 1.2 V ext-clock: 100 MHz cores: 1: 1550 2: 1550 3: 1550 4: 1550
    5: 1550 6: 1550 7: 3200 8: 1550 9: 1550 10: 1550 11: 1550 12: 1550
    bogomips: 76647
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: spec_store_bypass
    mitigation: Speculative Store Bypass disabled via prctl
  Type: spectre_v1
    mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2
    mitigation: Retpolines, IBPB: conditional, STIBP: disabled, RSB filling
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: NVIDIA GP104 [GeForce GTX 1070 Ti] vendor: ZOTAC driver: N/A
    alternate: nouveau, nvidia_drm, nvidia pcie: gen: 3 speed: 8 GT/s lanes: 16
    bus-ID: 23:00.0 chip-ID: 10de:1b82 class-ID: 0300
  Device-2: Oculus VR Rift CV1 Sensor type: USB driver: uvcvideo
    bus-ID: 4-2:2 chip-ID: 2833:0211 class-ID: ef00 serial: <filter>
  Display: server: X.Org v: 1.21.1.3 compositor: xfwm v: 4.16.1 driver: X:
    loaded: nvidia gpu: N/A display-ID: :0.0 screens: 1
  Screen-1: 0 s-res: 3840x1080 s-dpi: 92 s-size: 1060x301mm (41.73x11.85")
    s-diag: 1102mm (43.38")
  Monitor-1: DVI-D-0 pos: primary,left res: 1920x1080 dpi: 92
    size: 531x298mm (20.91x11.73") diag: 609mm (23.97") modes: N/A
  Monitor-2: HDMI-0 pos: right res: 1920x1080 hz: 60 dpi: 102
    size: 477x268mm (18.78x10.55") diag: 547mm (21.54") modes: N/A
  Message: Unable to show GL data. Required tool glxinfo missing.
Audio:
  Device-1: NVIDIA GP104 High Definition Audio vendor: ZOTAC driver: N/A
    alternate: snd_hda_intel pcie: gen: 3 speed: 8 GT/s lanes: 16
    bus-ID: 23:00.1 chip-ID: 10de:10f0 class-ID: 0403
  Device-2: AMD Family 17h HD Audio vendor: ASRock driver: snd_hda_intel
    v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16 bus-ID: 25:00.3
    chip-ID: 1022:1457 class-ID: 0403
  Device-3: C-Media Blue Snowball type: USB
    driver: hid-generic,snd-usb-audio,usbhid bus-ID: 3-4:2 chip-ID: 0d8c:0005
    class-ID: 0300 serial: <filter>
  Sound Server-1: ALSA v: k5.16.16-zen1-1-zen running: yes
  Sound Server-2: PulseAudio v: 15.0 running: no
  Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    vendor: ASRock driver: r8169 v: kernel pcie: gen: 1 speed: 2.5 GT/s
    lanes: 1 port: f000 bus-ID: 1f:00.0 chip-ID: 10ec:8168 class-ID: 0200
  IF: enp31s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IF-ID-1: virbr0 state: down mac: <filter>
Drives:
  Local Storage: total: 1.36 TiB used: 61.46 GiB (4.4%)
  SMART Message: Required tool smartctl not installed. Check --recommends
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Kingston model: SA2000M8500G
    size: 465.76 GiB block-size: physical: 512 B logical: 512 B
    speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: S5Z42105
    temp: 35.9 C scheme: MBR
  ID-2: /dev/sda maj-min: 8:0 vendor: Seagate model: ST1000DM010-2DM162
    size: 931.51 GiB block-size: physical: 4096 B logical: 512 B
    speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter> rev: CC46
    scheme: MBR
Partition:
  ID-1: / raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
  ID-2: /home raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
  ID-3: /var/log raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
  ID-4: /var/tmp raw-size: 465.76 GiB size: 465.76 GiB (100.00%)
    used: 59.14 GiB (12.7%) fs: btrfs block-size: 4096 B dev: /dev/nvme0n1p1
    maj-min: 259:1
Swap:
  Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
  ID-1: swap-1 type: zram size: 15.56 GiB used: 1024 KiB (0.0%)
    priority: 100 dev: /dev/zram0
Sensors:
  System Temperatures: cpu: N/A mobo: N/A gpu: nvidia temp: 39 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 39%
Info:
  Processes: 354 Uptime: 51m wakeups: 0 Memory: 15.56 GiB
  used: 2.45 GiB (15.8%) Init: systemd v: 250 tool: systemctl Compilers:
  gcc: 11.2.0 Packages: pacman: 1426 lib: 440 Shell: garuda-inxi (sudo)
  default: Bash v: 5.1.16 running-in: xfce4-terminal inxi: 3.3.14
Garuda (2.5.6-2):
  System install date:     2022-03-25
  Last full system update: 2022-03-28
  Is partially upgraded:   No
  Relevant software:       NetworkManager
  Windows dual boot:       No/Undetected
  Snapshots:               Snapper
  Failed units:            bluetooth-autoconnect.service

dmesg | grep -i vfio output:

[   61.945184] VFIO - User Level meta-driver version: 0.3
[ 1583.697820] vfio-pci 0000:23:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem:owns=io+mem
[ 1585.181051] vfio-pci 0000:23:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 1585.182543] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1585.183007] vfio-pci 0000:23:00.0: No more image in the PCI ROM
[ 1607.439170] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439201] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439335] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439349] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439365] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1607.439374] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908387] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908439] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908606] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908663] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908697] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1803.908714] vfio-pci 0000:23:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1842.316598] vfio-pci 0000:23:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem