GPU Passthrough: Nvidia GPU always loads nvidia driver instead of vfio-pci

I am trying to gpu passthrough my rtx4000 on a ThinkPad P53 with this guide: https://github.com/vanities/GPU-Passthrough-Arch-Linux-to-Windows10

After rebuilding the initramfs and checking the pci devices, it shows that vfio is not loading.

I found the same problem here: [SOLVED] Nvidia GPU always loads nvidia driver instead of the vfio-pci / Laptop Issues / Arch Linux Forums and I want to show you my dmesg output:

sudo dmesg | grep -i -e DMAR -e IOMMU
 
[    0.000000] Command line: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=453807b3-d5a8-4c9d-a57e-b3767248fda0 rw rootflags=subvol=@ quiet ... intel_iommu=on cryptdevice=UUID=b9eb309d-f441-4758-b04e-bb1bc6413e0a:luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a root=/dev/mapper/luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 resume=/dev/mapper/luks-96a6a9e9-eb3b-422f-8c2a-41016323067a loglevel=3
[    0.011591] ACPI: DMAR 0x000000008B4D3000 0000C8 (v01 LENOVO TP-N2N   00001340 PTEC 00000002)
[    0.011651] ACPI: Reserving DMAR table memory at [mem 0x8b4d3000-0x8b4d30c7]
[    0.079204] Kernel command line: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=453807b3-d5a8-4c9d-a57e-b3767248fda0 rw rootflags=subvol=@ quiet ... intel_iommu=on cryptdevice=UUID=b9eb309d-f441-4758-b04e-bb1bc6413e0a:luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a root=/dev/mapper/luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 resume=/dev/mapper/luks-96a6a9e9-eb3b-422f-8c2a-41016323067a loglevel=3
[    0.079290] DMAR: IOMMU enabled
[    0.184804] DMAR: Host address width 39
[    0.184805] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.184811] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.184814] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.184818] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.184820] DMAR: RMRR base: 0x0000008f97a000 end: 0x0000008f999fff
[    0.184822] DMAR: RMRR base: 0x0000009b000000 end: 0x0000009f7fffff
[    0.184823] DMAR: RMRR base: 0x0000008f9ab000 end: 0x0000008fa2afff
[    0.184825] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.184827] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.184828] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.187332] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.514645] iommu: Default domain type: Translated 
[    0.514647] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.552739] DMAR: No ATSR found
[    0.552740] DMAR: No SATC found
[    0.552741] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.552743] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.552744] DMAR: IOMMU feature nwfs inconsistent
[    0.552744] DMAR: IOMMU feature pasid inconsistent
[    0.552745] DMAR: IOMMU feature eafs inconsistent
[    0.552746] DMAR: IOMMU feature prs inconsistent
[    0.552747] DMAR: IOMMU feature nest inconsistent
[    0.552747] DMAR: IOMMU feature mts inconsistent
[    0.552748] DMAR: IOMMU feature sc_support inconsistent
[    0.552749] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.552750] DMAR: dmar0: Using Queued invalidation
[    0.552753] DMAR: dmar1: Using Queued invalidation
[    0.553034] pci 0000:00:00.0: Adding to iommu group 0
[    0.553054] pci 0000:00:01.0: Adding to iommu group 1
[    0.553065] pci 0000:00:02.0: Adding to iommu group 2
[    0.553076] pci 0000:00:04.0: Adding to iommu group 3
[    0.553087] pci 0000:00:08.0: Adding to iommu group 4
[    0.553104] pci 0000:00:12.0: Adding to iommu group 5
[    0.553127] pci 0000:00:14.0: Adding to iommu group 6
[    0.553143] pci 0000:00:14.2: Adding to iommu group 6
[    0.553166] pci 0000:00:15.0: Adding to iommu group 7
[    0.553177] pci 0000:00:15.1: Adding to iommu group 7
[    0.553199] pci 0000:00:16.0: Adding to iommu group 8
[    0.553210] pci 0000:00:16.3: Adding to iommu group 8
[    0.553227] pci 0000:00:1b.0: Adding to iommu group 9
[    0.553247] pci 0000:00:1b.4: Adding to iommu group 10
[    0.553266] pci 0000:00:1c.0: Adding to iommu group 11
[    0.553280] pci 0000:00:1c.5: Adding to iommu group 12
[    0.553298] pci 0000:00:1c.7: Adding to iommu group 13
[    0.553313] pci 0000:00:1d.0: Adding to iommu group 14
[    0.553329] pci 0000:00:1e.0: Adding to iommu group 15
[    0.553368] pci 0000:00:1f.0: Adding to iommu group 16
[    0.553380] pci 0000:00:1f.3: Adding to iommu group 16
[    0.553392] pci 0000:00:1f.4: Adding to iommu group 16
[    0.553405] pci 0000:00:1f.5: Adding to iommu group 16
[    0.553416] pci 0000:00:1f.6: Adding to iommu group 16
[    0.553421] pci 0000:01:00.0: Adding to iommu group 1
[    0.553425] pci 0000:01:00.1: Adding to iommu group 1
[    0.553430] pci 0000:01:00.2: Adding to iommu group 1
[    0.553435] pci 0000:01:00.3: Adding to iommu group 1
[    0.553452] pci 0000:02:00.0: Adding to iommu group 17
[    0.553469] pci 0000:03:00.0: Adding to iommu group 18
[    0.553490] pci 0000:04:00.0: Adding to iommu group 19
[    0.553519] pci 0000:05:00.0: Adding to iommu group 20
[    0.553549] pci 0000:05:01.0: Adding to iommu group 21
[    0.553580] pci 0000:05:02.0: Adding to iommu group 22
[    0.553606] pci 0000:05:04.0: Adding to iommu group 23
[    0.553643] pci 0000:06:00.0: Adding to iommu group 24
[    0.553679] pci 0000:2c:00.0: Adding to iommu group 25
[    0.553698] pci 0000:52:00.0: Adding to iommu group 26
[    0.553711] pci 0000:54:00.0: Adding to iommu group 27
[    0.553728] pci 0000:55:00.0: Adding to iommu group 28
[    0.557021] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.590530] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

Also I added the usb and bus controller ids to the vfio.conf so there are 4 ID's in it now.

Whats going wrong here?
Looking forward for any help, I am noob with linux

Can you provlde your

garuda-inxi

As the new topic template requires

λ garuda-inxi
System:
Kernel: 5.16.7-zen1-1-zen x86_64 bits: 64 compiler: gcc v: 11.1.0
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
root=UUID=453807b3-d5a8-4c9d-a57e-b3767248fda0 rw rootflags=subvol=@
quiet ... intel_iommu=on
cryptdevice=UUID=b9eb309d-f441-4758-b04e-bb1bc6413e0a:luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a
root=/dev/mapper/luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a quiet splash
rd.udev.log_priority=3 vt.global_cursor_default=0
resume=/dev/mapper/luks-96a6a9e9-eb3b-422f-8c2a-41016323067a loglevel=3
Desktop: KDE Plasma 5.24.0 tk: Qt 5.15.2 info: latte-dock wm: kwin_x11
vt: 1 dm: SDDM Distro: Garuda Linux base: Arch Linux
Machine:
Type: Laptop System: LENOVO product: 20QN000SGE v: ThinkPad P53
serial: <superuser required> Chassis: type: 10 serial: <superuser required>
Mobo: LENOVO model: 20QN000SGE v: SDK0K17763 WIN
serial: <superuser required> UEFI: LENOVO v: N2NET49W (1.34 )
date: 10/14/2021
Battery:
ID-1: BAT0 charge: 78.1 Wh (93.9%) condition: 83.2/90.0 Wh (92.5%)
volts: 12.1 min: 11.2 model: SMP 02DL028 type: Li-poly serial: <filter>
status: Discharging cycles: 187
CPU:
Info: model: Intel Core i9-9880H bits: 64 type: MT MCP arch: Coffee Lake
family: 6 model-id: 0x9E (158) stepping: 0xD (13) microcode: 0xEA
Topology: cpus: 1x cores: 8 tpc: 2 threads: 16 smt: enabled cache:
L1: 512 KiB desc: d-8x32 KiB; i-8x32 KiB L2: 2 MiB desc: 8x256 KiB
L3: 16 MiB desc: 1x16 MiB
Speed (MHz): avg: 1492 high: 3035 min/max: 800/4800 scaling:
driver: intel_pstate governor: powersave cores: 1: 2340 2: 1630 3: 1957
4: 1750 5: 870 6: 3035 7: 1568 8: 892 9: 996 10: 1250 11: 1267 12: 829
13: 2475 14: 1060 15: 842 16: 1120 bogomips: 73598
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Vulnerabilities:
Type: itlb_multihit status: KVM: VMX disabled
Type: l1tf status: Not affected
Type: mds status: Not affected
Type: meltdown status: Not affected
Type: spec_store_bypass
mitigation: Speculative Store Bypass disabled via prctl
Type: spectre_v1
mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2
mitigation: Enhanced IBRS, IBPB: conditional, RSB filling
Type: srbds mitigation: TSX disabled
Type: tsx_async_abort mitigation: TSX disabled
Graphics:
Device-1: Intel CoffeeLake-H GT2 [UHD Graphics 630] vendor: Lenovo
driver: i915 v: kernel bus-ID: 00:02.0 chip-ID: 8086:3e9b class-ID: 0300
Device-2: NVIDIA TU104GLM [Quadro RTX 4000 Mobile / Max-Q] vendor: Lenovo
driver: nvidia v: 510.47.03 alternate: nouveau,nvidia_drm bus-ID: 01:00.0
chip-ID: 10de:1eb6 class-ID: 0300
Device-3: Chicony ThinkPad T490 Webcam type: USB driver: uvcvideo
bus-ID: 1-8:2 chip-ID: 04f2:b681 class-ID: 0e02 serial: <filter>
Display: x11 server: X.Org 1.21.1.3 compositor: kwin_x11 driver:
loaded: modesetting,nvidia display-ID: :0 screens: 1
Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.0x11.2")
s-diag: 582mm (22.9")
Monitor-1: eDP-1 res: 1920x1080 hz: 60 dpi: 142
size: 344x194mm (13.5x7.6") diag: 395mm (15.5")
OpenGL: renderer: Mesa Intel UHD Graphics 630 (CFL GT2)
v: 4.6 Mesa 21.3.5 direct render: Yes
Audio:
Device-1: Intel Cannon Lake PCH cAVS vendor: Lenovo driver: snd_hda_intel
v: kernel alternate: snd_soc_skl,snd_sof_pci_intel_cnl bus-ID: 00:1f.3
chip-ID: 8086:a348 class-ID: 0403
Device-2: NVIDIA TU104 HD Audio vendor: Lenovo driver: snd_hda_intel
v: kernel bus-ID: 01:00.1 chip-ID: 10de:10f8 class-ID: 0403
Sound Server-1: ALSA v: k5.16.7-zen1-1-zen running: yes
Sound Server-2: PulseAudio v: 15.0 running: no
Sound Server-3: PipeWire v: 0.3.45 running: yes
Network:
Device-1: Intel Ethernet I219-LM vendor: Lenovo driver: e1000e v: kernel
port: N/A bus-ID: 00:1f.6 chip-ID: 8086:15bb class-ID: 0200
IF: enp0s31f6 state: down mac: <filter>
Device-2: Intel Wi-Fi 6 AX200 driver: iwlwifi v: kernel bus-ID: 52:00.0
chip-ID: 8086:2723 class-ID: 0280
IF: wlp82s0 state: up mac: <filter>
Bluetooth:
Device-1: Intel AX200 Bluetooth type: USB driver: btusb v: 0.8
bus-ID: 1-14:5 chip-ID: 8087:0029 class-ID: e001
Report: bt-adapter ID: hci0 rfk-id: 1 state: up address: <filter>
Drives:
Local Storage: total: 2.75 TiB used: 24.26 GiB (0.9%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/nvme0n1 maj-min: 259:4 vendor: Crucial model: CT1000P5SSD8
size: 931.51 GiB block-size: physical: 512 B logical: 512 B
speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: P4CR311
temp: 31.9 C scheme: GPT
ID-2: /dev/nvme1n1 maj-min: 259:7 vendor: Samsung
model: MZVLB1T0HBLR-000L7 size: 953.87 GiB block-size: physical: 512 B
logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter>
rev: 5M2QEXF7 temp: 34.9 C scheme: GPT
ID-3: /dev/nvme2n1 maj-min: 259:0 vendor: Sabrent model: Rocket Q
size: 931.51 GiB block-size: physical: 512 B logical: 512 B
speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: RKT30Q.2
temp: 26.9 C scheme: GPT
Partition:
ID-1: / raw-size: 897.15 GiB size: 897.15 GiB (100.00%)
used: 24.26 GiB (2.7%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
mapped: luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a
ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
used: 724 KiB (0.2%) fs: vfat dev: /dev/nvme2n1p1 maj-min: 259:1
ID-3: /home raw-size: 897.15 GiB size: 897.15 GiB (100.00%)
used: 24.26 GiB (2.7%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
mapped: luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a
ID-4: /var/log raw-size: 897.15 GiB size: 897.15 GiB (100.00%)
used: 24.26 GiB (2.7%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
mapped: luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a
ID-5: /var/tmp raw-size: 897.15 GiB size: 897.15 GiB (100.00%)
used: 24.26 GiB (2.7%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
mapped: luks-b9eb309d-f441-4758-b04e-bb1bc6413e0a
Swap:
Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
ID-1: swap-1 type: partition size: 34.06 GiB used: 0 KiB (0.0%)
priority: -2 dev: /dev/dm-1 maj-min: 254:1
mapped: luks-96a6a9e9-eb3b-422f-8c2a-41016323067a
ID-2: swap-2 type: zram size: 30.97 GiB used: 0 KiB (0.0%) priority: 100
dev: /dev/zram0
Sensors:
System Temperatures: cpu: 56.0 C mobo: N/A
Fan Speeds (RPM): fan-1: 2043 fan-2: 1821
Info:
Processes: 412 Uptime: 2m wakeups: 1874 Memory: 30.97 GiB
used: 3.85 GiB (12.4%) Init: systemd v: 250 tool: systemctl Compilers:
gcc: 11.1.0 clang: 13.0.1 Packages: pacman: 1422 lib: 336 Shell: fish
v: 3.3.1 default: Bash v: 5.1.16 running-in: konsole inxi: 3.3.12
Garuda (2.5.4-2):
System install date:     2022-02-09
Last full system update: 2022-02-09
Is partially upgraded:   No
Relevant software:       NetworkManager
Windows dual boot:       Probably (Run as root to verify)
Snapshots:               Snapper
Failed units:

Side-question: I am dual-boot win10 on a second drive and my geruda is encrypted - can it mess up something when I am booting or updating win10?

Hey, see
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

Read it carefully and post full output of

sudo dmesg
2 Likes