GPU Passthrough with GTX 1060 and RTX 3080 on AMD CPU

[FGD]

Tough one! But interesting!

@Ledge

cause I only have Nvidia GPUs

@Ledge

[ 0.580546] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported

Why are you loading AMD IOMMU if you only got NVIDIA cards? EDIT: Sorry, I think this is related to the Motherboard and not the video card, since you got an AMD CPU then that’s ok.
Is that the way it’s done with nvidia? EDIT: See above.
My experience is more on AMD iGPUs (which failed differently lol) and 1 Intel PCI (success 1st try). Not sure with nvidia.

How are you setup for passthrough?
Which hypervisor are you using? Virt-manager, VBox, Boxes, Proxmox (I guess not cuz you said you are in Garuda so it must be a Type 2 visor).

I have quite a few ideas though, some you may already have tried, some may not make sense and some others are for reading in order to maybe get more ideas (hey, Linux is all about ideas right? ).

• Since you did it with Suse, I assume you have turned on the proper settings in your BIOS, whatever updated or not.
• Try swapping slots of your video cards, if you can (doesn’t make sense, uh? I said some would not, but that exact “stupidity” fixed it on one thread I’ve seen in the many I have read about passthrough).
• The vfio modules you are loading seemed to be the right ones. Are you loading them at the right place (file) for Arch? I don’t know about Arch.
• The Kernel parms usually require 2 (sometimes more) parms, say amd_iommu=on iommu=pt for AMD boards. I don’t quite see that in your kernel command line, I only see the iommu=pt.
• You could try going the Mediated Device way (mdev). I am not familiar with that but I saw you tried to bind the card. I have no idea if mdev would answer your requirements.
• Try other distros than Suse and Garuda.
• Try a Debian distro.
• Try a more recent Kernel with Suse.
• Try to compare all the command lines possible between your Suse working install and Garuda failing install. Maybe you could find a pattern where it breaks.
• And finally you could read those:
  PCI Passthrough - Proxmox VE
  Enable Proxmox PCIe Passthrough - Thomas-Krenn-Wiki-en
  Proxmox GPU Passthough to Ubuntu VM - Hello, world! I'm NOPResearcher

I doubt they will fix your issue, but maybe they will hop you onto something which would then later on fix the issue. Funny how often that happens…

[Ledge]

thank you for the answer, I'll try some of your proposition I ain't tried. Here is a list that will (I hope) answer to some of your solutions.

AMD IOMMU is enable by default on all linux distro I've tried (Ubuntu, KUbuntu, Garuda, fedora, PopOs, Suse) so I don't need to speficy amd_iommu=on.

I'm using Virt-Manager hypervisor, making a virtual machine with it always works, but I'm not able to pass my GPU to it cause I can't load the correct driver on it (my problem).

I've check twice my bios settings and these are correctly set

I already swap my videos cards multiple times (it makes sense, cause the boot GPU and the IOMMU order depend on where the GPU is on the board).

in my research, I found that on every arch distro, I must specifies all supplementaries modules in my initramfs, and mkinitcpio.conf is the initramfs config for garuda. So I think this is the right place.

Additional informations : I tried GPU passthrough on an old MB (MSI X570 Gaming Edge Wifi), and change to an ASUS ROG Strix X570, tried GPU passthrough, nothing change, both MB are able to do GPU passthrough.

[FGD]

You're right, it sounds like there is a config missing somewhere.