Btrfs don't boot after vfio binding to gpu

Ok guys don’t know if this is the correct way of reporting a bug, didn’t find any tool to it so here it goes

Garuda inxi

System:
Kernel: 6.7.2-1-vfio arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
clocksource: tsc available: hpet,acpi_pm
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-vfio
root=UUID=119ddacb-f7c1-4125-ba3c-922afc01b144 rw rootflags=subvol=@
quiet splash iommu=pt ibt=off
Desktop: KDE Plasma v: 5.27.10 tk: Qt v: 5.15.12 wm: kwin_wayland vt: 1
dm: SDDM Distro: Garuda Linux base: Arch Linux
Machine:
Type: Desktop Mobo: Gigabyte model: B650M DS3H serial: <superuser required>
UEFI: American Megatrends LLC. v: F21a date: 12/21/2023
CPU:
Info: model: AMD Ryzen 5 7600X bits: 64 type: MT MCP arch: Zen 4 gen: 5
level: v4 note: check built: 2022+ process: TSMC n5 (5nm) family: 0x19 (25)
model-id: 0x61 (97) stepping: 2 microcode: 0xA601206
Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
L1: 384 KiB desc: d-6x32 KiB; i-6x32 KiB L2: 6 MiB desc: 6x1024 KiB
L3: 32 MiB desc: 1x32 MiB
Speed (MHz): avg: 2382 high: 4845 min/max: 400/5453 scaling:
driver: amd-pstate-epp governor: powersave cores: 1: 4845 2: 400 3: 4771
4: 400 5: 4644 6: 400 7: 3684 8: 400 9: 400 10: 3483 11: 4762 12: 400
bogomips: 112634
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Vulnerabilities: <filter>
Graphics:
Device-1: AMD Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT]
vendor: Sapphire driver: amdgpu v: kernel arch: RDNA-2 code: Navi-2x
process: TSMC n7 (7nm) built: 2020-22 pcie: gen: 4 speed: 16 GT/s
lanes: 16 ports: active: none empty: DP-1, DP-2, DP-3, HDMI-A-1
bus-ID: 03:00.0 chip-ID: 1002:73df class-ID: 0300
Device-2: AMD Raphael vendor: Gigabyte driver: amdgpu v: kernel
arch: RDNA-2 code: Navi-2x process: TSMC n7 (7nm) built: 2020-22 pcie:
gen: 4 speed: 16 GT/s lanes: 16 ports: active: HDMI-A-2
empty: DP-4,DP-5,DP-6 bus-ID: 11:00.0 chip-ID: 1002:164e class-ID: 0300
temp: 47.0 C
Display: wayland server: X.org v: 1.21.1.11 with: Xwayland v: 23.2.4
compositor: kwin_wayland driver: X: loaded: amdgpu
unloaded: modesetting,radeon alternate: fbdev,vesa dri: radeonsi
gpu: amdgpu,amdgpu display-ID: 0
Monitor-1: HDMI-A-2 res: 1920x1080 size: N/A modes: N/A
API: EGL v: 1.5 hw: drv: amd radeonsi platforms: device: 0 drv: radeonsi
device: 1 drv: radeonsi device: 2 drv: swrast surfaceless: drv: radeonsi
wayland: drv: radeonsi x11: drv: radeonsi inactive: gbm
API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 23.3.4-arch1.1
glx-v: 1.4 direct-render: yes renderer: AMD Radeon Graphics (radeonsi
raphael_mendocino LLVM 16.0.6 DRM 3.56 6.7.2-1-vfio) device-ID: 1002:164e
memory: 500 MiB unified: no display-ID: :1.0
API: Vulkan v: 1.3.276 layers: 10 device: 0 type: integrated-gpu name: AMD
Radeon Graphics (RADV RAPHAEL_MENDOCINO) driver: mesa radv
v: 23.3.4-arch1.1 device-ID: 1002:164e surfaces: xcb,xlib,wayland
device: 1 type: discrete-gpu name: AMD Radeon RX 6700 XT (RADV NAVI22)
driver: mesa radv v: 23.3.4-arch1.1 device-ID: 1002:73df
surfaces: xcb,xlib,wayland device: 2 type: cpu name: llvmpipe (LLVM
16.0.6 256 bits) driver: mesa llvmpipe v: 23.3.4-arch1.1 (LLVM 16.0.6)
device-ID: 10005:0000 surfaces: xcb,xlib,wayland
Audio:
Device-1: AMD Navi 21/23 HDMI/DP Audio driver: snd_hda_intel v: kernel pcie:
gen: 4 speed: 16 GT/s lanes: 16 bus-ID: 03:00.1 chip-ID: 1002:ab28
class-ID: 0403
Device-2: AMD Rembrandt Radeon High Definition Audio driver: snd_hda_intel
v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16 bus-ID: 11:00.1
chip-ID: 1002:1640 class-ID: 0403
Device-3: AMD Family 17h/19h HD Audio vendor: Gigabyte
driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
bus-ID: 11:00.6 chip-ID: 1022:15e3 class-ID: 0403
API: ALSA v: k6.7.2-1-vfio status: kernel-api tools: N/A
Server-1: PipeWire v: 1.0.1 status: active with: 1: pipewire-pulse
status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
4: pw-jack type: plugin tools: pactl,pw-cat,pw-cli,wpctl
Network:
Device-1: Realtek RTL8125 2.5GbE vendor: Gigabyte driver: r8169 v: kernel
pcie: gen: 2 speed: 5 GT/s lanes: 1 port: e000 bus-ID: 0d:00.0
chip-ID: 10ec:8125 class-ID: 0200
IF: enp13s0 state: up speed: 100 Mbps duplex: full mac: <filter>
IF-ID-1: virbr0 state: down mac: <filter>
Drives:
Local Storage: total: 2.77 TiB used: 1.59 TiB (57.3%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/sda maj-min: 8:0 vendor: Lenovo model: E660 SSD-2.5-2TB
size: 1.86 TiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
tech: SSD serial: <filter> fw-rev: 429 scheme: GPT
ID-2: /dev/sdb maj-min: 8:16 vendor: Western Digital
model: WD10EZEX-00RKKA0 size: 931.51 GiB block-size: physical: 4096 B
logical: 512 B speed: 6.0 Gb/s tech: N/A serial: <filter> fw-rev: 0A80
scheme: GPT
Partition:
ID-1: / raw-size: 394.1 GiB size: 394.1 GiB (100.00%)
used: 51.62 GiB (13.1%) fs: btrfs dev: /dev/sda1 maj-min: 8:1
ID-2: /boot/efi raw-size: 1000 MiB size: 998 MiB (99.80%)
used: 18.8 MiB (1.9%) fs: vfat dev: /dev/sda2 maj-min: 8:2
ID-3: /home raw-size: 394.1 GiB size: 394.1 GiB (100.00%)
used: 51.62 GiB (13.1%) fs: btrfs dev: /dev/sda1 maj-min: 8:1
ID-4: /var/log raw-size: 394.1 GiB size: 394.1 GiB (100.00%)
used: 51.62 GiB (13.1%) fs: btrfs dev: /dev/sda1 maj-min: 8:1
ID-5: /var/tmp raw-size: 394.1 GiB size: 394.1 GiB (100.00%)
used: 51.62 GiB (13.1%) fs: btrfs dev: /dev/sda1 maj-min: 8:1
Swap:
Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default) zswap: no
ID-1: swap-1 type: zram size: 30.5 GiB used: 0 KiB (0.0%) priority: 100
comp: zstd avail: lzo,lzo-rle,lz4,lz4hc,842 max-streams: 12 dev: /dev/zram0
Sensors:
System Temperatures: cpu: 51.0 C mobo: 34.0 C
Fan Speeds (rpm): N/A
GPU: device: amdgpu temp: 47.0 C device: amdgpu temp: 42.0 C mem: 36.0 C
fan: 0 watts: 11.00
Info:
Processes: 354 Uptime: 1h 9m wakeups: 1 Memory: total: 32 GiB note: est.
available: 30.5 GiB used: 6.38 GiB (20.9%) Init: systemd v: 255
default: graphical tool: systemctl Compilers: gcc: 13.2.1 clang: 16.0.6
Packages: pm: pacman pkgs: 1716 libs: 504 tools: gnome-software,octopi,paru
Shell: fish v: 3.7.0 running-in: konsole inxi: 3.3.31
Garuda (2.6.22-1):
System install date:     2024-01-25
Last full system update: 2024-01-27
Is partially upgraded:   No
Relevant software:       snapper NetworkManager dracut
Windows dual boot:       No/Undetected
Failed units:

Bug is as follows: When trying to do a gpu passthrough, doesn’t matter if it’s my pcie gpu, or igpu, system won’t boot after binding to vfio.

Installed Fedora using btrfs and after binding to vfio, same problem happens, it won’t boot, but if i use ext4, system boots normally after the binding.

I found something, see if it helps!
https://bbs.archlinux.org/viewtopic.php?id=290283

Thanks bro, tried that grub config (didn’t see the point of following xorg config since my igpu is newer)
and following exactly like he did, my system boots, but the gpu is not binded to vfio, it needs vfio.conf set up on modprobe.d. After setting it up, the same problem happens
The only difference is the error message now
Before the last line booting while stuck was:Starting Dracut pre-udev hook, following this settings now is: Starting dracut ask for additional cmline parameters (maybe is missing a config ?)

He was also using hooks with mkinitcpio, but i didn’t find how to set up hooks on dracut man and i also don’t believe it needs to be set up (Maybe i’m wrong and needs to be set up, but how ? lol)

I initially thought this was an arch problem, so i went straight to fedora, followed this guide: GPU Passthrough Fedora · GitHub
and it worked. Decided to put the same commands here, gpu actually binds to vfio, but the same problem again.
So i tried reinstalling fedora using btrfs, just to see if my suspicion was correct, and following the same guide above, it generated the same problem here on garuda.

The setup is not complex, i’m almost certain this is a bug, i don’t know if it’s on dracut unable to mount something on btrfs or it’s on btrfs side. Or maybe this is just a missing config on dracut, but all the guides just post the same stuff so i’m out of ideas

Final update on this problem: managed to find out exactly where the error is at
when setting up the system /boot needs to be exactly ext4 in order for vfio binding to work, if set to btfrs or fat32 it will give me this problem.

Managed to get it working on a btfrs fedora just setting /boot to ext4, unfortunetly garuda doesn’t give me this option so for now it is as is

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.