Snapper Corruption - Failed to Switch Root (RO SYSROOT) - All current and ongoing images impacted (options?)

Hi There,
I’m relatively new to Garuda Linux and Arch but not Linux in general.

I’m looking for some insight from those experienced with the Distro around what seems to be a snapper corruption that is impacting all snapshots (past present and future).

I’ve provided some screenshots for examples and I do think I could manually restore them if I had to but I really liked this feature of being able to boot from snapshots on the fly.

Is there any way to rebuild snapper or even core garuda packages, maybe someone could point me in the right direction around dependencies for root cause analysis?

Closest topic I could reference was this and it was sort of inconclusive (seems most were concerned around some sort of file system corruption) - Booting Into Snapshot: Read-Only File System - #8 by BluishHumility


Kernel: 6.5.7-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
clocksource: tsc available: hpet,acpi_pm
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
root=UUID=56d1f10b-cb16-46c0-9169-6bc3151b9b5d rw rootflags=subvol=@
intel_iommu=on iommu=pt isolcpus rcu_nocbs transparent_hugepage=never
quiet nvidia-drm.modeset=1 quiet rd.udev.log_priority=3
resume=UUID=45d3cd5d-df18-46f9-8497-133a5a31ae08 loglevel=3 ibt=off
Desktop: KDE Plasma v: 5.27.8 tk: Qt v: 5.15.11 wm: kwin_x11 vt: 2
dm: SDDM Distro: Garuda Linux base: Arch Linux
Type: Desktop Mobo: ASUSTeK model: MAXIMUS IX HERO v: Rev 1.xx
serial: <superuser required> UEFI: American Megatrends v: 1301
date: 03/14/2018
Info: model: Intel Core i7-7700K bits: 64 type: MT MCP arch: Kaby Lake
gen: core 7 level: v3 note: check built: 2018 process: Intel 14nm family: 6
model-id: 0x9E (158) stepping: 9 microcode: 0xF4
Topology: cpus: 1x cores: 4 tpc: 2 threads: 8 smt: enabled cache:
L1: 256 KiB desc: d-4x32 KiB; i-4x32 KiB L2: 1024 KiB desc: 4x256 KiB
L3: 8 MiB desc: 1x8 MiB
Speed (MHz): avg: 1200 high: 1201 min/max: 800/4800 scaling:
driver: intel_pstate governor: powersave cores: 1: 1200 2: 1200 3: 1200
4: 1200 5: 1200 6: 1200 7: 1201 8: 1200 bogomips: 67200
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Vulnerabilities: <filter>
Device-1: NVIDIA GA102 [GeForce RTX 3080 Lite Hash Rate] vendor: Gigabyte
driver: nvidia v: 535.113.01 alternate: nouveau,nvidia_drm non-free: 535.xx+
status: current (as of 2023-09) arch: Ampere code: GAxxx
process: TSMC n7 (7nm) built: 2020-23 pcie: gen: 2 speed: 5 GT/s lanes: 16
link-max: gen: 4 speed: 16 GT/s ports: active: none off: DP-1,DP-2
empty: DP-3, HDMI-A-1, HDMI-A-2, HDMI-A-3 bus-ID: 01:00.0
chip-ID: 10de:2216 class-ID: 0300
Device-2: Lite-On HP Full-HD Camera driver: uvcvideo type: USB rev: 2.0
speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-6:3 chip-ID: 04ca:7078
class-ID: 0e02 serial: <filter>
Device-3: Oculus VR Rift CV1 Sensor driver: uvcvideo type: USB rev: 3.0
speed: 5 Gb/s lanes: 1 mode: 3.2 gen-1x1 bus-ID: 2-5:3 chip-ID: 2833:0211
class-ID: ef00 serial: <filter>
Display: x11 server: X.Org v: 21.1.8 with: Xwayland v: 23.2.1
compositor: kwin_x11 driver: X: loaded: nvidia unloaded: modesetting,nouveau
alternate: fbdev,nv,vesa gpu: nvidia,nvidia-nvswitch display-ID: :0
screens: 1
Screen-1: 0 s-res: 7040x1440 s-dpi: 91 s-size: 1962x401mm (77.24x15.79")
s-diag: 2003mm (78.84")
Monitor-1: DP-1 mapped: DP-0 note: disabled pos: primary,right
model: Samsung LS49AG95 serial: <filter> built: 2021 res: 5120x1440 hz: 240
dpi: 130048 gamma: 1.2 size: 1x1mm (0.04x0.04") diag: 1239mm (48.8")
modes: max: 3840x1080 min: 640x480
Monitor-2: DP-2 note: disabled pos: left model: Dell P2419H
serial: <filter> built: 2019 res: 1920x1080 hz: 60 dpi: 93 gamma: 1.2
size: 527x296mm (20.75x11.65") diag: 604mm (23.8") ratio: 16:9 modes:
max: 1920x1080 min: 640x480
API: EGL v: 1.5 hw: drv: nvidia platforms: gbm: drv: nvidia
API: OpenGL v: 4.6.0 vendor: nvidia v: 535.113.01 glx-v: 1.4
direct-render: yes renderer: NVIDIA GeForce RTX 3080/PCIe/SSE2
memory: 9.77 GiB
API: Vulkan v: 1.3.264 layers: 14 device: 0 type: discrete-gpu
name: NVIDIA GeForce RTX 3080 driver: nvidia v: 535.113.01
device-ID: 10de:2216 surfaces: xcb,xlib device: 1 type: discrete-gpu
name: NVIDIA GeForce RTX 3080 driver: nvidia v: 535.113.01
device-ID: 10de:2216 surfaces: xcb,xlib device: 2 type: cpu name: llvmpipe
(LLVM 16.0.6 256 bits) driver: mesa llvmpipe v: 23.2.1-arch1.2 (LLVM
16.0.6) device-ID: 10005:0000 surfaces: xcb,xlib
Device-1: Intel 200 Series PCH HD Audio vendor: ASUSTeK
driver: snd_hda_intel v: kernel alternate: snd_soc_avs bus-ID: 00:1f.3
chip-ID: 8086:a2f0 class-ID: 0403
Device-2: NVIDIA GA102 High Definition Audio vendor: Gigabyte
driver: snd_hda_intel v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16
link-max: gen: 4 speed: 16 GT/s bus-ID: 01:00.1 chip-ID: 10de:1aef
class-ID: 0403
Device-3: JMTek LLC. LCS USB Audio
driver: hid-generic,snd-usb-audio,usbhid type: USB rev: 1.1 speed: 12 Mb/s
lanes: 1 mode: 1.1 bus-ID: 1-7:5 chip-ID: 0c76:2018 class-ID: 0300
serial: <filter>
API: ALSA v: k6.5.7-zen1-1-zen status: kernel-api
tools: alsactl,alsamixer,amixer
Server-1: sndiod v: N/A status: off tools: aucat,midicat,sndioctl
Server-2: PipeWire v: 0.3.82 status: active with: 1: pipewire-pulse
status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
4: pw-jack type: plugin tools: pactl,pw-cat,pw-cli,wpctl
Device-1: Intel Ethernet I219-V vendor: ASUSTeK driver: e1000e v: kernel
port: N/A bus-ID: 00:1f.6 chip-ID: 8086:15b8 class-ID: 0200
IF: enp0s31f6 state: up speed: 1000 Mbps duplex: full mac: <filter>
IF-ID-1: br0 state: up speed: 10000 Mbps duplex: unknown mac: <filter>
IF-ID-2: vnet0 state: unknown speed: 10000 Mbps duplex: full mac: <filter>
Local Storage: total: 16.84 TiB used: 10.1 TiB (60.0%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Samsung model: SSD 950 PRO 512GB
size: 476.94 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
lanes: 4 tech: SSD serial: <filter> fw-rev: 2B0QBXX7 temp: 27.9 C
scheme: GPT
ID-2: /dev/nvme1n1 maj-min: 259:2 vendor: Samsung model: SSD 980 PRO 2TB
size: 1.82 TiB block-size: physical: 512 B logical: 512 B speed: 63.2 Gb/s
lanes: 4 tech: SSD serial: <filter> fw-rev: 5B2QGXA7 temp: 28.9 C
scheme: GPT
ID-3: /dev/sda maj-min: 8:0 vendor: Seagate model: ST2000DX002-2DV164
size: 1.82 TiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
tech: HDD rpm: 7200 serial: <filter> fw-rev: CC41 scheme: GPT
ID-4: /dev/sdb maj-min: 8:16 vendor: Seagate model: ST4000DX001-1CE168
size: 3.64 TiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
tech: HDD rpm: 5980 serial: <filter> fw-rev: DA11 scheme: GPT
ID-5: /dev/sdc maj-min: 8:32 vendor: Seagate model: ST10000VN0004-1ZD101
size: 9.1 TiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
tech: HDD rpm: 7200 serial: <filter> fw-rev: SC60 scheme: GPT
ID-1: / raw-size: 1.75 TiB size: 1.75 TiB (100.00%) used: 893.82 GiB (49.8%)
fs: btrfs dev: /dev/nvme1n1p2 maj-min: 259:4
ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
used: 576 KiB (0.2%) fs: vfat dev: /dev/nvme1n1p1 maj-min: 259:3
ID-3: /home raw-size: 1.75 TiB size: 1.75 TiB (100.00%)
used: 893.82 GiB (49.8%) fs: btrfs dev: /dev/nvme1n1p2 maj-min: 259:4
ID-4: /var/log raw-size: 1.75 TiB size: 1.75 TiB (100.00%)
used: 893.82 GiB (49.8%) fs: btrfs dev: /dev/nvme1n1p2 maj-min: 259:4
ID-5: /var/tmp raw-size: 1.75 TiB size: 1.75 TiB (100.00%)
used: 893.82 GiB (49.8%) fs: btrfs dev: /dev/nvme1n1p2 maj-min: 259:4
Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default) zswap: no
ID-1: swap-1 type: zram size: 62.73 GiB used: 15.2 MiB (0.0%)
priority: 100 comp: zstd avail: lzo,lzo-rle,lz4,lz4hc,842 max-streams: 8
dev: /dev/zram0
ID-2: swap-2 type: partition size: 69.01 GiB used: 0 KiB (0.0%)
priority: -2 dev: /dev/nvme1n1p3 maj-min: 259:5
System Temperatures: cpu: 34.0 C mobo: N/A gpu: nvidia temp: 23 C
Fan Speeds (rpm): N/A gpu: nvidia fan: 0%
Processes: 332 Uptime: 12h 52m wakeups: 1 Memory: total: 64 GiB note: est.
available: 62.73 GiB used: 52.95 GiB (84.4%) Init: systemd v: 254
default: graphical tool: systemctl Compilers: gcc: 13.2.1 Packages:
pm: pacman pkgs: 1633 libs: 483 tools: octopi,paru Shell: fish v: 3.6.1
default: Bash v: 5.1.16 running-in: konsole inxi: 3.3.30
Garuda (2.6.17-1):
System install date:     2023-10-06
Last full system update: 2023-10-15
Is partially upgraded:   No
Relevant software:       snapper NetworkManager dracut nvidia-dkms
Windows dual boot:       No/Undetected
Failed units:

Can you describe what you are doing leading up to the failed boot? It sounds like you are trying to boot off a snapshot from Grub, is that right?

Try reinstalling your kernel and paste the output into the thread so we can take a look.

sudo pacman -S linux-zen

Regenerate the Grub config and paste into the thread.

sudo update-grub

Let’s also see your fstab.

cat /etc/fstab
1 Like

Confirmed and implemented, snapshot restoration reason (this time) was with graphics drivers.

Have been playing with optimising the graphics setup on this rig, Starfield Nvidia driver 535.43.13 Vulkan-based works fully on Ampere GPU but its not perfectly compatible with this Distro yet, Utils don’t work etc, I do think I could get it to work but I imagine it will be less than a few weeks until another major release. Ended up reverting for now.

On recovery (from garuda grub menu) I noticed the snapshots have been down for some time (although did originally work, tested at least once).

Will test snapshot recovery post kernel refresh now - I will only report back if it was a success.

[🧱] × sudo pacman -S linux-zen
[sudo] password for XXX:
warning: linux-zen-6.5.7.zen1-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) linux-zen-6.5.7.zen1-1

Total Installed Size:  132.00 MiB
Net Upgrade Size:        0.00 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                     [------------------------------------] 100%
(1/1) checking package integrity                                   [------------------------------------] 100%
(1/1) loading package files                                        [------------------------------------] 100%
(1/1) checking for file conflicts                                  [------------------------------------] 100%
(1/1) checking available disk space                                [------------------------------------] 100%
:: Running pre-transaction hooks...
(1/2) Performing snapper pre snapshots for the following configurations...
==> root: 142
(2/2) Saving Linux kernel modules...
:: Processing package changes...
(1/1) reinstalling linux-zen                                       [------------------------------------] 100%
:: Running post-transaction hooks...
( 1/11) Restoring Linux kernel modules...
++ uname -r
+ KVER=6.5.7-zen1-1-zen
+ test -e /usr/lib/modules/backup/6.5.7-zen1-1-zen
+ rsync -AHXal --ignore-existing /usr/lib/modules/backup/6.5.7-zen1-1-zen /usr/lib/modules/
+ rm -rf /usr/lib/modules/backup
( 2/11) Arming ConditionNeedsUpdate...
( 3/11) Updating module dependencies...
( 4/11) Updating initramfs...
:: Building initramfs for linux-zen (6.5.7-zen1-1-zen)
:: Building fallback initramfs for linux-zen (6.5.7-zen1-1-zen)
( 5/11) Updating linux initcpios...
( 6/11) Refreshing PackageKit...
( 7/11) Foreign/AUR package notification
libsidplay 1.36.59-10
( 8/11) Orphaned package notification...
cmake 3.27.7-1
fltk 1.3.8-2
gptfdisk 1.0.9-2
innoextract 1.9-9
kjs 5.110.0-1
lib32-gamemode 1.7-1
lib32-mangohud 0.7.0-2
libindicator-gtk3 12.10.1-10
libmfx 23.2.2-2
libsidplay 1.36.59-10
libvips 8.14.5-1
linux-steam-integration 0.7.3-7
lua53 5.3.6-2
mangohud 0.7.0-2
nasm 2.16.01-1
openfortivpn 1.20.5-1
python-evdev 1.6.1-2
python-protobuf 24.3-1
python-pypresence 4.3.0-1
python-rx 3.2.0-4
re2 1:20230901-2
spdlog 1.12.0-1
sysfsutils 2.1.1-1
tinycompress 1.2.8-2
winetricks 20230212-1
xorg-font-util 1.4.1-1
xorg-util-macros 1.20.0-2
xorg-xgamma 1.0.7-1
xtrans 1.5.0-1
( 9/11) Checking for .pacnew and .pacsave files...
.pac* files found:
Please check and merge
(10/11) Performing snapper post snapshots for the following configurations...
==> root: 143
(11/11) Syncing all file systems...

╭─XXX@XXX in /etc/X11/xinit🔒 as 🧙 took 34s
╰─λ sudo update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/garuda-dr460nized/theme.txt
Found linux image: /boot/vmlinuz-linux-zen
Found initrd image: /boot/intel-ucode.img /boot/initramfs-linux-zen.img
Found fallback initrd image(s) in /boot:  intel-ucode.img initramfs-linux-zen-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Garuda Linux on /dev/nvme1n1p2
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2023-10-17 16:11:19 | @/.snapshots/143/snapshot | post   | linux-zen                    |
Found snapshot: 2023-10-17 16:10:52 | @/.snapshots/142/snapshot | pre    | pacman -S linux-zen          |
Found snapshot: 2023-10-16 15:30:02 | @/.snapshots/141/snapshot | post   | qt5-charts stacer            |
Found snapshot: 2023-10-16 15:30:00 | @/.snapshots/140/snapshot | pre    | pacman -S --noconfirm stacer |
Found snapshot: 2023-10-16 02:09:33 | @/.snapshots/139/snapshot | single | Test Post Grub Update        |
Found 5 snapshot(s)
Unmount /tmp/grub-btrfs.JcG8JvKrc3 .. Success
Found memtest86+ image: /boot/memtest86+/memtest.bin
/usr/bin/grub-probe: warning: unknown device type nvme1n1.

EDIT: Tested, no luck, still failed switch root. hmmm, have to give some thought to possible causes. Img file permissions??

When you test booting the snapshots, are you testing the fallback image as well?


Please, no pictures from terminal.
Always post command, too.

cat /etc/fstab

That would be a great help.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.