Can't boot into snapshots from GRUB menu

I can’t figure out why it’s doing this whatsoever and would really like to fix… whatever’s going on here. It’s been doing this for a few months now but I’ve just been kinda relying on downgrade where necessary. However, I’d like to fix this at this point but I’m not sure what’s going on. I select a snapshot from the grub menu, try and boot into it and it’ll hang after a bit. I can swap to a different TTY and see a couple errors while it hangs. There are two about samba daemons failing to start but also another with rotate log files, running systemctl status logrotate.service seems to reveal that it’s in read only mode. I assume this to be the problem but am not certain how to fix it and need some help.

System:
Kernel: 6.9.6-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 14.1.1
clocksource: tsc avail: hpet,acpi_pm
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
root=UUID=9cb34755-a24b-463c-9215-498d1e3ea350 rw rootflags=subvol=@
quiet quiet rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
ibt=off
Desktop: KDE Plasma v: 6.1.0 tk: Qt v: N/A info: frameworks v: 6.3.0
wm: kwin_x11 tools: avail: swayidle,swaylock vt: 2 dm: SDDM Distro: Garuda
base: Arch Linux
Machine:
Type: Desktop System: ASUS product: N/A v: N/A serial: <superuser required>
Mobo: ASUSTeK model: TUF GAMING B550-PLUS WIFI II v: Rev X.0x
serial: <superuser required> part-nu: SKU uuid: <superuser required>
UEFI: American Megatrends v: 3205 date: 08/14/2023
CPU:
Info: model: AMD Ryzen 7 5800X3D bits: 64 type: MT MCP arch: Zen 3+ gen: 4
level: v3 note: check built: 2022 process: TSMC n6 (7nm) family: 0x19 (25)
model-id: 0x21 (33) stepping: 2 microcode: 0xA20120A
Topology: cpus: 1x cores: 8 tpc: 2 threads: 16 smt: enabled cache:
L1: 512 KiB desc: d-8x32 KiB; i-8x32 KiB L2: 4 MiB desc: 8x512 KiB
L3: 96 MiB desc: 1x96 MiB
Speed (MHz): avg: 3740 high: 4401 min/max: 2200/4549 boost: enabled
scaling: driver: acpi-cpufreq governor: performance cores: 1: 4153 2: 3555
3: 3593 4: 3551 5: 3694 6: 3400 7: 3590 8: 4401 9: 3548 10: 3593 11: 3972
12: 4399 13: 3591 14: 3584 15: 3633 16: 3592 bogomips: 108600
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3
Vulnerabilities: <filter>
Graphics:
Device-1: AMD Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]
vendor: Sapphire driver: amdgpu v: kernel arch: RDNA-1 code: Navi-1x
process: TSMC n7 (7nm) built: 2019-20 pcie: gen: 4 speed: 16 GT/s
lanes: 16 ports: active: DP-2,HDMI-A-1 empty: DP-1,DP-3 bus-ID: 05:00.0
chip-ID: 1002:731f class-ID: 0300
Display: x11 server: X.Org v: 21.1.13 with: Xwayland v: 24.1.0
compositor: kwin_x11 driver: X: loaded: amdgpu unloaded: modesetting,radeon
alternate: fbdev,vesa dri: radeonsi gpu: amdgpu display-ID: :0 screens: 1
Screen-1: 0 s-res: 3840x1080 s-dpi: 96 s-size: 1016x285mm (40.00x11.22")
s-diag: 1055mm (41.54")
Monitor-1: DP-2 mapped: DisplayPort-1 pos: primary,right model: VG245
serial: <filter> built: 2017 res: 1920x1080 dpi: 92 gamma: 1.2
size: 531x299mm (20.91x11.77") diag: 609mm (24") ratio: 16:9 modes:
max: 1920x1080 min: 720x400
Monitor-2: HDMI-A-1 mapped: HDMI-A-0 pos: left model: LG (GoldStar) W2442
serial: <filter> built: 2010 res: 1920x1080 hz: 60 dpi: 92 gamma: 1.2
size: 531x299mm (20.91x11.77") diag: 609mm (24") ratio: 16:9 modes:
max: 1920x1080 min: 720x400
API: EGL v: 1.5 hw: drv: amd radeonsi platforms: device: 0 drv: radeonsi
device: 1 drv: swrast surfaceless: drv: radeonsi x11: drv: radeonsi
inactive: gbm,wayland
API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.1.2-arch1.1
glx-v: 1.4 direct-render: yes renderer: AMD Radeon RX 5700 XT (radeonsi
navi10 LLVM 17.0.6 DRM 3.57 6.9.6-zen1-1-zen) device-ID: 1002:731f
memory: 7.81 GiB unified: no
API: Vulkan v: 1.3.279 layers: 14 device: 0 type: discrete-gpu name: AMD
Radeon RX 5700 XT (RADV NAVI10) driver: mesa radv v: 24.1.2-arch1.1
device-ID: 1002:731f surfaces: xcb,xlib device: 1 type: cpu name: llvmpipe
(LLVM 17.0.6 256 bits) driver: mesa llvmpipe v: 24.1.2-arch1.1 (LLVM
17.0.6) device-ID: 10005:0000 surfaces: xcb,xlib
Audio:
Device-1: AMD Navi 10 HDMI Audio driver: snd_hda_intel v: kernel pcie:
gen: 4 speed: 16 GT/s lanes: 16 bus-ID: 05:00.1 chip-ID: 1002:ab38
class-ID: 0403
Device-2: AMD Starship/Matisse HD Audio vendor: ASUSTeK
driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
bus-ID: 09:00.4 chip-ID: 1022:1487 class-ID: 0403
Device-3: C-Media Blue Snowball driver: hid-generic,snd-usb-audio,usbhid
type: USB rev: 1.1 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-10:5
chip-ID: 0d8c:0005 class-ID: 0300 serial: <filter>
API: ALSA v: k6.9.6-zen1-1-zen status: kernel-api with: aoss
type: oss-emulator tools: N/A
Server-1: PipeWire v: 1.0.7 status: active with: 1: pipewire-pulse
status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
4: pw-jack type: plugin tools: pactl,pw-cat,pw-cli,wpctl
Network:
Device-1: MEDIATEK MT7921 802.11ax PCI Express Wireless Network Adapter
vendor: AzureWave driver: mt7921e v: kernel pcie: gen: 2 speed: 5 GT/s
lanes: 1 bus-ID: 06:00.0 chip-ID: 14c3:7961 class-ID: 0280
IF: wlp6s0 state: up mac: <filter>
Device-2: Realtek RTL8125 2.5GbE vendor: ASUSTeK driver: r8169 v: kernel
pcie: gen: 2 speed: 5 GT/s lanes: 1 port: e000 bus-ID: 07:00.0
chip-ID: 10ec:8125 class-ID: 0200
IF: enp7s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
IF-ID-1: virbr0 state: down mac: <filter>
Info: services: NetworkManager, smbd, sshd, systemd-timesyncd,
wpa_supplicant
Bluetooth:
Device-1: IMC Networks Wireless_Device driver: btusb v: 0.8 type: USB
rev: 2.1 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-5:2 chip-ID: 13d3:3563
class-ID: e001 serial: <filter>
Report: btmgmt ID: hci0 rfk-id: 0 state: up address: <filter> bt-v: 5.2
lmp-v: 11 status: discoverable: no pairing: no class-ID: 6c0104
Report-ID: ID: hci1 rfk-id: 1 state: up address: <filter> bt-v: 4.1
lmp-v: 7 status: discoverable: no pairing: no class-ID: 6c0104
Drives:
Local Storage: total: 5 TiB used: 3.84 TiB (76.7%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/sda maj-min: 8:0 vendor: Western Digital
model: WD10EZEX-08WN4A0 size: 931.51 GiB block-size: physical: 4096 B
logical: 512 B speed: 6.0 Gb/s tech: HDD rpm: 7200 serial: <filter>
fw-rev: 1A02 scheme: GPT
ID-2: /dev/sdb maj-min: 8:16 vendor: Seagate model: ST3500418AS
size: 465.76 GiB block-size: physical: 512 B logical: 512 B speed: 3.0 Gb/s
tech: HDD rpm: 7200 serial: <filter> fw-rev: CC38 scheme: GPT
ID-3: /dev/sdc maj-min: 8:32 vendor: Western Digital
model: WD30EZRZ-00GXCB0 size: 2.73 TiB block-size: physical: 4096 B
logical: 512 B speed: 6.0 Gb/s tech: HDD rpm: 5400 serial: <filter>
fw-rev: 0A80 scheme: GPT
ID-4: /dev/sdd maj-min: 8:48 vendor: Crucial model: CT1000MX500SSD4
size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
tech: SSD serial: <filter> fw-rev: 023 scheme: GPT
Partition:
ID-1: / raw-size: 931.22 GiB size: 931.22 GiB (100.00%)
used: 804.2 GiB (86.4%) fs: btrfs dev: /dev/sdd2 maj-min: 8:50
ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
used: 25.8 MiB (8.6%) fs: vfat dev: /dev/sdd1 maj-min: 8:49
ID-3: /home raw-size: 931.22 GiB size: 931.22 GiB (100.00%)
used: 804.2 GiB (86.4%) fs: btrfs dev: /dev/sdd2 maj-min: 8:50
ID-4: /var/log raw-size: 931.22 GiB size: 931.22 GiB (100.00%)
used: 804.2 GiB (86.4%) fs: btrfs dev: /dev/sdd2 maj-min: 8:50
ID-5: /var/tmp raw-size: 931.22 GiB size: 931.22 GiB (100.00%)
used: 804.2 GiB (86.4%) fs: btrfs dev: /dev/sdd2 maj-min: 8:50
Swap:
Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default) zswap: no
ID-1: swap-1 type: zram size: 31.22 GiB used: 13.5 MiB (0.0%)
priority: 100 comp: zstd avail: lzo,lzo-rle,lz4,lz4hc,842 max-streams: 16
dev: /dev/zram0
Sensors:
System Temperatures: cpu: 30.0 C mobo: 37.0 C gpu: amdgpu temp: 56.0 C
mem: 62.0 C
Fan Speeds (rpm): fan-1: 921 fan-2: 521 fan-3: 520 fan-4: 0 fan-5: 0
fan-6: 596 fan-7: 522 gpu: amdgpu fan: 591
Info:
Memory: total: 32 GiB available: 31.22 GiB used: 8.15 GiB (26.1%)
Processes: 480 Power: uptime: 3m states: freeze,mem,disk suspend: deep
avail: s2idle wakeups: 0 hibernate: platform avail: shutdown, reboot,
suspend, test_resume image: 12.45 GiB services: org_kde_powerdevil,upowerd
Init: systemd v: 256 default: graphical tool: systemctl
Packages: pm: pacman pkgs: 2641 libs: 644 tools: octopi,pamac,paru
Compilers: clang: 17.0.6 gcc: 14.1.1 Shell: garuda-inxi default: Bash
v: 5.2.26 running-in: konsole inxi: 3.3.35
Garuda (2.6.26-1):
System install date:     2022-05-10
Last full system update: 2024-06-26
Is partially upgraded:   No
Relevant software:       snapper NetworkManager mkinitcpio
Windows dual boot:       Probably (Run as root to verify)
Failed units:

I had another go at it and it appears that after I select a snapshot, at the top (which I didn’t quite realize at first) there were some additional errors, systemd[1]; Failed to start Remount Root and Kernel File Systems and hid-generic 003:0D8C:0005.0006: No inputs registered. leaving

Did you do any hardware (USB, etc.) change around the time it started to stop booting from snapshots?

You can get the journal log from the failed boot sessions.

journalctl -b -1 # for the previous boot
journalctl -b -2 # for one before the previous boot, etc.

Post logs at garuda bin, or other web paste service, and post the links.

1 Like

I did need to replace my motherboard and CPU back in March but I did some looking into my previous experiences about this issue, and I first noticed it at the start of May. I suppose it is possible I didn’t notice until then because I had no need to try and restore.

Here is the output of journalctl -b 0, my current session I am on: Garuda's PrivateBin
This is journalctl -b -1: Garuda's PrivateBin
And journalctl -b -2: Garuda's PrivateBin

Notably, -b -1 is significantly shorter compared to the others. I booted in, swapped to TTY 2, filled in my credentials and shut it down from the command line.

1 Like

The only time i had problems loading snap shots, was when i ran more than one kernel, and did not have the running kernel in first position on the grub menu.

Do you use more than one kernel? And has the running kernel not been your first grub menu option?

No, I have only been using the zen kernel since I installed.

The point of previous boot sessions is not for curiosity. It is to explain how to get the correct boot session.

We would need the failed boot session after choosing a snapshot.
The previous boot session (-b -1), does not look like it is trying to boot to a snapshot (or I need some more sleep :smiling_face: ).
A succesful boot is not interesting :wink: .

1 Like

Entirely fair, I’m a dumbass at the best of times so I figured I’d include it just in case.

-b -1 is from when I attempted to boot into a snapshot, or was the only log from that point in time. I can try again just in case all the same.

Okay so… I tried booting into one again to grab the log but journalctl isn’t showing me the session. The output of journalctl --list-boots shows the following

journalctl --list-boots
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
-5 99cc37c100c94176b0901a2b7a6a2056 Fri 2024-06-28 14:12:43 CST Fri 2024-06-28 16:48:44 CST
-4 7638c3ec2dd049aabf60a34623a7d133 Fri 2024-06-28 17:02:25 CST Fri 2024-06-28 17:25:40 CST
-3 ebf11e65c5644df38419be37cdb65156 Fri 2024-06-28 17:26:13 CST Fri 2024-06-28 17:26:14 CST
-2 499d66bbd188477da06fe158864e3cc7 Fri 2024-06-28 17:30:13 CST Fri 2024-06-28 19:37:03 CST
-1 da61a957d2ca4bcd86cd4b469ef6d0b4 Fri 2024-06-28 19:39:32 CST Fri 2024-06-28 19:41:33 CST
0 8a81e7b6682a44c0b4d0a0bf6d620061 Fri 2024-06-28 19:49:42 CST Fri 2024-06-28 19:50:52 CST

I tried a couple times to get one, at first between -2 and -1 and then again between -1 and 0. My only guess is that the lack has something to do with my system being in read-only mode so it cannot actually create it.

Okay so I had some free time to do some more looking into this.

I booted back into a snapshot and tabbed over to a TTY and logged in and opened up journalctl -bb 0 and grabbed some pics of the log I thought might be relevant, which are below. There were some errors about the display server failing to start, though I did get the idea to try running startplasma-wayland and that seems to have worked in getting everything up visually. It was notably slow however and kept throwing errors at me about everything being in read-only but I did eventually get in and was presented with the option from Snapper to restore the snapshot.

Summary










I couldn’t upload two and they are instead here

Do you have unmerged pacnew files?

What do you have on the HOOKS line in /etc/mkinitcpio.conf?

Assuming that sudo find /etc -type f -name "*.pacnew" is the correct way to find them, it appears that I have a few, yes.

As for the hooks line, it reads as follows: HOOKS="base udev autodetect modconf block keyboard keymap consolefont plymouth filesystems grub-btrfs-overlayfs"

Do you have a pacnew for /etc/mkinitcpio by chance? :face_with_monocle:

It looks like you are missing the microcode hook. There was an upstream change a couple months ago to bundle the microcode into the initramfs. See this announcement here: Arch Linux - News: mkinitcpio hook migration and early microcode

Add the microcode hook between autodetect and modconf, so the whole line looks like this:

HOOKS="base udev autodetect microcode modconf block keyboard keymap consolefont plymouth filesystems grub-btrfs-overlayfs"

Next, check in /etc/mkinitcpio.d/ to see if you have any kernel preset files in there. If you do, edit the files to remove this line:

ALL_microcode=(/boot/*-ucode.img)

Just deleting the line is fine. If it is the only line you can delete the file altogether.

After making those changes, regenerate the initramfs.

sudo mkinitcpio -P

Next, edit /etc/default/grub and add this line somewhere in the file:

GRUB_EARLY_INITRD_LINUX_STOCK=''

Regenerate the Grub configuration file.

sudo update-grub

Finally, generate a new snapshot (either manually or by adding/removing a package) and try booting into it.

1 Like

Looks like I do, yes. And opening it, it also looks like it’s got the mirocode hook in the right spot too.

As for the line in the kernel preset, it doesn’t look like that was present. I’ll make the other changes and update this once I’ve gotten them done and all.

As for the .pacnew files, are the rest of any concern as well?

You should examine them to see what the changes are. In some cases, pacnew files have changes which should be incorporated into your version of the config.

Be sure to not overwrite the existing file, unless you are absolutely certain you know what you are doing. See also this article:

pacman/Pacnew and Pacsave - ArchWiki

Note: Even when you have not edited a file manually, do not blindly overwrite a live file with its .pacnew version. Always compare and merge the changes manually.

And this forum thread: Dealing With .pacnew Files / Pacman & Package Upgrade Issues / Arch Linux Forums

Using a tool like Meld or Diff is handy for pacnew files because it makes it easy to see what changes have been made. The default Fish config has a handy alias built-in for using Meld, see this line here: garuda-fish-config/config.fish · main · Garuda Linux 🦅 / PKGBUILDs · GitLab

If you have this line you can just run pacdiff and you will get this full command from the alias:

sudo -H DIFFPROG=meld pacdiff
3 Likes

Well, so far no luck after following those steps, I’m afraid. I will look into the pacnew stuff you’ve linked here though.

Post the outputs of mkinitcpio -P and update-grub into the thread so we can take a look.

sudo mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-zen.preset: 'default'
==> Using default configuration file: '/etc/mkinitcpio.conf'
-> -k /boot/vmlinuz-linux-zen -g /boot/initramfs-linux-zen.img
==> Starting build: '6.9.8-zen1-1-zen'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [microcode]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-zen.img'
-> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-zen.preset: 'fallback'
==> Using default configuration file: '/etc/mkinitcpio.conf'
-> -k /boot/vmlinuz-linux-zen -g /boot/initramfs-linux-zen-fallback.img -S autodetect
==> Starting build: '6.9.8-zen1-1-zen'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [microcode]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-zen-fallback.img'
-> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful
sudo update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/garuda-dr460nized/theme.txt
Found linux image: /boot/vmlinuz-linux-zen
Found initrd image: /boot/initramfs-linux-zen.img
Found fallback initrd image(s) in /boot:  initramfs-linux-zen-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Windows Boot Manager on /dev/sdd1@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2024-07-09 11:12:18 | @/.snapshots/1038/snapshot | post | botan botan2 corectrl discord dxvk-mingw-git gdb gdb-common keepassxc li |
Found snapshot: 2024-07-09 11:11:52 | @/.snapshots/1037/snapshot | pre  | pacman -Su                          |
Found snapshot: 2024-07-08 10:36:13 | @/.snapshots/1036/snapshot | post | abaddon-git b43-fwcutter blender btrfsmaintenance ca-certificates-mozill |
Found snapshot: 2024-07-08 10:35:18 | @/.snapshots/1035/snapshot | pre  | pacman -Su                          |
Found snapshot: 2024-07-04 13:58:15 | @/.snapshots/1034/snapshot | post | abaddon-git adobe-source-code-pro-fonts aha akonadi akonadi-calendar ako |
Found snapshot: 2024-07-04 13:57:04 | @/.snapshots/1033/snapshot | pre  | pacman -Su                          |
Found snapshot: 2024-06-26 17:33:53 | @/.snapshots/1032/snapshot | post | abaddon-git balena-etcher blender boost boost-libs bullet-dp electron31  |
Found snapshot: 2024-06-26 17:33:43 | @/.snapshots/1031/snapshot | pre  | pacman -Su                          |
Found 8 snapshot(s)
Unmount /tmp/grub-btrfs.EGFnpPhWCZ .. Success
Found memtest86+ image: /boot/memtest86+/memtest.bin
done

That looks fine as far as I can tell. You added the GRUB_EARLY_INITRD_LINUX_STOCK='' line to /etc/default/grub before regenerating right?

When you tested booting a snapshot, did you use a new snapshot (i.e., one that was created after making these changes)?

Alternately you can update grub-btrfs.cfg by running the script in /etc/grub.d:

sudo /etc/grub.d/41_snapshots-btrfs

Then regenerate the Grub config again.

sudo update-grub
1 Like