Cannot boot into snapshots after the update

Issues & Assistance
1

Hey guys,
so i did an update today, it installed KDE 5.27...i accidentally clicked on yes regarding the theme migration and it destroyed my rice (i thought it would just change the things, not activate them).

I didn't want to make all the necessary changes right now and wanted to just reboot into a snapshot from before the update and deal with that stuff later...

Interestingly, after that update i cannot boot into ANY snapshot anymore. Regardless of how old or new it is, regardless if i use the fallback or lts kernel...I know that i could boot into snapshots yesterday BEFORE the update just fine.

This is what i get:

IMG20230221143745
IMG202302211437451920×864 108 KB

I then made a new manual shapshot with the update in place (so as it is right now) and can't boot into that one aswell...so yeah my snapshots are broken somehow. I really really want these to work... can someone help me out here?

System:
  Kernel: 6.1.12-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 12.2.1
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen
    root=UUID=c3ef0d61-a4ef-469b-96b1-b7a7583d021e rw [email protected]
    quiet rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
    ibt=off
  Desktop: KDE Plasma v: 5.27.0 tk: Qt v: 5.15.8 info: latte-dock, polybar
    wm: kwin_x11 vt: 1 dm: SDDM Distro: Garuda Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: ASUSTeK model: ROG STRIX X570-I GAMING v: Rev X.0x
    serial: <superuser required> BIOS: American Megatrends v: 4403
    date: 04/27/2022
CPU:
  Info: model: AMD Ryzen 7 5800X3D bits: 64 type: MT MCP arch: Zen 3+ gen: 4
    level: v3 note: check built: 2022 process: TSMC n6 (7nm) family: 0x19 (25)
    model-id: 0x21 (33) stepping: 2 microcode: 0xA20120A
  Topology: cpus: 1x cores: 8 tpc: 2 threads: 16 smt: enabled cache:
    L1: 512 KiB desc: d-8x32 KiB; i-8x32 KiB L2: 4 MiB desc: 8x512 KiB
    L3: 96 MiB desc: 1x96 MiB
  Speed (MHz): avg: 3509 high: 3600 min/max: 2200/4549 boost: enabled
    scaling: driver: acpi-cpufreq governor: performance cores: 1: 3599 2: 3593
    3: 3400 4: 3588 5: 3590 6: 3598 7: 3400 8: 3400 9: 3400 10: 3590 11: 3599
    12: 3593 13: 3600 14: 3400 15: 3400 16: 3400 bogomips: 108797
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
  Vulnerabilities: <filter>
Graphics:
  Device-1: NVIDIA GA104 [GeForce RTX 3070] vendor: ASUSTeK driver: nvidia
    v: 525.89.02 alternate: nouveau,nvidia_drm non-free: 525.xx+
    status: current (as of 2023-02) arch: Ampere code: GAxxx
    process: TSMC n7 (7nm) built: 2020-22 pcie: gen: 3 speed: 8 GT/s lanes: 16
    link-max: gen: 4 speed: 16 GT/s bus-ID: 0a:00.0 chip-ID: 10de:2484
    class-ID: 0300
  Display: x11 server: X.Org v: 21.1.7 with: Xwayland v: 22.1.8 compositors:
    1: kwin_x11 2: Picom v: git-f70d0 driver: X: loaded: nvidia gpu: nvidia
    display-ID: :0 screens: 1
  Screen-1: 0 s-res: 4520x1920 s-dpi: 51 s-size: 2251x301mm (88.62x11.85")
    s-diag: 2271mm (89.41")
  Monitor-1: DP-0 pos: top-left res: 1080x1920 hz: 60 dpi: 92
    size: 299x531mm (11.77x20.91") diag: 609mm (23.99") modes: N/A
  Monitor-2: DP-2 pos: primary,bottom-r res: 3440x1440 dpi: 109
    size: 800x330mm (31.5x12.99") diag: 865mm (34.07") modes: N/A
  API: OpenGL v: 4.6.0 NVIDIA 525.89.02 renderer: NVIDIA GeForce RTX
    3070/PCIe/SSE2 direct-render: Yes
Audio:
  Device-1: NVIDIA GA104 High Definition Audio vendor: ASUSTeK
    driver: snd_hda_intel v: kernel pcie: gen: 3 speed: 8 GT/s lanes: 16
    link-max: gen: 4 speed: 16 GT/s bus-ID: 0a:00.1 chip-ID: 10de:228b
    class-ID: 0403
  Device-2: AMD Starship/Matisse HD Audio vendor: ASUSTeK
    driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
    bus-ID: 0c:00.4 chip-ID: 1022:1487 class-ID: 0403
  Sound API: ALSA v: k6.1.12-zen1-1-zen running: yes
  Sound Interface: sndio v: N/A running: no
  Sound Server-1: PulseAudio v: 16.1 running: no
  Sound Server-2: PipeWire v: 0.3.66 running: yes
Network:
  Device-1: Intel Wi-Fi 6 AX200 driver: iwlwifi v: kernel pcie: gen: 2
    speed: 5 GT/s lanes: 1 bus-ID: 05:00.0 chip-ID: 8086:2723 class-ID: 0280
  IF: wlp5s0 state: down mac: <filter>
  Device-2: Intel I211 Gigabit Network vendor: ASUSTeK driver: igb v: kernel
    pcie: gen: 1 speed: 2.5 GT/s lanes: 1 port: f000 bus-ID: 06:00.0
    chip-ID: 8086:1539 class-ID: 0200
  IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
Bluetooth:
  Device-1: ASUSTek Broadcom BCM20702A0 Bluetooth type: USB driver: btusb
    v: 0.8 bus-ID: 5-4.1.1:7 chip-ID: 0b05:17cb class-ID: fe01 serial: <filter>
  Report: bt-adapter ID: hci0 rfk-id: 1 state: down
    bt-service: enabled,running rfk-block: hardware: no software: yes
    address: <filter>
Drives:
  Local Storage: total: 3.19 TiB used: 1.51 TiB (47.2%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:2 vendor: Samsung model: SSD 970 EVO 250GB
    size: 232.89 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
    lanes: 4 type: SSD serial: <filter> rev: 1B2QEXE7 temp: 41.9 C scheme: MBR
  ID-2: /dev/nvme1n1 maj-min: 259:0 vendor: SK Hynix
    model: HFM256GDHTNG-8310A size: 238.47 GiB block-size: physical: 512 B
    logical: 512 B speed: 15.8 Gb/s lanes: 2 type: SSD serial: <filter>
    rev: 80070C00 temp: 52.9 C scheme: MBR
  ID-3: /dev/sda maj-min: 8:0 vendor: Samsung model: SSD 870 EVO 2TB
    size: 1.82 TiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
    type: SSD serial: <filter> rev: 2B6Q scheme: GPT
  ID-4: /dev/sdb maj-min: 8:16 vendor: Crucial model: CT1000MX500SSD1
    size: 931.51 GiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
    type: SSD serial: <filter> rev: 033 scheme: GPT
Partition:
  ID-1: / raw-size: 232.88 GiB size: 232.88 GiB (100.00%)
    used: 138.89 GiB (59.6%) fs: btrfs dev: /dev/nvme0n1p1 maj-min: 259:3
  ID-2: /home raw-size: 232.88 GiB size: 232.88 GiB (100.00%)
    used: 138.89 GiB (59.6%) fs: btrfs dev: /dev/nvme0n1p1 maj-min: 259:3
  ID-3: /var/log raw-size: 232.88 GiB size: 232.88 GiB (100.00%)
    used: 138.89 GiB (59.6%) fs: btrfs dev: /dev/nvme0n1p1 maj-min: 259:3
  ID-4: /var/tmp raw-size: 232.88 GiB size: 232.88 GiB (100.00%)
    used: 138.89 GiB (59.6%) fs: btrfs dev: /dev/nvme0n1p1 maj-min: 259:3
Swap:
  Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
  ID-1: swap-1 type: zram size: 62.71 GiB used: 0 KiB (0.0%) priority: 100
    dev: /dev/zram0
Sensors:
  System Temperatures: cpu: 49.5 C mobo: 47.0 C gpu: nvidia temp: 58 C
  Fan Speeds (RPM): fan-1: 881 fan-2: 1836 fan-5: 947 fan-7: 0 gpu: nvidia
    fan: 60%
Info:
  Processes: 483 Uptime: 2m wakeups: 0 Memory: 62.71 GiB used: 5.21 GiB (8.3%)
  Init: systemd v: 253 default: graphical tool: systemctl Compilers:
  gcc: 12.2.1 alt: 11 clang: 15.0.7 Packages: 2233 pm: pacman pkgs: 2207
  libs: 539 tools: pamac,paru pm: flatpak pkgs: 26 Shell: fish v: 3.6.0
  running-in: alacritty inxi: 3.3.25
Garuda (2.6.15-1):
  System install date:     2023-02-18
  Last full system update: 2023-02-21
  Is partially upgraded:   No
  Relevant software:       snapper NetworkManager mkinitcpio nvidia-dkms
  Windows dual boot:       <superuser required>
  Failed units:

PS: Ignore the "System Install date:" that's completely wrong. I don't know why though. This installation is ATLEAST 9 months old at this point.

2

Booting a snapshot will not revert your theming anyway, as snapshotting the home folder is not default behavior in Garuda Linux. .bak files were created in your .config folder. You can restore them to restore your old theme.

2 Likes
3

Yeah i remembered that aswell...wasn't really awake when this happened (thats why i wanted to revert it in the first place)

I now riced everything back to what i wanted, so i don't really need to revert anything.

But i need to have my snapshots working again. Those saved my butt more than a dozen times already and i need the peace of mind that i can revert to a working system at any point

4

Your snapshots work, you just can't boot into any of them. Can you share your /boot/grub/grub-btrfs.cfg?

5
6

I also have this issue on 2 of 3 machines I tested it on, one of which I installed the OS (KDE-Lite) only 4 days ago. (it was an immediate issue) Let me know if want any info from me to compare, if that's helpful, such as garuda-inxi, grub configs, etc.

7

Sadly i don't think i have the necessary understanding to fix this problem myself, even with additional information from you to compare with.

So i must rely on some more tech-savvy people to find out what might be the issue.
@TNE have you found the time to look into the config i provided?

Oh and btw. i just browsed through every single snapshot via dolphin in root mode, and everyone of them contains /sbin/init

So....

8

This recent thread bears some similarities to yours:

Try switching to a TTY and restoring a snapshot from the command line as described in that thread and see if you have any better luck.

9

Well the difference is i CAN boot. There is 0 problem with my current system. It ONLY effects my snapshots.

And tbh i don't want to risk my installation by restoring a snapshot that doesn't work.

10

If you boot into a snapshot that "doesn't work", just don't restore it. :man_shrugging:

How did you make out booting into your snapshots from the command line? Or did you not even try for some reason?

What is the problem we are trying to solve here?

11

It's not a specific snapshot that doesn't work. It's all snapshots, regardless of how old or new, that don't work.

So as of right now, i cannot use the snapshot functionality and cannot rely on my system backups. So if i were to brick my installation right now, my system would be lost and i could not restore any snapshot.

I didn't try "booting" into a snapshot from commandline because i don't know how. You said i should RESTORE a snapshot from TTY and that is something i don't want to do right now as my installation is working right now and i don't want to brick it by restoring snapshots that don't work.

To summarize:

My installation works fine, i can boot it, i can use it, 100% works. But if i go into the Snapshot section in grub when booting, and trying to boot in ANY of the snapshots (even ones i made just seconds ago on my working system) i get the "bailing out bcs sbin/init missing" error.

I can then easily reboot into my system and use the current installation, but i don't have access to ANY of my snapshots, making the backup / snapshot system broken right now.

That stresses me out because i need to have a working snapshot system in the case i break my installation at some point...

12

until this is fixed, back up your installed packages via comm -23 <(pacman -Qqett | sort) <(pacman -Qqgg base-devel | sort | uniq)

This will not save an installation but at least allows you to go back to a similar state. I suggest making regular non-btrfs backups of your home directory too.

13

I had the same problem.

Fortunately, I was also able to start my PC and then started BTRFS Assistant.

There on
Snapper-> Brows/Restore-> Snapshot selected and clicked on restore.
Restart and everything was good.

Strangely enough, since then the snapshots via Grub work again.

14

I am doing regular back-in-time snapshots of my home directory and i am also doing lists of all my installed packages.

Maybe i try what @bluepower said, which is basically the same as @BluishHumility (both are blue, coincidence? :smiley: ) and just restore one of those snapshots from my working system and see if it fixes it...

15

yeah alright, so restoring a snapshot from my working system DOES indeed work and does NOT brick my boot lol

So that means i have 95% working snapshots, that will save me in any case that allows me to still access my system. If i break that however...i cannot access this...

I assume i would still be able to boot into a live-usb of garuda and chroot into my install maybe? And restore the snapshot from there? But that is certainly a much more in-depth process than simply chosing the last working snapshot in grub.

@bluepower - sadly restoring a snapshot did not repair the grub-functionality :frowning:

16

Yes, you can always restore a snapshot from the live system. That would be sort of the "original" way to restore a snapshot. The method is described here: Snapper - ArchWiki

It obviously more clunky than the grub-btrfs system where you just choose a snapshot from a menu and it's done, but it is a good backup plan. You can also use Btrfs Assistant from the live environment.

Regarding the actual issue, I cannot figure out how to reproduce this issue. The /boot/grub/grub-btrfs.cfg posted above does not contain any obvious clues to me, all those snapshots look pretty much normal.

This seems especially odd to me:

I brought up a fresh KDE lite installation yesterday, and booting to snapshots is working out of the box. I even have non-standard root subvolumes set up and Grub still automatically finds the snapshots and booting to them works fine.

Now that I think of it, I guess I did reinstall Grub and regenerate the configuration file on the back of changing the default subvolumes. It seems like a long shot, but have you tried doing that yet?

sudo grub-install --no-nvram
sudo update-grub
3 Likes
17

That was not me :smiley: I didn't install KDE lite 4 days ago.

sudo grub-install --no-nvram

Gives me the following error:

Installing for i386-pc platform.
grub-install: error: install device isn't specified.

Updating grub said it found 28 snapshots, memtext86+ and my linux images and fallback...But this problem persists beyond a kernel update that i made via garuda-update that then did an grub-update aswell...

18

For you, the command is like this:

sudo grub-install --target=i386-pc /dev/sdX

sdX is the disk (not a partition--so sda, not sda1 for example). Then regenerate the configuration file afterward with sudo update-grub.

Like I said, it's a long shot anyway. :man_shrugging:

19

for me it wasn't sda but rather nvme0n1 but that didn't solve it either..

well now i am just deleting every single snapshot, scrubbing and rebalancing my btrfs and then i'll do a new grub-install and update.

Then i'll just create a completely new snapshot and see if that works.

20

That didn't fix it either. But i stumbled across something that may or may not be relevant...

doing snapper list showed me no snapshots after my deletion-oddesey was done. But grub-update said it found two snapshots... It said N/A for both the name and description. The snapper gui doesn't show those two aswell but the grub menu when booting does show those two.... But grub-update did find them and said it put those into the menu....maybe those two contain the borked sbin/init file ?

How could i go about deleting those two if even snapper itself doesn't find them

2023-02-25_22-09
2023-02-25_22-091241×712 131 KB

These are not in the .snapshots folder aswell...backup never seen this folder and don't know where it would be