Garuda worked flawless for months now. After update I'm being dropped to emergency shell :-( How to fix?

Hello.

Post your terminal/konsole in- and output as text (no pictures) from:

garuda-inxi cannot copy the output currently.

This is what is being shown after I enter my LUKS password which is being accepted.

I don't really understand as I just did the regular updates and they finished without issues but now my grub menu is set back to generic and my custom grub vanished and I get dropped to this emergency shell.

I also cannot add images here right now so here is a link to the error message and blkid output.

Thanks for your help in advance. I am super grateful for any hints I can get.

I googled for quite a while and it seems I have to reboot with my ISO and then chroot and then I can fix it.

But the posts are all for different linux flavors and I am really scared to do something wrong as I had everything setup so perfectly over weeks.

Thanks!

EDIT: I can see all my BTRFS snapshots and tried to boot the one before the recent update but no dice.

I would like to know that in advance too. I lost my kali installation to this problem. I do not want it to happen here. Please if someone know the answer, please tell.

Create a Garuda USB stick, boot from it, then install arch-install-scripts on the live system. Then chroot to your current install. Check the arch wiki for more info on how to do this. Next run sudo pacman -Syyu. This should fix your installed system. If not, try pacman -Qqn | pacman -S - This will reinstall all packages and refresh your entire system.

Note: in order to decrypt the LUKS drive, additional steps maybe needed before mounting it.

Thanks for the suggestions. I will try that soon and log all outputs.

But again the issue started with a normal "yay" system upgrade ( and I followed it the whole time and there were zero errors) which is what really is nerve-wracking :-). I mean I did that every other day for months and suddenly it cracks...

I suspect there is something wrong with fstab or just grub.

The grub menu it shows before throwing the emergency shell is not the my customized grub I usually have.

Will report back when I am able to access all the relevant files.

Okay for chrooting into btrfs I'll go with these instructions:
https://blog.arch-libertas.net/2021/07/chroot-v2-0-btrfs-inside/

still getting my heard around how I will unlock the luks partition before chrooting..
hopefully it will just work via disks gui..

You're right. Check if your /etc/fstab matches the expected values from blkid as well as regenerating your grub config and/or reinstalling grub on your EFI partition.

1 Like

For chroot, reinstall grub, etc. you can do it with a live USB (Garuda Welcome) or manually like here:

2 Likes

I always have rolling images of my machine stored in a separate external hard disk.

I use such system, given that there is no perfect OS. Garuda Linux is Arch-based, that means it is cutting-edge, any update can potentially break it. It does not delay updates from upstream Arch, hence always have an airbag with you (external image of system), aside from the seatbelt (timeshift/snapper)

3 Likes

In addition, are we dealing with a Windows dual boot here?

It's not like Win, use
Ctrl + Shift + C for copy
Ctrl + Shift + V for paste
in terminal/konsole

3 Likes

As much as I love car analogies, us old timers refer to this as the belt and suspenders approach. :smile:

2 Likes

@tbg No, windows only in VM's. I dare u :slight_smile:

@SGS Yeah my problem was rather how to copy the output when inside the emergency shell. Thanks anyway.

@Maynne Yeas, I was about to complain that when I realised I am just a spoiled brat from months of stability.

Can you believe haven't had time to work on this yet?

Next weekend I will report and properly close this thread.

well... I don't blame linux...

I too can't find the disk with UUID=fe781....etc....

dunno how the uuid changed, but isn't it just the case of updating to the correct uuid on the system's fstab and grub?

My apology for the delay but today I found time to finally attempt a fix.

  1. I booted into Live Garuda from usb.

sudo blkid
/dev/loop1: TYPE="squashfs"
/dev/sdb2: SEC_TYPE="msdos" LABEL_FATBOOT="MISO_EFI" LABEL="MISO_EFI" UUID="4480-7804" BLOCK_SIZE="512" TYPE="vfat"
/dev/sdb1: BLOCK_SIZE="2048" UUID="2022-01-31-17-08-00-00" LABEL="GARUDA_XFCE_WHITETAILEDEAGLE" TYPE="iso9660"
/dev/loop2: TYPE="squashfs"
/dev/loop0: TYPE="squashfs"
/dev/nvme1n1p2: UUID="4f00be9f-88d0-4710-9d54-64b39bffd653" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="ee4e9568-763a-b040-839f-ea70f9ed1bd4"
/dev/nvme1n1p1: LABEL_FATBOOT="NO_LABEL" LABEL="NO_LABEL" UUID="D7A5-0A70" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="140322bf-a2d2-d048-b250-a6f322679ad8"
/dev/sda1: LABEL="data" UUID="3e296ed7-73ee-4f95-9713-0e0dbb69c2a4" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="62dd0551-01"
/dev/zram0: LABEL="zram0" UUID="9440f2d6-edb4-4d45-a4fc-1000f45db867" TYPE="swap"
/dev/loop3: TYPE="squashfs"
/dev/nvme0n1: PTUUID="e1996a80" PTTYPE="dos"
/dev/mapper/mydrive: UUID="fe781c27-e3fe-49f6-896e-9611aa0f7838" UUID_SUB="97d2db13-5269-434e-b714-f993c1285a10" BLOCK_SIZE="4096" TYPE="btrfs"

I am able to mount the encrypted partition via thunar and was able to get the content of the current
/etc/fstab from the folder called @

@root 
@home

are accessible and show my data. same for the other drive that is /dev/sda.


# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=D7A5-0A70                            /boot/efi      vfat    umask=0077 0 2
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 / btrfs subvol=/@,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 /home btrfs subvol=/@home,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 /root btrfs subvol=/@root,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 /srv btrfs subvol=/@srv,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 /var/cache btrfs subvol=/@cache,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 /var/log btrfs subvol=/@log,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)
/dev/mapper/luks-4f00be9f-88d0-4710-9d54-64b39bffd653 /var/tmp btrfs subvol=/@tmp,defaults,noatime,noautodefrag,compress=zstd 0 0 #Modified_by_garuda-hotfixes(1)

I tried the following now:

sudo cryptsetup luksOpen /dev/nvme1n1p2 mydrive
sudo mount /dev/mapper/mydrive /mnt/

sudo btrfs subvolume list /mnt
ID 257 gen 109297 top level 5 path @
ID 258 gen 109297 top level 5 path @home
ID 259 gen 109281 top level 5 path @root
ID 260 gen 109297 top level 5 path @srv
ID 261 gen 109297 top level 5 path @cache
ID 262 gen 109297 top level 5 path @log
ID 263 gen 109297 top level 5 path @tmp
ID 272 gen 109263 top level 257 path @/.snapshots
...
docker btrfs volumes
and more snapshots
...

cleaning up...
sudo umount /mnt


sudo mount -t btrfs -o [email protected] /dev/mapper/mydrive /mnt/

ls /mnt/
bin  boot  crypto_keyfile.bin  desktopfs-pkgs.txt  dev   etc  home  lib  lib64  mnt  opt  proc  root  rootfs-pkgs.txt  run  sbin  srv  sys  tmp  usr  var
cat /mnt/boot/grub/grub.cfg
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-fe781c27-e3fe-49f6-896e-9611aa0f7838' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod cryptodisk
	insmod luks
	insmod gcry_rijndael
	insmod gcry_rijndael
	insmod gcry_sha256
	insmod btrfs
	set root='cryptouuid/4f00be9f88d047109d5464b39bffd653'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='cryptouuid/4f00be9f88d047109d5464b39bffd653'  fe781c27-e3fe-49f6-896e-9611aa0f7838
	else
	  search --no-floppy --fs-uuid --set=root fe781c27-e3fe-49f6-896e-9611aa0f7838
	fi
	echo	'Loading Linux linux-zen ...'
	linux	/@/boot/vmlinuz-linux-zen root=UUID=fe781c27-e3fe-49f6-896e-9611aa0f7838 rw [email protected]  loglevel=3 quiet
	echo	'Loading initial ramdisk ...'
	initrd	/@/boot/intel-ucode.img /@/boot/initramfs-linux-zen.img
}

The uuid of nvm1n1p2 is the same as in the first if block of grub.cfg.

It apparently does not find it and continues to the else block where a uuid is set that belongs to the luks mapper. It seems to be correct no?

So right now I try:

sudo garuda-chroot /mnt/

pacman -Syyu

looks like it is currently upgrading all packages.
My hope is it will now update grub as well and fix itself.

I have not mounted @home and @root currently.

Got the following expected errors

grub-probe: error: cannot find a GRUB drive for /dev/sdb1.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb1.  Check your device.map.

Got the following UNexpected errors

Found Ubuntu 21.10 on /dev/mapper/mydrive
Found Ubuntu 21.04 on /dev/mapper/mydrive
Found Ubuntu 18.04.4 LTS on /dev/mapper/mydrive
Adding boot menu entry for UEFI Firmware Settings ...
probably unrelated

(17/41) Refreshing PackageKit...
Error connecting: Could not connect: No such file or directory
error: command failed to execute correctly

Otherwise it finished ok.

Rebooting now and crossing my fingers.

Am I correct that arch-chroot and garuda-chroot will mount ALL needed things automatically?
normal chroot needs to mount sys proc etc in addition right?

I do NOT need to do that here correct?

No dice unfortunately so far.

I tried a few of my older snapshots now but again the same issue persists.

Trying pacman -Qqn | pacman -S - now

EDIT:

The BTRFS Assistant on the live System shows the same correct uuid that grub apparently cannot find..

Next thing I try is How to chroot Garuda Linux

No dice yet