Failed to boot after update due to installer misconfig

[timothy-b]

Howdy,

I installed Garuda a couple of weeks ago or so, and recently hit this error after using garuda-update (which included a newer Linux kernel) and then rebooting. After Grub, but before logging in, I ran into this:

ERROR: device 'UUID=...' not found. Skipping fsck.
mount: /new_root: can't find UUID=...
You are now being dropped into and emergency shell.

I was able to restore a previous snapshot (very nice) and figure it out from there. I found tons of posts on the internet regarding this error, but none of them had to do with this particular cause, which is why I'm posting this here.

I eventually ended up searching through Calamares to figure out how it was setting up encryption, and this is where I finally figured out what happened: calamares/main.py at 29ef9e909df58e025868e1f0eea1c4f0fc30cf3a · calamares/calamares · GitHub

Then I verified that /etc/dracut.conf.d/calamares-luks.conf was created and configured, but /etc/mkinitcpio.conf looked like the default file.

Here's what happened: apparently the live installer (Calamares) configured and used dracut to create the initial ramdisk, but there's some Pacman hook that triggers mkinitcpio after installing a kernel. I don't think this would've been a problem if I hadn't chosen the "Encrypt System" option during installation, since otherwise the default /etc/mkinitcpio.conf probably would've sufficed, and the relevant kernel parameters passed to Grub by Calamares seem to only vary for encryption in this case.

Here's how I fixed it: I configured /etc/mkinitcpio.conf, /etc/default/grub, ran mkinitcpio -P, update-grub, and rebooted to a new kernel verify the fix.

/etc/mkinitcpio.conf changes:
Added both of these:
FILES=(/etc/crypttab /crypto_keyfile.bin)

Added encrypt:
HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block encrypt filesystems fsck)

/etc/default/grub changes:

Added cryptodisk to the start of GRUB_PRELOAD_MODULES (maybe this is unnecesary, I haven't checked):
GRUB_PRELOAD_MODULES="cryptodisk part_gpt part_msdos"

Changed GRUB_CMDLINE_LINUX_DEFAULT:

#before
#GRUB_CMDLINE_LINUX_DEFAULT='rd.luks.uuid=<a uuid> rd.udev.log_priority=3 vt.global_cursor_default=0 resume=UUID=<another uuid> loglevel=3'
#after
GRUB_CMDLINE_LINUX_DEFAULT='cryptdevice=UUID=<the uuid>:luks-<the name in /etc/crypttab> root=/dev/mapper/<the name in /etc/crypttab> rd.udev.log_priority=3 vt.global_cursor_default=0 resume=UUID=<another uuid> loglevel=3'

Here's what I'd recommend to prevent this from happening (cc: @dr460nf1r3 ): if Garuda is installed & configured with dracut, then disable the mkinitcpio -> pacman hook and wire in dracut to use that hook instead. I'd be happy to help solve this if you'll point me in the right direction.

---

Getting to the bottom of this issue took me more time than I'd like to admit, so hopefully this at least helps out someone else

[BluishHumility]

Unless you are using an old ISO, mkinitcpio should not even be installed in the first place. Back in March, Garuda switched to using dracut as the initial image creation tool. Since then, the ISOs ship with a package called garuda-dracut-support which explicitly conflicts with mkinitcpio:

pacman -Sii garuda-dracut-support
Repository      : garuda
Name            : garuda-dracut-support
Version         : 1.0.1-1
Description     : Dracut support for Garuda Linux
Architecture    : any
URL             : None
Licenses        : GPL
Groups          : None
Provides        : None
Depends On      : dracut  zstd
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : mkinitcpio  mkinitcpio-openswap  mkinitcpio-systemd-tool  garuda-bootctl-dracut  dracut-hook
...

It is possible to install mkinitcpio after the installation is up, but not without removing garuda-dracut-support.

Did you install mkinitcpio on your system?

@timothy-b

but /etc/mkinitcpio.conf looked like the default file.

Can you explain what you mean by this? On recent Garuda ISOs (again, since March), /etc/mkinitcpio.conf is not a file that exists on the system at all (since mkinitcpio is no longer installed by default).

If you would like to switch back to dracut, it is very simple to do and is described here: Dracut | Garuda Linux wiki. Adding the needed modules for encryption is done automatically by dracut, so you can spare yourself the grief of setting it up manually.

@timothy-b

Here’s what I’d recommend to prevent this from happening (cc: @dr460nf1r3 ): if Garuda is installed & configured with dracut, then disable the mkinitcpio → pacman hook and wire in dracut to use that hook instead. I’d be happy to help solve this if you’ll point me in the right direction.

On a stock Garuda installation, this is already done. The mkinitcpio hook has been removed (and would not run even if it were still there, since mkinitcpio itself is not installed), and instead /usr/share/libalpm/hooks/90-dracut-install.hook will automatically run the /usr/share/libalpm/scripts/dracut-install script whenever a kernel or dkms package is installed or upgraded:

cat /usr/share/libalpm/hooks/90-dracut-install.hook
File: /usr/share/libalpm/hooks/90-dracut-install.hook
[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Target = usr/lib/modules/*/vmlinuz
Target = usr/lib/dracut/*
Target = usr/lib/systemd/systemd

[Trigger]
Operation = Install
Operation = Remove
Operation = Upgrade
Type = Package
Target = *-dkms*

[Action]
Description = Updating initramfs...
When = PostTransaction
Exec = /usr/share/libalpm/scripts/dracut-install
NeedsTargets

My guess would be the crux of the issue is mkinitcpio was installed at some point, when perhaps that is not what you wanted.

[timothy-b]

Thanks for the reply!

@BluishHumility

Back in March, Garuda switched to using dracut as the initial image creation tool.

Ah ok, my ISO is dated 03/19/2023, and I downloaded it on April 12th, so that part adds up.

@BluishHumility

It is possible to install mkinitcpio after the installation is up, but not without removing garuda-dracut-support.

Did you install mkinitcpio on your system?

Aha! I explicitly installed mkinitcpio-numlock after reading this article, but a few minutes later I edited ~/.config/sway/config.d/input to enable numlock on boot via that method instead. And then the mkinitcpio hook must’ve ran for the first time when I updated my system (which included a kernel update). Ah man, I didn’t know what I was doing, but I’ve certainly learned a lot since then

What would you think about updating that first wiki page to include a warning against pursuing that method if your system is already configured with dracut? I didn’t know what mkinitcpio was at the time, or that I had dracut installed, and maybe I saw the conflict warning from Pacman, but didn’t think much of it given that it was 1am.

@BluishHumility

If you would like to switch back to dracut, it is very simple to do and is described here: Dracut | Garuda Linux wiki. Adding the needed modules for encryption is done automatically by dracut, so you can spare yourself the grief of setting it up manually.

Perfect! I will.

@BluishHumility

On a stock Garuda installation, this is already done. The mkinitcpio hook has been removed (and would not run even if it were still there, since mkinitcpio itself is not installed), and instead /usr/share/libalpm/hooks/90-dracut-install.hook will automatically run the /usr/share/libalpm/scripts/dracut-install script whenever a kernel or dkms package is installed or upgraded:

Glad to hear it, thanks for the details, and thanks again for your help in getting to the bottom of this!

[BluishHumility]

I’m glad I was able to help crack the case!

@timothy-b

What would you think about updating that first wiki page to include a warning against pursuing that method if your system is already configured with dracut?

Personally I don’t think we should warn people not to use mkinitcpio. It is a perfectly good tool that works well, and people should feel welcome to use it if they wish. Part of that choice would be accepting that mkinitcpio must be manually configured with the needed hooks and modules–it is not automatic like dracut.

@timothy-b

and maybe I saw the conflict warning from Pacman

Pacman will not lecture you if you wish to uninstall something, even if there will be major consequences. If Pacman mentions something must be uninstalled to proceed with a choice, try to find out what it is first. For better or worse, Pacman assumes you know what you are doing.

@timothy-b

I explicitly installed mkinitcpio-numlock after reading this article, but a few minutes later I edited ~/.config/sway/config.d/input to enable numlock on boot via that method instead.

I enable numlock in a Sway config as well. It works perfectly fine for me, however of course it doesn’t kick in until Sway is up.

If you want numlock activated in early boot (to use during device decryption, for example), this person whipped up a Bash script which can be copy/pasted into the dracut modules directory:

GitHub - FivEawE/dracut-numlock: A dracut module which enables numlock in the...

Just bear in mind, the command for rebuilding the initramfs on Garuda is not sudo dracut -f, it is sudo dracut-rebuild.

[timothy-b]

@BluishHumility

Personally I don’t think we should warn people not to use mkinitcpio.

That’s totally fair, but I’m not suggesting that, just that we add a warning here to the effect of “mkinitcpio conflicts with dracut, make sure you know which you’re using before you attempt this method”. Maybe that’s obvious for most people, but it wasn’t for me, and so it might help others.

@BluishHumility

I enable numlock in a Sway config as well. It works perfectly fine for me, however of course it doesn’t kick in until Sway is up.

If you want numlock activated in early boot (to use during device decryption, for example), this person whipped up a Bash script which can be copy/pasted into the dracut modules directory:

Just bear in mind, the command for rebuilding the initramfs on Garuda is not sudo dracut -f, it is sudo dracut-rebuild.

Good to know, thanks for the tips!