I try to make posting here a last resort to solve my Garuda issues but this really has me stumped.
I got home from vacation about 2 weeks ago and updated from the terminal like I always do (I read an official post stating the preferred way to do it now is with the "garuda-update" command in Konsole) and then it crashed on boot and dropped me into an emergency shell. All I could really think to do at the time was to load a snapshot of a working version and hope the problem fixed itself with a newer update. (There was one time I had issues after an update and then a newer update fixed it, that was my approach to this issue.) Unfortunately, it didn't work. I also noticed some strange comments about the kernels during the update that I normally never see, something akin to, "Good news! this version of kernel xyz matches the current one". It says this for several different kernels including the main zen one I use. I would have to crash my system again as a result of the system update to recreate the comment so I could copy/paste it.
The only other thing I could think to do was to reinstall all of the software packages to see if maybe some settings got messed up that would sort themselves out with a reinstall. (it worked in the past for me when I screwed up some settings). But I'm getting a 404 error returned from the servers when I attempt that which is also first.
*** I hate to come here only when I have problems because I love Garuda so much. I guess I'll consider donating after this gets resolved. The last time I had an issue though I made sure to thoroughly document the solution so that maybe it could help others and contribute, in my own small way, to the community. Also, I am a bit of a newb to Linux which is why my attempted solutions are so basic. People might wonder why I'm using Arch Linux as a newb but Garuda makes it so easy with all of the GUI options and BTRFS snapshots make fatal screwups easy to reverse. It's a testament to how good Garuda is that someone like me uses it with such ease without issue 99% of the time. ***
Do you have your USB drive with Garuda on it? It might be worth trying the Grub Repair tool from the live environment, or if that doesn't work reinstalling grub altogether from chroot.
I went to the text console using ctrl + alt + f2 and activated the web console then tried to refresh the mirror lists using the "sudo reflector -a6 -f5 --save /etc/pacman.d/mirrorlist" command but every single one failed to rate. So I just went to the garuda assistant to tweak the mirror list and turned on the worldwide setting instead of just the United States and refreshed (my line of thinking was to access as many servers as possible to get the packages I need to ensure a proper update). I'm attempting to update again in Garuda OS right now and it looks like I'm getting some of the same strange comments from the updater as last time. So, if I crash my system and have to load a snapshot I can at least post that. I'm going to restart now and if it goes to the emergency shell I'll try the startx command to see if that's an issue before loading a snapshot.
Error! Module version 6.1.34 for vboxdrv.ko.zst
is not newer than what is already found in kernel 5.17.9-zen1-1-zen (6.1.34).
You may override by specifying --force.
Good news! Module version 6.1.34 for vboxnetflt.ko.zst
exactly matches what is already found in kernel 5.17.9-zen1-1-zen.
DKMS will not replace this module.
You may override by specifying --force.
Good news! Module version 6.1.34 for vboxnetadp.ko.zst
exactly matches what is already found in kernel 5.17.9-zen1-1-zen.
DKMS will not replace this module.
You may override by specifying --force.
==> dkms install --no-depmod vboxhost/6.1.34_OSE -k 5.17.14-hardened1-1-hardened
==> dkms install --no-depmod vboxhost/6.1.34_OSE -k 5.18.3-zen1-1-zen
==> dkms install --no-depmod vboxhost/6.1.34_OSE -k 5.15.46-1-lts
==> dkms install --no-depmod openrazer-driver/3.3.0 -k 5.15.46-1-lts
==> dkms install --no-depmod openrazer-driver/3.3.0 -k 5.17.14-hardened1-1-hardened
==> depmod 5.18.3-zen1-1-zen
==> depmod 5.17.9-zen1-1-zen
==> depmod 5.15.46-1-lts
==> depmod 5.17.14-hardened1-1-hardened
(16/42) Restarting cronie for libc upgrade...
(17/42) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux-hardened.preset: 'default'
-> -k /boot/vmlinuz-linux-hardened -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-hardened.img
==> Starting build: 5.17.14-hardened1-1-hardened
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
-> Running build hook: [openswap]
==> WARNING: swap_device variable is not set
==> WARNING: crypt_swap_name variable is not set
-> Running build hook: [resume]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-hardened.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-hardened.preset: 'fallback'
-> -k /boot/vmlinuz-linux-hardened -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-hardened-fallback.img -S autodetect
==> Starting build: 5.17.14-hardened1-1-hardened
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: bfa
==> WARNING: Possibly missing firmware for module: qed
==> WARNING: Possibly missing firmware for module: qla1280
==> WARNING: Possibly missing firmware for module: qla2xxx
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
-> Running build hook: [openswap]
==> WARNING: swap_device variable is not set
==> WARNING: crypt_swap_name variable is not set
-> Running build hook: [resume]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-hardened-fallback.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
-> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 5.15.46-1-lts
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
-> Running build hook: [openswap]
==> WARNING: swap_device variable is not set
==> WARNING: crypt_swap_name variable is not set
-> Running build hook: [resume]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
-> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
==> Starting build: 5.15.46-1-lts
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: bfa
==> WARNING: Possibly missing firmware for module: qed
==> WARNING: Possibly missing firmware for module: qla1280
==> WARNING: Possibly missing firmware for module: qla2xxx
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
-> Running build hook: [openswap]
==> WARNING: swap_device variable is not set
==> WARNING: crypt_swap_name variable is not set
-> Running build hook: [resume]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-lts-fallback.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-zen.preset: 'default'
-> -k /boot/vmlinuz-linux-zen -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-zen.img
==> Starting build: 5.18.3-zen1-1-zen
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
-> Running build hook: [openswap]
==> WARNING: swap_device variable is not set
==> WARNING: crypt_swap_name variable is not set
-> Running build hook: [resume]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-zen.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-zen.preset: 'fallback'
-> -k /boot/vmlinuz-linux-zen -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-zen-fallback.img -S autodetect
==> Starting build: 5.18.3-zen1-1-zen
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: bfa
==> WARNING: Possibly missing firmware for module: qed
==> WARNING: Possibly missing firmware for module: qla1280
==> WARNING: Possibly missing firmware for module: qla2xxx
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
-> Running build hook: [plymouth]
-> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
-> Running build hook: [openswap]
==> WARNING: swap_device variable is not set
==> WARNING: crypt_swap_name variable is not set
-> Running build hook: [resume]
-> Running build hook: [filesystems]
-> Running build hook: [grub-btrfs-overlayfs]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-zen-fallback.img
==> Image generation successful
(18/42) Refreshing PackageKit...
(19/42) GRUB update after transactions...
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/garuda-dr460nized/theme.txt
Found linux image: /boot/vmlinuz-linux-zen
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-zen.img
Found fallback initrd image(s) in /boot: amd-ucode.img initramfs-linux-zen-fallback.img
Found linux image: /boot/vmlinuz-linux-lts
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-lts.img
Found fallback initrd image(s) in /boot: amd-ucode.img initramfs-linux-lts-fallback.img
Found linux image: /boot/vmlinuz-linux-hardened
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-hardened.img
Found fallback initrd image(s) in /boot: amd-ucode.img initramfs-linux-hardened-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Garuda Linux on /dev/mapper/luks-b8e4a7b6-9b6a-44fd-ae21-5f821357034c
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2022-06-13 17:33:09 | @/.snapshots/149/snapshot | pre | pacman -Su |
Found snapshot: 2022-06-13 15:52:46 | @/.snapshots/148/snapshot | post | python2 python2-gobject python2-numpy python2-wxpython3 qt5-script |
Found snapshot: 2022-06-13 15:52:45 | @/.snapshots/147/snapshot | pre | pacman -Rns python2-gobject python2-numpy python2-wxpython3 qt5-script |
Found snapshot: 2022-06-13 12:48:28 | @/.snapshots/146/snapshot | post | akonadi-calendar akonadi-contacts akonadi-import-wizard akonadi-mime ako |
Found snapshot: 2022-06-13 12:47:00 | @/.snapshots/145/snapshot | pre | pacman -Su |
Found snapshot: 2022-06-13 12:40:26 | restore_backup_@_125214759 | N/A | N/A |
Found snapshot: 2022-06-13 12:33:48 | @/.snapshots/144/snapshot | post | akonadi-calendar akonadi-contacts akonadi-import-wizard akonadi-mime ako |
Found snapshot: 2022-06-13 12:32:14 | @/.snapshots/143/snapshot | pre | pacman -Su |
Found snapshot: 2022-06-10 13:04:28 | @/.snapshots/142/snapshot | post | bpytop |
Found snapshot: 2022-06-10 13:04:28 | @/.snapshots/141/snapshot | pre | /usr/bin/pacman -S --noconfirm community/bpytop |
Found snapshot: 2022-06-07 12:19:03 | restore_backup_@_124026659 | N/A | N/A |
Found snapshot: 2022-06-07 12:11:27 | @/.snapshots/140/snapshot | post | ananicy-rules-git birdfont dxvk-mingw-git edk2-armvirt edk2-ovmf frei0r- |
Found snapshot: 2022-06-07 12:10:42 | @/.snapshots/139/snapshot | pre | pacman -Su |
Found snapshot: 2022-06-07 12:01:49 | restore_backup_@_121903847 | N/A | N/A |
Found snapshot: 2022-05-05 01:16:45 | restore_backup_@_120149576 | N/A | N/A |
Found snapshot: 2022-05-05 01:07:36 | restore_backup_@_20220505011645185 | N/A | N/A |
Found snapshot: 2022-05-05 00:59:46 | restore_backup_@_20220505010736327 | N/A | N/A |
Found snapshot: 2022-05-05 00:56:54 | restore_backup_@_005946249 | N/A | N/A |
Found snapshot: 2022-05-05 00:54:35 | @_backup_20220505005654133 | N/A | N/A |
Found snapshot: 2022-04-13 23:45:30 | @_backup_20220505005435446 | N/A | N/A |
Found snapshot: 2022-04-13 23:40:03 | restore_backup_@_20221304234530414 | N/A | N/A |
Found 21 snapshot(s)
Unmount /tmp/grub-btrfs.Lm7KMbAR9O .. Success
Found memtest86+ image: /boot/memtest86+/memtest.bin
done
(20/42) Reloading system bus configuration...
(21/42) Warn about old perl modules
WARNING: '/usr/lib/perl5/5.34' contains data from at least 3 packages which will NOT be used by the installed perl interpreter.
-> Run the following command to get a list of affected packages: pacman -Qqo '/usr/lib/perl5/5.34'
(22/42) Copy FireDragon Distribution Settings
(23/42) Copy Firefox Distribution Settings
(24/42) Updating fontconfig cache...
(25/42) Updating 32-bit fontconfig cache...
(26/42) Foreign/AUR package notification
frozen-bubble 2.2.1beta1-17
kipi-plugins 21.12.3-1
libkipi 21.12.3-1
noisetorch 0.11.5-1
openrazer-daemon 3.3.0-1
openrazer-driver-dkms 3.3.0-1
openrazer-meta 3.3.0-1
perl-alien-sdl 1.446-12
perl-sdl 2.548-6
python-openrazer 3.3.0-1
samba-mounter-git 186.29ebad4-1
sdl_pango 0.1.2-9
(27/42) Probing GDK-Pixbuf loader modules...
(28/42) Probing 32-bit GDK-Pixbuf loader modules...
(29/42) Compiling GSettings XML schema files...
(30/42) Fix 'grub' and 'os-prober'
(31/42) Updating icon theme caches...
(32/42) Updating GTK4 module cache...
(33/42) Copy Librewolf Distribution Settings
(34/42) Orphaned package notification...
ffmpeg-compat-57 3.4.11-1
(35/42) Checking for .pacnew and .pacsave files...
.pac* files found:
/etc/locale.gen.pacnew
/etc/pamac.conf.pacnew
/etc/pacman.d/mirrorlist.pacnew
/etc/retroarch.cfg.pacnew
Please check and merge
(36/42) Updating the info directory file...
(37/42) Copy Thunderbird Distribution Settings
(38/42) Updating the desktop file MIME type cache...
(39/42) Updating the vlc plugin cache...
(40/42) Updating Vim help tags...
(41/42) Performing snapper post snapshots for the following configurations...
==> root: 150
(42/42) Syncing all file systems...
Sorry about that. I should have been more specific. I edited my post to clarify. I went to the text console using ctrl + alt + f2 and then used the command "sudo reflector -a6 -f5 --save /etc/pacman.d/mirrorlist" but every mirror list failed to rate.
I just tried the garuda-update command in the text console and it just did the same update I pasted that's been making my system crash. I think I'm just going to reinstall it.
I'm really not sure why people are suggesting reinstalling grub. This is very clearly an issue with a required systemd service failing to start and therefore kicking you into an emergency shell. I don't know how reinstalling grub is supposed to change anything there
Take a look at your failed systemd services after you enter an emergency shell and their logs.
I also tried updating from text only terminal (ctrl+alt+f2) and using sudo pacman -Syyu
I can't check for errors with systemctl from emergency shell because the keyboard input is really wonky and delayed to where it only seems to pick up random keystrokes.
I checked the grub.cfg file in live environment and the uuid matches up with the uuid of my root partition.
I'm pretty bewildered at this point
I'm not sure the uuid for grub boot cfg and other cfgs are correct. I saved copies of fstab, grub, grub-btrfs, lsblk output, garuda boot options kernel parameters, and device info from KDE partition manager.
This is why we try to only suggest reinstalling the OS as an absolute last resort. Not only does it potentially cost you data you may want to keep, it does nothing to teach users how to troubleshoot issues to correct matters.
Nuke and pave is the Windows way. The whole point to using a rolling release distro is that you should never have to reinstall. Some people have kept their initial install of Arch working for a dozen or more years. That is what learning about your system can accomplish for you. Don’t give up so easily, the reward is far greater if you wait.
I'm still learning. But I spent so much time trying to fix this it's definitely going to be ingrained in my memory that such a simple solution was lying right under my nose the entire time. I keep most of my important data on totally separate drives which is why I'm so inclined to nuke and pave. But, even nuking and paving wouldn't initially fix this. That was also a very important lesson.
Since I'm running lts will the issues that were preventing me from using the zen kernel eventually be updated and make it usable again? I only ask because, from my basic understanding, zen is optimized for gaming. I haven't even tried to play anything while using lts yet.
I should have gone with the ockhams razor approach and tried lts first instead of beating myself over the head with these other, more complex, approaches. Your link also seems like it will have an answer to my previous question about zen.
I just wanted to check back in and detail how the issue was solved instead of immediately disappearing back into the ether once everything was working again.
As far as I understand, the kernel bug (first of my two links above) is already fixed, and should get into the next rc for 5.19 (5.19rc3) for the mainline tree, so you could test it when the next linux-mainline release (5.19rc3) will be available in the chaotic-aur.
I don’t think it will enter into a next 5.18.x release, but I don’t know how this is handled.
Whenever it will enter in the linux kernel, I think it could get into the linux-zen. But, also here, I’m not sure.
If this will not be the case, if the fix will work on mainline, it could be fair in my opinion to enter a bug report in the linux-zen archrepo to have it included.
Did you try this?
