System does not start after update

[hennifant]

Hello Garuda users.

After the update, the loading bar stops, the logo still rotates. I can't access the console via ctrl+alt+f2/3. the only keyboard shortcut that still works is ctrl+alt+del. then the game starts all over again... Thank you very much for effort and time.

I‘m using garuda LXQT-Kwin.

[filo]

Hi there, welcome.
In these cases the starting point is always:

Computer doesn’t boot, boots to a black screen, or stops at a message

While there are many potential reasons why your installation doesn’t boot to a desktop, the primary reason is that the graphics drivers haven’t loaded correctly so your login/display manager (DM) doesn’t start. This is why you get a “black screen” or stuck at an [ OK ] Started some service message. The normal reason for this is that you have the incorrect driver version for the kernel. This can happens when the mirror you are updating from is not fully synchronised, or times out, and so y…

When you get into your system, either booting to a TTY or chrooting, if none of the tutorial suggestions work, please provide your garuda-inxi and the input/output of what you tried.

[hennifant]

Hi,
thanks for your reply.

I followed the guide on this page How to chroot Garuda Linux and also tried "Chroot" in the Garuda Welcome. Alacritty shows up and shortly after that it disappears.

I tried to chroot into my system but i got a error message after "sudo garuda-chroot /mnt/broken/@" chroot: failed to run command '/bin/bash' : No such file or directory.

[filo]

Are you sure you mounted the correct partition to /mnt/broken?
Could you please provide from your live USB:

lsblk -f
sudo fdisk -l

and ref. to which partition you used in the chroot?

[hennifant]

nvme0n1p1
│    vfat   FAT32 NO_LABEL
│                       C41A-30DE
├─nvme0n1p2
│    btrfs              57f5ad94-c608-42d8-9e32-f63d624c1623  179.1G    21% /mnt/broken
└─nvme0n1p3
     swap   1     swap  d4df4282-a710-49b3-98dd-2a794acb5731

sudo fdisk -l
Disk /dev/nvme0n1: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: SAMSUNG MZVLB256HAHQ-00000
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: FB9BF1F1-D5F2-8D43-BB41-ADC3F60565DB

Device             Start       End   Sectors   Size Type
/dev/nvme0n1p1      4096    618495    614400   300M EFI System
/dev/nvme0n1p2    618496 481648511 481030016 229.4G Linux filesystem
/dev/nvme0n1p3 481648512 500103449  18454938   8.8G Linux swap


Disk /dev/sdc: 3.76 GiB, 4038066176 bytes, 7886848 sectors
Disk model: Flash Disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot   Start     End Sectors  Size Id Type
/dev/sdc1  *         64 4715075 4715012  2.2G  0 Empty
/dev/sdc2       4715076 4723267    8192    4M ef EFI (FAT-12/16/32)


Disk /dev/loop0: 14.46 MiB, 15167488 bytes, 29624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 338.77 MiB, 355229696 bytes, 693808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 1.14 GiB, 1227935744 bytes, 2398312 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 656.68 MiB, 688582656 bytes, 1344888 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 7.51 GiB, 8066695168 bytes, 1969408 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

[garuda@garuda-lxqt ~]$ sudo mount /dev/nvme0n1p2 /mnt/broken
[garuda@garuda-lxqt ~]$ sudo garuda-chroot /mnt/broken/@
chroot: failed to run command '/bin/sh': No such file or directory

[filo]

Strange, that looks like the correct procedure.
My only idea would be to start checking what is in there, e.g.

ls /mnt/broken/@

I hope someone else can jump in with other suggestions...

[hennifant]

Thanks for your help.

[garuda@garuda-lxqt ~]$ ls /mnt/broken/@
boot dev etc home lib lib64 opt proc root run srv sys tmp usr var

[filo]

/bin seems to be missing at all!
Can you try booting onto a previously working snapshot from the Grub menu and then restore it?

Restoring snapshots to get the system back to working state

[hennifant]

This question may sound stupid, but how do i enter the grub menu?

[filo]

The menu that appears, for a few seconds normally, when you turn on your machine.
Normally it has a first line “Garuda Linux”, but instead you should see and select Garuda snapshots" and select a snapshot with date/time before the problem occurred.

[hennifant]

I only see for a few seconds that Garuda is selected. Or to express it differently, I can't choose anything. Is there a key combination to get into the menu?

[filo]

To be honest, there seems to be something really broken in your system in my opinion.
If this is a new installation, I suggest you should reinstall (and, believe me, this is the first time I do it).
Maybe wait a little bit if there are other suggestions. I might be missing something even trivial, but I'm at wits' end...

[hennifant]

Thank you for taking your time to help me.
I really appreciate that.
Unfortunately my whole work is on the system.

[filo]

First off, backup everything you can (from the live USB you should be able to open your system folders).
My only crazy idea would be to copy the /bin folder from the live USB to /mnt/broken/@ and retry chrooting.
I've never done that, and this could be not the only missing piece, but in the worst case you just delete it again...

[hennifant]

Thank you very much. I copied the /bin and /sbin directory from the live USB and now everything runs again as before. I can't tell you how grateful I am to you!

[tbg]

WTF, how do those directories just disappear. Is there some fun new game out that all the kids are playing? Perhaps the new game is Linux Roulette. If you lose, it deletes a random directory. All the rage I hear.

[filo]

To be honest, after a "beyond broken system", this is a "beyond PEBCAK case" .
All good anyway! Maybe...

[tbg]

I think as you suggested earlier a backup and reinstall is the only way to be sure things are in a good working state. Another perfect example of why full backups are so important.

Don't get caught with your pants down kids, implement a full backup routine before disaster strikes.

[hennifant]

I can't explain it either, the update went through normally and after the reboot it got stuck. Very true...
I will now make regular backups again. So far, however, the system has been so wonderfully stable that I had displaced it.

[FGD]

@hennifant Hmmm, I'd immediately do a

sudo btrfs check /[YOUR_DISK_AND_PARTITION] (like /dev/nvme0n1p2 in your case it seems)

from Live ISO to make sure your filesystem is still top notch.

I've never ever had btrfs issues, but starting about 2 months ago I had 3. Would have lost data without my external backups.

Something like doing a garuda-update, it fails at a certain point complaining it cannot write to disk and all of a suddent in findmnt -l my subvolumes are now read-only. I btrfs check from a Live ISO and a BUNCH of btrfs inode issues and stuff like that.
Subvolumes changing to read-only happened twice in the last 2 months.
Could be related to hardware failing with time, but STILL plz issue a btrfs check just in case to get your ass covered there, cuz if you got any errors, it ain't pretty.