RO FS for major stuff & kernels lost after 0 bytes left on disk

Issues & Assistance
1

After I tried to yay -Syu with little space left my Latte and then everything crashed.

After a few tries to fix stuff after launching from timeshift's snapshots in grub even more kernels went missing from there (grub), but, thankfully, two points out of six are now fine.

Pacman's db.lck was both there and not in the process.
Now it's just there. And I can't do anything to it.
And there seem to be quite some other locked system files now.
Can't remove them from Live CD no matter how I try.
File system checks (btrfs) don't show errors.

I want my system -- my Pacman and other things, incl. pretty grub with new kernels -- back.

2

If you filled your disk mid-way through an update process then those updated packages will be broken.

You can try cleaning the package cache and removing older snapshots, or restoring the snapshot from just before you broke the system

If, instead of restoring a snapshot, you can recover some free space then you will want to re-run the update to make sure the files are installed correctly. If the system won't boot then you can do that from within a chroot and fix the bootloader etc. from there.

The other option is to mark this down as experience (to not fill your disk) and wipe and reinstall.

4 Likes
3

Timeshift doesn't seem to work and update grub from withing the OS now.

I have neither garuda-chroot not arch-chroot installed now.
And my Pacman is locked. This is pretty much the only issue. I'll want to update the system.
All of the snapshots have db.lck locked. It's locked from Live Ubuntu, too. Should I try live Garuda & use garuda-chroot there to rm it?

I have 80G of free space now.

What's the best to do now?

4

Snapshots will be locked because they’re read-only. If you’re using a chroot then you can remove the /var/lib/pacman/db.lck file manually.

1 Like
5

I'll try that with Garuda Live and reply

6

Still can't. Mounted it, chrooted into into it (/mnt, not /mnt/@, is that correct or should I try with @, too?) under all roots... Nothing seems to work, hdparm -r0 before mounting, too.

By the way, what's that root subpartition (@) thing that Timeshift tells about? I seem to have removed the (@) thing in one command by following someone somewhere in the Internet and now it seems to only be able to rsync the system, but it doesn't matter as much as...

What do we do with package mgmt? Maybe with some other now locked tools. I must say that Firefox thinks it's old now and works OK with --allow-downgrade option only,

7

It’s a subvolume. @ is used as the root subvolume for btrfs filesystems.

No idea what you’ve done there, then. Following someone somewhere’s instructions tends to end badly.

I don’t know what state your system is in, or how you appear to have got it booted.

At this point I’d probably put this down as a lost cause - too much is in a bad state, and while it might be fixable it would need a significant amount of time and back-and-forth.

3 Likes
8

How do I back major things up like my browser profile settings (got a lot) and Latte? Lists of yay and pip packages will be reinstalled, what else...

And how do I really back things up without FS being able to fail the way I stop having full control over it. During a regular update. That's nonsense.

I actually think that locked pacman is my only problem now. There must be a way to remove this lock.

Can some other package manager continue to work with my installations now? A great one must be able to do that.

9

Same like above but

image
image1727×1002 78.5 KB

pacman is best and enough.

BTW.
Before you open a new help request, read relevant sections of the Arch and Garuda wiki.
Thoroughly search your issue and any error messages in the forum and on the web.

Report everything you have already attempted to solve your problem.

For each “problem” use one topic, that facilitates the forum search to help others with the same problem.

2 Likes
10

I've read it all and nothing works.

BTW.
Before you publish a new reply, try being useful and at least reading what was written. At least in the headline.

I repeat.
I even mounted the disk with hdparm -r0 from Live CD before trying everything under all roots and chroots.
Is there a way to delete this file at the lower level that hasn't been described pretty much anywhere when it comes to simple deletion of non-immutable 000? Error keeps being thrown for a small number of such files in the system: "Read-only filesystem". There must be some sort of superroot.

Or can I somehow make the system forget that I have this broken pacman installed or db.lck there?
Maybe let pacman-static not check how pacman is doing?
Another instance of pacman to replace that locked one would be fine.

If you can't help with with that, please, suggest, what is the best now installable tool to replace pacman (that sucks) and continue to work with my installed packages?

11

You can't replace pacman. It is the package manager.

We need more info to help you. Can you start by booting off of a live ISO, mounting the root of the btrfs partition to /mnt and then provide the output of

sudo btrfs subvolume list /mnt
12 
ID 257 gen 294293 top level 5 path @home
ID 258 gen 294292 top level 5 path @root
ID 259 gen 292643 top level 5 path @srv
ID 260 gen 294276 top level 5 path @cache
ID 261 gen 294293 top level 5 path @log
ID 262 gen 294293 top level 5 path @tmp
ID 451 gen 292667 top level 5 path timeshift-btrfs/snapshots/2021-10-02_05-24-36/@
ID 464 gen 294275 top level 5 path timeshift-btrfs/snapshots/2021-10-26_17-29-50/@
ID 555 gen 292643 top level 5 path timeshift-btrfs/snapshots/2021-11-11_19-54-28/@
ID 557 gen 292667 top level 5 path timeshift-btrfs/snapshots/2021-11-11_22-14-14/@
ID 558 gen 292667 top level 5 path timeshift-btrfs/snapshots/2021-11-11_22-14-07/@
ID 559 gen 292667 top level 5 path timeshift-btrfs/snapshots/2021-11-11_22-22-46/@
ID 560 gen 292667 top level 5 path timeshift-btrfs/snapshots/2021-11-11_22-22-33/@
ID 561 gen 292672 top level 5 path @
13

Can you share the output of sudo btrfs subvolume show /mnt/@

15 
ERROR: cannot find real path for '/mnt/@': No such file or directory
16

You need to do it with btrfs root mounted to /mnt. If you actually mounted @ to /mnt then change the command to sudo btrfs subvolume show /mnt

18

I mounted my /dev/device to /mnt w/o @
(or with?)

Here's the output of sudo btrfs subvolume show /mnt

timeshift-btrfs/snapshots/2021-10-02_05-24-36/@
	Name: 			@
	UUID: 			cc007234-7376-0b44-ba37-491613ba0420
	Parent UUID: 		066a5497-fb05-654d-93d7-9fbb5edd91f3
	Received UUID: 		-
	Creation time: 		2021-10-02 02:24:36 +0000
	Subvolume ID: 		451
	Generation: 		292667
	Gen at creation: 	96285
	Parent ID: 		5
	Top level ID: 		5
	Flags: 			readonly
	Snapshot(s):
	Quota group:		0/451
	  Limit referenced:	-
	  Limit exclusive:	-
	  Usage referenced:	27.32GiB
	  Usage exclusive:	1.81GiB
19

OK, can we see the output of findmnt /mnt

20 
TARGET SOURCE                                                     FSTYPE OPTIONS
/mnt   /dev/nvme0n1p5[/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@]
                                                                  btrfs  rw,rela
21

You mounted @ to a /mnt

You mounted a snapshot to /mnt.

You need to mount the root of the filesystem to /mnt

22

I thing I've deleted @, yes.
I've done sudo btrfs property set -ts /mnt ro false.
Should I try to delete db.lck now?

TIME MACHINE!!!
(there's a limit for posts)
After

sudo umount /mnt
sudo mount /dev/nvme0n1p5 /mnt -o subvolid=5
sudo btrfs subvolume show /mnt/@

:

@
	Name: 			@
	UUID: 			70933d23-545a-9746-9276-4970f80fb409
	Parent UUID: 		84f93f8f-2c54-434a-8524-8688d9861b44
	Received UUID: 		-
	Creation time: 		2021-11-11 19:22:46 +0000
	Subvolume ID: 		561
	Generation: 		292672
	Gen at creation: 	289575
	Parent ID: 		5
	Top level ID: 		5
	Flags: 			-
	Snapshot(s):
	Quota group:		0/561
	  Limit referenced:	-
	  Limit exclusive:	-
	  Usage referenced:	26.63GiB
	  Usage exclusive:	1.66MiB

UPDATE:
Should I put this UUID (what UUID?) after search --no-floppy --fs-uuid --set=root there in /boot/grub/grub.cfg?

UPDATE:
grub.cfg:


# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod btrfs
search --no-floppy --fs-uuid --set=root fac55994-2534-431e-a77f-14b5a0fdbd25
    font="/@/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
insmod part_gpt
insmod btrfs
search --no-floppy --fs-uuid --set=root fac55994-2534-431e-a77f-14b5a0fdbd25
insmod gfxmenu
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_12.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_14.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_16.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_24.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_48.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_bold_16.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/dejavu_sans_mono_12.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-12.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-14.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-16.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-18.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-b12.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-b14.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-b16.pf2
loadfont ($root)/@/usr/share/grub/themes/garuda-dr460nized/terminus-b18.pf2
insmod png
set theme=($root)/@/usr/share/grub/themes/garuda-dr460nized/theme.txt
export theme
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/35_fwupd ###
### END /etc/grub.d/35_fwupd ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/41_snapshots-btrfs ###
submenu 'Garuda Linux snapshots' {
    configfile "${prefix}/grub-btrfs.cfg"
}
### END /etc/grub.d/41_snapshots-btrfs ###

### BEGIN /etc/grub.d/60_memtest86+ ###
if [ "${grub_platform}" == "pc" ]; then
    menuentry "Memory Tester (memtest86+)" --class memtest86 --class gnu --class tool {
        search --fs-uuid --no-floppy --set=root  fac55994-2534-431e-a77f-14b5a0fdbd25
        linux16 /@/boot/memtest86+/memtest.bin 
    }
fi
### END /etc/grub.d/60_memtest86+ ###

### BEGIN /etc/grub.d/61_custom_leave_options ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry "Shutdown" --class shutdown {
    echo "System shutting down..."
    halt
}

menuentry "Restart" --class restart {
    echo "System rebooting..."
    reboot
}

#if [ ${grub_platform} == "efi" ]; then
#    menuentry "Firmware Setup (UEFI)" --class recovery {
#        fwsetup
#    }
#fi
### END /etc/grub.d/61_custom_leave_options ###

There are only two of the oldest snapshots in a snapshot subfolder in grub now after I tried to automatically fix my grub somehow, can't remember how (using grub-mkconfig, as it states, yes).

grub-btrfs.cfg:

submenu 'Snapshot: 2021-10-26 17:29:50  timeshift-btrfs/snapshots/2021-10-26_17-29-50/@' {
    submenu '---> Snapshot: 2021-10-26 17:29:50  timeshift-btrfs/snapshots/2021-10-26_17-29-50/@ <---' { echo }

    menuentry 'vmlinuz-linux-zen & initramfs-linux-zen-fallback.img & amd-ucode.img' --class snapshots --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-snapshots-fac55994-2534-431e-a77f-14b5a0fdbd25' {
        if [ x$feature_all_video_module = xy ]; then
        insmod all_video
        fi
        set gfxpayload=keep
        insmod btrfs
        if [ x$feature_platform_search_hint = xy ]; then
            search --no-floppy --fs-uuid  --set=root  fac55994-2534-431e-a77f-14b5a0fdbd25
        else
            search --no-floppy --fs-uuid  --set=root fac55994-2534-431e-a77f-14b5a0fdbd25
        fi
        echo 'Loading Snapshot: 2021-10-26 17:29:50 timeshift-btrfs/snapshots/2021-10-26_17-29-50/@'
        echo 'Loading Kernel: vmlinuz-linux-zen ...'
        linux "/timeshift-btrfs/snapshots/2021-10-26_17-29-50/@/boot/vmlinuz-linux-zen" root=UUID=fac55994-2534-431e-a77f-14b5a0fdbd25  quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3 rootflags=defaults,noatime,space_cache,autodefrag,compress=zstd,subvol="timeshift-btrfs/snapshots/2021-10-26_17-29-50/@"
        echo 'Loading Microcode & Initramfs: amd-ucode.img initramfs-linux-zen-fallback.img ...'
        initrd "/timeshift-btrfs/snapshots/2021-10-26_17-29-50/@/boot/amd-ucode.img" "/timeshift-btrfs/snapshots/2021-10-26_17-29-50/@/boot/initramfs-linux-zen-fallback.img"
    }

    menuentry 'vmlinuz-linux-zen & initramfs-linux-zen.img & amd-ucode.img' --class snapshots --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-snapshots-fac55994-2534-431e-a77f-14b5a0fdbd25' {
        if [ x$feature_all_video_module = xy ]; then
        insmod all_video
        fi
        set gfxpayload=keep
        insmod btrfs
        if [ x$feature_platform_search_hint = xy ]; then
            search --no-floppy --fs-uuid  --set=root  fac55994-2534-431e-a77f-14b5a0fdbd25
        else
            search --no-floppy --fs-uuid  --set=root fac55994-2534-431e-a77f-14b5a0fdbd25
        fi
        echo 'Loading Snapshot: 2021-10-26 17:29:50 timeshift-btrfs/snapshots/2021-10-26_17-29-50/@'
        echo 'Loading Kernel: vmlinuz-linux-zen ...'
        linux "/timeshift-btrfs/snapshots/2021-10-26_17-29-50/@/boot/vmlinuz-linux-zen" root=UUID=fac55994-2534-431e-a77f-14b5a0fdbd25  quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3 rootflags=defaults,noatime,space_cache,autodefrag,compress=zstd,subvol="timeshift-btrfs/snapshots/2021-10-26_17-29-50/@"
        echo 'Loading Microcode & Initramfs: amd-ucode.img initramfs-linux-zen.img ...'
        initrd "/timeshift-btrfs/snapshots/2021-10-26_17-29-50/@/boot/amd-ucode.img" "/timeshift-btrfs/snapshots/2021-10-26_17-29-50/@/boot/initramfs-linux-zen.img"
    }
}
submenu 'Snapshot: 2021-10-02 05:24:36  timeshift-btrfs/snapshots/2021-10-02_05-24-36/@' {
    submenu '---> Snapshot: 2021-10-02 05:24:36  timeshift-btrfs/snapshots/2021-10-02_05-24-36/@ <---' { echo }

    menuentry 'vmlinuz-linux-zen & initramfs-linux-zen-fallback.img & amd-ucode.img' --class snapshots --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-snapshots-fac55994-2534-431e-a77f-14b5a0fdbd25' {
        if [ x$feature_all_video_module = xy ]; then
        insmod all_video
        fi
        set gfxpayload=keep
        insmod btrfs
        if [ x$feature_platform_search_hint = xy ]; then
            search --no-floppy --fs-uuid  --set=root  fac55994-2534-431e-a77f-14b5a0fdbd25
        else
            search --no-floppy --fs-uuid  --set=root fac55994-2534-431e-a77f-14b5a0fdbd25
        fi
        echo 'Loading Snapshot: 2021-10-02 05:24:36 timeshift-btrfs/snapshots/2021-10-02_05-24-36/@'
        echo 'Loading Kernel: vmlinuz-linux-zen ...'
        linux "/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@/boot/vmlinuz-linux-zen" root=UUID=fac55994-2534-431e-a77f-14b5a0fdbd25  quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3 rootflags=defaults,noatime,space_cache,autodefrag,compress=zstd,subvol="timeshift-btrfs/snapshots/2021-10-02_05-24-36/@"
        echo 'Loading Microcode & Initramfs: amd-ucode.img initramfs-linux-zen-fallback.img ...'
        initrd "/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@/boot/amd-ucode.img" "/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@/boot/initramfs-linux-zen-fallback.img"
    }

    menuentry 'vmlinuz-linux-zen & initramfs-linux-zen.img & amd-ucode.img' --class snapshots --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-snapshots-fac55994-2534-431e-a77f-14b5a0fdbd25' {
        if [ x$feature_all_video_module = xy ]; then
        insmod all_video
        fi
        set gfxpayload=keep
        insmod btrfs
        if [ x$feature_platform_search_hint = xy ]; then
            search --no-floppy --fs-uuid  --set=root  fac55994-2534-431e-a77f-14b5a0fdbd25
        else
            search --no-floppy --fs-uuid  --set=root fac55994-2534-431e-a77f-14b5a0fdbd25
        fi
        echo 'Loading Snapshot: 2021-10-02 05:24:36 timeshift-btrfs/snapshots/2021-10-02_05-24-36/@'
        echo 'Loading Kernel: vmlinuz-linux-zen ...'
        linux "/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@/boot/vmlinuz-linux-zen" root=UUID=fac55994-2534-431e-a77f-14b5a0fdbd25  quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3 rootflags=defaults,noatime,space_cache,autodefrag,compress=zstd,subvol="timeshift-btrfs/snapshots/2021-10-02_05-24-36/@"
        echo 'Loading Microcode & Initramfs: amd-ucode.img initramfs-linux-zen.img ...'
        initrd "/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@/boot/amd-ucode.img" "/timeshift-btrfs/snapshots/2021-10-02_05-24-36/@/boot/initramfs-linux-zen.img"
    }
}