Weird error No space left on device, progress cannot be queried

Issues & Assistance Xfce
1

After booting up my notebook after a long time I thought I might want to update.
So i did garuda-update. Done without errors. Next bootup I got the message unable to lock database so i used the garuda tool to unlock the debase which removes the lock file. Checked and the file was gone great. Same issue though. So i did df -H and saw no issue (will post output below). I tried btrfs scrub which helped me last time i had a similar issue. “btrfs scrub start /” no luck. there.
I also die clean the cache files of browser and package leftovers. I also did a stupid thing an excidently delet snapper images i think. There was a message popup for old snapper images, “want to delete it?” id lciked yes because i thought i need no old ones. “snappertools list” shows my recent ones. But restoring them is not possible becuase of errors. So time to post some outputs.

df -H

Filesystem      Size  Used Avail Use% Mounted on
dev             4,1G     0  4,1G   0% /dev
run             4,2G   19M  4,1G   1% /run
efivarfs        132k  126k   658 100% /sys/firmware/efi/efivars
/dev/sda2       512G  453G   59G  89% /
tmpfs           4,2G     0  4,2G   0% /dev/shm
/dev/sda2       512G  453G   59G  89% /root
/dev/sda2       512G  453G   59G  89% /srv
/dev/sda2       512G  453G   59G  89% /home
/dev/sda2       512G  453G   59G  89% /var/cache
/dev/sda2       512G  453G   59G  89% /var/log
/dev/sda2       512G  453G   59G  89% /var/tmp
tmpfs           4,2G  8,2k  4,2G   1% /tmp
/dev/sda1       314M  599k  314M   1% /boot/efi
/dev/sdb1       251G  125G  114G  53% /mnt/data
tmpfs           821M  222k  821M   1% /run/user/1000

btrfs scrub start /

WARNING: failed to open the progress status socket at /var/lib/btrfs/scrub.progress.4229126a-236f-4ce8-a461-0c7071445d9e: No space left on device, progress cannot be queried
WARNING: failed to write the progress status file: No space left on device, status recording disabled
scrub started on /, fsid 4229126a-236f-4ce8-a461-0c7071445d9e (pid=135755)
Starting scrub on devid 1

btrfs balance start /
ERROR: error during balancing '/': No space left on device

dmesg errors:

[    9.408640] systemd-journald[342]: Failed to open system journal: No space left on device
[ 2381.436367] BTRFS info (device sda2): 477 enospc errors during balance

snapper-tools list

421     Mo. Juni 3 09:58:05 2024        pacman -Su
422     Mo. Juni 3 10:01:04 2024        adwaita-cursors adwaita-icon-theme adwaita-icon-theme-legacy alsa-card-p
423     Mo. Juni 3 11:37:03 2024        pacman -U --config /etc/pacman.conf -- /home/zim/.cache/yay/opera-ffmpeg
424     Mo. Juni 3 11:37:11 2024        arttime-git opera opera-ffmpeg-codecs opera-ffmpeg-codecs-debug vdhcoapp
425     Mo. Juni 10 00:52:28 2024       pacman -Su

snapper-tools restore 421

terminate called after throwing an instance of 'QUnhandledException'
  what():  std::exception
Aborted (core dumped)

I myself can create files and folders. copied a 4gb file via samba with no problem.

garuda-inxi

System:
  Kernel: 6.9.3-arch1-1 arch: x86_64 bits: 64 compiler: gcc v: 14.1.1 clocksource: tsc
    avail: hpet,acpi_pm parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux
    root=UUID=4229126a-236f-4ce8-a461-0c7071445d9e rw rootflags=subvol=@ quiet quiet
    rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 ibt=off
  Console: pty pts/0 DM: LightDM v: 1.32.0 Distro: Garuda base: Arch Linux
Machine:
  Type: Laptop System: SAMSUNG product: 550P5C/550P7C v: P09ABI serial: <filter> Chassis: type: 9
    serial: N/A
  Mobo: SAMSUNG model: NP550P5C-S0ADE v: SEC_SW_REVISION_1234567890ABCD serial: <filter>
    part-nu: SAMSUNG SENS Series uuid: 5cfed280-44a6-11e2-a8b6-e408ac593700
    UEFI: American Megatrends v: P09ABI.026.141024.dg date: 10/24/2014
Battery:
  ID-1: BAT1 charge: 39.1 Wh (80.1%) condition: 48.8/48.8 Wh (100.0%) volts: 11.4 min: 11.1
    model: SAMSUNG Electronics SR Real Battery type: Li-ion serial: <filter> status: not charging
    cycles: 7
CPU:
  Info: model: Intel Core i7-3630QM socket: rPGA988B bits: 64 type: MT MCP arch: Ivy Bridge
    gen: core 3 level: v2 built: 2012-15 process: Intel 22nm family: 6 model-id: 0x3A (58)
    stepping: 9 microcode: 0x21
  Topology: cpus: 1x cores: 4 tpc: 2 threads: 8 smt: enabled cache: L1: 256 KiB
    desc: d-4x32 KiB; i-4x32 KiB L2: 1024 KiB desc: 4x256 KiB L3: 6 MiB desc: 1x6 MiB
  Speed (MHz): avg: 2776 high: 3400 min/max: 1200/3400 base/boost: 2400/3800 scaling:
    driver: intel_cpufreq governor: performance volts: 2.9 V ext-clock: 100 MHz cores: 1: 3400
    2: 3174 3: 3161 4: 1806 5: 3193 6: 1838 7: 3400 8: 2237 bogomips: 38333
  Flags: avx ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities: <filter>
Graphics:
  Device-1: Intel 3rd Gen Core processor Graphics vendor: Samsung Co driver: i915 v: kernel
    arch: Gen-7 process: Intel 22nm built: 2012-13 ports: active: LVDS-1 empty: DP-1,HDMI-A-1,VGA-1
    bus-ID: 00:02.0 chip-ID: 8086:0166 class-ID: 0300
  Device-2: NVIDIA GK107M [GeForce GT 650M] vendor: Samsung Co driver: N/A alternate: nouveau
    non-free: series: 470.xx+ status: legacy-active (EOL~2024-09-xx) arch: Kepler code: GKxxx
    process: TSMC 28nm built: 2012-2018 pcie: gen: 1 speed: 2.5 GT/s lanes: 16 bus-ID: 01:00.0
    chip-ID: 10de:0fd1 class-ID: 0300
  Device-3: Silicon Motion WebCam SC-13HDL11939N driver: uvcvideo type: USB rev: 2.0
    speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-1.4:4 chip-ID: 2232:1029 class-ID: 0e02
    serial: <filter>
  Display: server: X.org v: 1.21.1.13 with: Xwayland v: 24.1.0 driver: X: loaded: modesetting
    dri: crocus gpu: i915 tty: 120x30
  Monitor-1: LVDS-1 model: Seiko Epson 0x314b built: 2011 res: 1600x900 dpi: 118 gamma: 1.2
    size: 344x194mm (13.54x7.64") diag: 395mm (15.5") ratio: 16:9 modes: 1600x900
  API: EGL v: 1.5 hw: drv: intel crocus platforms: device: 0 drv: crocus device: 1 drv: swrast
    surfaceless: drv: crocus inactive: gbm,wayland,x11
  API: OpenGL v: 4.5 compat-v: 4.2 vendor: mesa v: 24.1.1-arch1.1 note: console (EGL sourced)
    renderer: Mesa Intel HD Graphics 4000 (IVB GT2), llvmpipe (LLVM 17.0.6 256 bits)
  API: Vulkan v: 1.3.279 layers: 4 device: 0 type: integrated-gpu name: Intel HD Graphics 4000
    (IVB GT2) driver: mesa intel v: 24.1.1-arch1.1 device-ID: 8086:0166 surfaces: N/A device: 1
    type: cpu name: llvmpipe (LLVM 17.0.6 256 bits) driver: mesa llvmpipe v: 24.1.0-arch1.1 (LLVM
    17.0.6) device-ID: 10005:0000 surfaces: N/A
Audio:
  Device-1: Intel 7 Series/C216 Family High Definition Audio vendor: Samsung Co
    driver: snd_hda_intel v: kernel bus-ID: 00:1b.0 chip-ID: 8086:1e20 class-ID: 0403
  API: ALSA v: k6.9.3-arch1-1 status: kernel-api tools: N/A
  Server-1: PipeWire v: 1.0.7 status: off with: 1: pipewire-pulse status: off 2: wireplumber
    status: off 3: pipewire-alsa type: plugin 4: pw-jack type: plugin
    tools: pactl,pw-cat,pw-cli,wpctl
Network:
  Device-1: Intel Wi-Fi 6E AX210/AX1675 2x2 [Typhoon Peak] driver: iwlwifi v: kernel pcie: gen: 2
    speed: 5 GT/s lanes: 1 bus-ID: 02:00.0 chip-ID: 8086:2725 class-ID: 0280
  IF: wlp2s0 state: up mac: <filter>
  Device-2: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet vendor: Samsung Co
    driver: r8169 v: kernel pcie: gen: 1 speed: 2.5 GT/s lanes: 1 port: d000 bus-ID: 03:00.0
    chip-ID: 10ec:8168 class-ID: 0200
  IF: enp3s0 state: down mac: <filter>
  Info: services: NetworkManager, sshd, systemd-timesyncd, wpa_supplicant
Bluetooth:
  Device-1: Intel AX210 Bluetooth driver: btusb v: 0.8 type: USB rev: 2.0 speed: 12 Mb/s lanes: 1
    mode: 1.1 bus-ID: 2-1.5:3 chip-ID: 8087:0032 class-ID: e001
  Report: btmgmt ID: hci0 rfk-id: 0 state: down bt-service: enabled,running rfk-block:
    hardware: no software: yes address: <filter> bt-v: 5.3 lmp-v: 12 status: discoverable: no
    pairing: no
Drives:
  Local Storage: total: 715.41 GiB used: 537.72 GiB (75.2%)
  ID-1: /dev/sda maj-min: 8:0 vendor: Samsung model: SSD 860 PRO 512GB family: based SSDs
    size: 476.94 GiB block-size: physical: 512 B logical: 512 B sata: 3.1 speed: 6.0 Gb/s tech: SSD
    serial: <filter> fw-rev: 2B6Q temp: 57 C scheme: GPT
  SMART: yes state: enabled health: PASSED on: 209d 8h cycles: 566 written: 7.23 TiB
  ID-2: /dev/sdb maj-min: 8:16 vendor: Samsung model: SSD 830 Series family: based SSDs
    size: 238.47 GiB block-size: physical: 512 B logical: 512 B sata: 3.0 speed: 3.0 Gb/s tech: SSD
    serial: <filter> fw-rev: 3B1Q temp: 28 C scheme: MBR
  SMART: yes state: enabled health: PASSED on: 3y 251d 16h cycles: 2437 written: 124.81 TiB
Partition:
  ID-1: / raw-size: 476.64 GiB size: 476.64 GiB (100.00%) used: 421.33 GiB (88.4%) fs: btrfs
    block-size: 4096 B dev: /dev/sda2 maj-min: 8:2
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%) used: 584 KiB (0.2%) fs: vfat
    block-size: 512 B dev: /dev/sda1 maj-min: 8:1
  ID-3: /home raw-size: 476.64 GiB size: 476.64 GiB (100.00%) used: 421.33 GiB (88.4%) fs: btrfs
    block-size: 4096 B dev: /dev/sda2 maj-min: 8:2
  ID-4: /var/log raw-size: 476.64 GiB size: 476.64 GiB (100.00%) used: 421.33 GiB (88.4%)
    fs: btrfs block-size: 4096 B dev: /dev/sda2 maj-min: 8:2
  ID-5: /var/tmp raw-size: 476.64 GiB size: 476.64 GiB (100.00%) used: 421.33 GiB (88.4%)
    fs: btrfs block-size: 4096 B dev: /dev/sda2 maj-min: 8:2
Swap:
  Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default) zswap: no
  ID-1: swap-1 type: zram size: 7.65 GiB used: 0 KiB (0.0%) priority: 100 comp: zstd
    avail: lzo,lzo-rle,lz4,lz4hc,842 max-streams: 8 dev: /dev/zram0
Sensors:
  System Temperatures: cpu: 62.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Info:
  Memory: total: 8 GiB available: 7.65 GiB used: 930.6 MiB (11.9%) igpu: 64 MiB
  Processes: 194 Power: uptime: 1h 1m states: freeze,mem,disk suspend: deep avail: s2idle
    wakeups: 0 hibernate: platform avail: shutdown, reboot, suspend, test_resume image: 3.02 GiB
    Init: systemd v: 255 default: graphical tool: systemctl
  Packages: pm: pacman pkgs: 2405 libs: 604 tools: gnome-software,pamac,paru,yay pm: flatpak
    pkgs: 0 Compilers: clang: 17.0.6 gcc: 14.1.1 Shell: garuda-inxi default: Bash v: 5.2.26
    running-in: pty pts/0 (SSH) inxi: 3.3.34
Garuda (2.6.26-1):
head: cannot open '/var/log/pacman.log' for reading: No such file or directory
  System install date:
  Last full system update: 2024-06-03
  Is partially upgraded:   No
  Relevant software:       snapper NetworkManager mkinitcpio
  Windows dual boot:       No/Undetected
  Failed units:            garuda-pacman-snapshot-reject.service ipp-usb.service lightdm.service logrotate.service nmb.service optimus-manager.service preload.service smb.service systemd-random-seed.service systemd-update-utmp.service vncserver-x11-serviced.service [email protected] [email protected]
2

Please

Make sure you have done the following before you post:

and

  • Format terminal output (including your garuda-inxi) as a code block by clicking the preformatted text button (</>) , or put three tildes (~) above and below the text
1 Like
3

done that otherwise I wouldn’t have written here.

Terminal output formatting i forgot. sorry for that.
Trying to edit the post.

4

grafik
grafik997×652 73.7 KB

1 Like
5

as mentioned above i searched the forum, there was o solution that did the trick.
Why would I write if a solution was there? I can install new but that does give me no insight of what happened.

Garuda forum search gave me:
https://forum.garudalinux.org/t/unable-to-install-kernel-on-chroot-no-space-left-on-device/32342

https://forum.garudalinux.org/t/problems-updating-with-limited-disk-space/16125/4

https://forum.garudalinux.org/t/fake-no-space-left-on-device-error/9385

Some are 2 years old. Checked it anyways.
Tried google solutions and arch manuals but I got no progress.

Everything seems fine, just that the OS cannot write.

6

Too many problems for me, sorry.

TTY or chroot?

Which DE?

1 Like
7

TTY

Install DAte:

stat / | grep "Birth" | sed 's/Birth: //g' | cut -b 2-11
2022-12-13

The failed units errors:
Failed units errors are there because the system cannot write.
Last time i fixed the btrfs error those issues were gone after that.

8

Would be interesting to know, GNOME, Cinnamon???

1 Like
9

I might be in the same boat. At this point I know its some kind btrfs issue what interfered with the update. But as mentioned earlier already tried solutions. But that’s basically the same issue I have.

https://forum.garudalinux.org/t/failed-to-reboot-after-sudo-pacman-syu-cannot-restore-snapshot/36080/12

Traying a chroot from a SUB live boot again. Maybe I was missing something.
Will keep it running so I can instantly try something if anything comes up.

btrfs-assistant --list
1       @       421     Mon Jun 3 09:58:05 2024 pre     @/.snapshots/421/snapshot       4229126a-236f-4ce8-a461-0c7071445d9e
2       @       422     Mon Jun 3 10:01:04 2024 post    @/.snapshots/422/snapshot       4229126a-236f-4ce8-a461-0c7071445d9e
3       @       423     Mon Jun 3 11:37:03 2024 pre     @/.snapshots/423/snapshot       4229126a-236f-4ce8-a461-0c7071445d9e
4       @       424     Mon Jun 3 11:37:11 2024 post    @/.snapshots/424/snapshot       4229126a-236f-4ce8-a461-0c7071445d9e
5       @       425     Mon Jun 10 00:52:28 2024        pre     @/.snapshots/425/snapshot       4229126a-236f-4ce8-a461-0c7071445d9e

btrfs-assistant -r 422
Error: Failed to parse snapshot list
1 Like
10

XFCE4 + lightdm

most often I use the console without graphics.

11

so after a while I tried the chroot thing but its not how I remembered it when I used it in previews years.

[root@arconet /]# mount /dev/sda2 /mnt/
[root@arconet /]# arch-chroot /mnt
mount: /mnt/proc: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.
==> ERROR: failed to setup chroot /mnt
[root@arconet /]# dmesg
[ 1278.084994] BTRFS: device fsid 4229126a-236f-4ce8-a461-0c7071445d9e devid 1 transid 566191 /dev/sda2 (8:2) scanned by mount (3321)
[ 1278.085530] BTRFS info (device sda2): first mount of filesystem 4229126a-236f-4ce8-a461-0c7071445d9e
[ 1278.085572] BTRFS info (device sda2): using crc32c (crc32c-intel) checksum algorithm
[ 1278.085579] BTRFS info (device sda2): using free-space-tree
[root@arconet /]# arch-chroot /mnt/@
@/                          @_backup_20241006014433747/ @_backup_20241006015031682/ @home/                      @tmp/
@_backup_20240306095543887/ @_backup_20241006014647914/ @_backup_20241006015051751/ @log/
@_backup_20241006014406713/ @_backup_20241006014855540/ @_backup_20242105170409176/ @root/
@_backup_20241006014421662/ @_backup_20241006015011198/ @cache/                     @srv/
[root@arconet /]# arch-chroot /mnt/@/
==> WARNING: /mnt/@/ is not a mountpoint. This may have undesirable side effects.
[root@arconet /]# ls
bin  boot  dev  efi  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@arconet /]# pacman -Syu
:: Synchronizing package databases...
error: failed to synchronize all databases (unable to lock database)
[root@arconet /]#

arconet was the only bootstick I had on hand sorry.

12

I got this error message several years ago, when I was doing a lot of distro hopping and so reinstalling many times many distros on the same machine (so your issue is most likely different and the same error message could be misleading), and I solved by deleting the many dump files I had in the efivars directory (rm /sys/firmware/efi/efivars/dump-* to be executed as root).

2 Likes
13

sooo

btrfs check --repair /dev/sda2
found no errors. Glad to read that.

also cleaned evivars. there was an old windows entry there from idk when.
First attempt went well with garuda-update. Started working with no warnings.
And just before it was done it complained that there is no space left on the device.

Could it be that I am stuck in snapshot? How could I check that?
My assumption is that it boots directly into one of the snapshots i tried to recover from.

14

maybe you trying this sometimes are corrupted files after update.

start your usb drive with the garuda on it. Open gparted and right click on your root partition and select check with left click. it is like `Auto repair filesystem when errors are detected

During the filesystem check, if errors are detected, you can get “fsck” to auto repair the filesystem with the -a flag. For example:)`in gui way.

15

Sorry @Duke187 Duke187 only have seen your post after I finished repairing. But I am thankful still. @filo Cleaning efivars did. Not sure why and how there was so much trash in there. Even an old win7 boot entry… Anyways. it worked. So for anyone who arrives from google and needs help and for learning and understanding of course. below is my experience and fix.

So after work I had time again to find a solution.
Cleaning /sys/firmware/efi/efivars was the beginning.

Clearing out everything and rewriting efivar enabled me to do a “normal” boot.
I had mount errors and other errors but i was in the system. I could see but not write.

I Did boot a live usb garuda und used chroot to connect to my real system.

btrfs check 
sudo btrfsck /dev/sdc2

So it was broken…

I repaired the broken stuff with:
btrfsck --check --force /dev/sdb4

This is highly dangerous (but i had nothing important on the drive. I do my backups)
Remember i tried to fix this because i wanted to learn. Reinstalling would do the same.

I ended up with a broken system of course.
garuda usb-live boot chroot again and was able to use a read/write filesystem.
So progress…

The fixing starts here:

sudo pacman -Qk

listed my broken packages
pacman -Qk 2>/dev/null | grep -v ' 0 missing files' | cut -d: -f1

created a list for reinstall.
Then i remembered the btrfs filesystem error and decidet to reinstall all packages.

from arhwiki: pacman/Tips and tricks - ArchWiki
created the list.
pacman -Qqen > packages.txt

installed from it.
pacman -S - < packages.txt

to fix the other errors like:

ldconfig: File /usr/lib/libsvn_repos-1.so is empty, not checked.

I ran my script:

LANG=de_DE ldconfig 2> err.txt

for file in $(cut -d" " -f3 err.txt)
do
echo $file
pacman -S $(pacman -Fq $file) --noconfirm --overwrite “*”
done

After that I Reinstalled Grub

After that I rebooted with mount error.
garuda usb-live boot chroot again.

I generated a new fstab file (thought installing would also change this):
genfstab -U -p /mnt >> /mnt/etc/fstab
Rebooted and it worked fine.

From here I could finally do:

garuda-update

Since then everything is fine and dandy.
There are steps in between where I had to read logs and test (trial and error) stuff which I skipped here for obvious reasons.
Love you Garuda-Linux.

Epilogue:
Nice live boot experience. You really thought this through.
Using btrfs-instead of ext4 was also a smart move.
btrfs-assistant and snapper-tools… I don’t want to be without them anymore.

It is nice to have a forum, which helped me a lot in other cases. Hope it keeps growing.
I am using Linux over 25 years now and seeing it evolve is great.
Keep it up Team. Thx for this OS. Will donate at the end of the year as always with my OS of choice.

3 Likes
16

As the clearest of the spaces that was out of space was efivars, clearing efivars is more than deleting a couple of menu entries.

ls /sys/firmware/efi/efivars | grep "^dump" &&
sudo rm /sys/firmware/efi/efivars/dump-*

Check out Archwiki article.

3 Likes
17

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.