Garuda performance/stability issues: some comments and suggestions

Hi there!

This is my first post in this forum. I've been an Arch Linux user for several years and came to Garuda a couple of weeks ago just to test BTRFS (with its features like subvolumes, snapshots ...) - and here I am ::. I've been a KDE user forever, and I'm running the KDE Barebones version under Wayland (as the other versions are too bloated for me).

I've seen many complaints in this forum about performance and stability problems causing some people even leaving Garuda. Here are some thoughts about these issues:

  1. Performance problems
    Yes, I had noticed them, too. I had tried to find the culprit by disabling/masking the services of the various Garuda-specific goodies - but to no avail.

But finally I found one setting that made a huge difference: disabling quota by executing

sudo btrfs quota disable /

and (important!) by disabling quota support in Timeshift.

Since then my system is fast again!

However, one irritating result is that with every update Timeshift produces errors like:

 E: ERROR: can't list qgroups: quotas not enabled

E: btrfs returned an error: 256
E: Failed to query subvolume quota
------------------------------------------------------------------------------
Removing snapshot: 2021-03-22_14-42-12
Deleting subvolume: @ (Id:830)
Deleted subvolume: @ (Id:830)

Destroying qgroup: 0/830
E: Failed to destroy qgroup: '0/830'
E: Failed to remove snapshot: 2021-03-22_14-42-12

... but snapshots are reliably created (and deleted) nevertheless.

Extra tip: You might also run into performance issues with very large files, e.g., large Virtualbox image files. My Windows10 VM had become unbearably slow. Solution: Disable copy-on-write for those files and disable compression for, at least, the large files. Result: My Windows10 VM is as fast as before!

  1. Stability problems
    I don't have any but can't tell about other DEs than KDE. My impression is that these problems might be caused by the fact that Garuda is very user-friendly and, thus, attracting users which are not familiar with rolling distros. And rolling distros need some discipline and maintenance! Fortunately, this can be automated to some extent.
  • First of all: Never ever install a package by just executing
sudo pacman -S package

Rather, always use

sudo pacman -Syu package

in order to avoid partial upgrades.

To make things a bit easier I've created the following alias in my .bashrc:

alias i="sudo pacman -Syu"

So executing

i package

is all you need.

  • I update my system using the following alias:
alias u='paru -Pw
while true; do
    read -n 1 -p "Continue? [Any key/N(o)]" ANSWER
    
    if [ "$ANSWER" == "N" ]
    then
    
    echo
    echo "  *** Break! ***"
    break
    
    else
    
    echo
    echo
            
        sudo reflector --verbose --country Germany --protocol https --latest 20 --score 10 --sort score --ipv4 --save /etc/pacman.d/mirrorlist && \
        sudo powerpill -Syu && \
        paru -Sua && \
        sudo updatedb && locate --existing --regex '\.pac(new|save)$' > pacnew.log && \
        sudo paccache -ruk0 && sudo paccache -rvk2 
        
               
        pacman -Qqn > pkglist.txt
        pacman -Qqm > pkglist-foreign.txt
        
              
      if [ -s pacnew.log ]

      then
      
      echo

	  echo "ATTENTION! .pacnew and/or .pacsave files available":
	  
	  cat pacnew.log

      fi
        
    break
    
     
    fi    
    
       
done'

Remarks:

  • paru -Pw (or, if you prefer, yay -Pw) looks for news on archlinux.org. If there aren't any or you've read them, just press any key to continue. If the news suggests a manual intervention before updating, press the N key to interrupt the update process.
  • I let reflector sort for the mirror score (not for the fastest as they would have to be tested by reflector which needs time) as they usually belong to the fastest mirrors anyhow, and by using powerpill with 5 parallel downloads my bandwidth is fully used.
  • paru -Sua (or yay -Sua updates the AUR packages
  • The updatedb line locates .pac(new|save) files. (See the link in the second post.)
  • The paccache line cleans the pacman cache. (See the link in the second post.)
  • Finally, if there are .pac(new|save) files they are displayed so I can immediately merge/modify them by using the alias
alias pacdiff='sudo -H DIFFPROG=meld pacdiff'
  • If important packages like the kernel or the DE are updated I reboot immediately in order to avoid running into problems.
  • I automatically check for updates several times a day. If only a handful of packages are updated and you're running into problems, it's relatively easy to narrow down the culprits. If you update only once a week or every second week, that's much harder if several hundred packages are updated.

By applying this strategy I've had a stable Arch system over several years. So I hope that these comments will be helpful for some of you.

P.S.: I just found out that new users can only add 5 links to a post. So the remaining links will be added to a follow-up post.

5 Likes

The other links supposed to be in above post are:

https://wiki.archlinux.org/index.php/Pacman/Pacnew_and_Pacsave#Locating_.pac*_files

https://wiki.archlinux.org/index.php/Pacman#Cleaning_the_package_cache

1 Like

Thank you for your post.

Most of what you say regarding performance issues is well documented on the forum. The devs have been aware of the issues regarding BTRFS and timeshift for a while. Most of the recommendations you made I discovered from researching the btrfs performance issues and I tested disabling quotas some time ago which does help. I have taken a few further steps to improve performance.

I do not perform auto snapshots. I make manual timeshift snapshots periodically until I reach 4-5 snapshots. As manual snapshots are not automatically added to the grub boot menu I wrote a shutdown service that runs update-grub at shutdown. I do a manual delete of all timeshift snapshots once I have 4-5 snapshots saved.

After deleting the snapshots I perform a btrfs balancing. This appears to be another key factor to maintaining performance with btrfs. The devs are also aware of this issue and tests are being run to see if performing balancing on a regular schedule via a service may be added by default in the future.

Thank you for taking the time to document all your findings.

3 Likes

Thanks for your reply!

I have 2 questions:

  1. What is the reason why you don't perform auto snapshots? Do you see any disadvantages?
  2. BTRFS balancing is done automatically once a month by btrfs-balance.timer (since the btrfsmaintenance package was added recently). Do you think that's not often enough?

actually i added it some days prior
:sweat_smile:

so many people will see improvement in coming days in regards to performance

6 Likes

Are you referring to the btrfsmaintenance package?

I use a small SSD as my system drive. Creating snapshots so frequently is wastage if you are only installing a single program why bother taking a bunch of snapshots in this case. In the past I simply wrote my own service to initiate a timeshift backup at install. However, I wrote the service so that I could cancel the backup by keyboard intervention if it was only a single package install being done. This saved wasted time and space and I really liked that option when using ext4.

That really isn't possible with btrfs snapshots, so I simply create my timeshift snapshots manually. This allows me to keep fewer snapshots on my small drive that are actually more selective restore points. I think if you are flushing a large amount of data and using a small drive (or partition) for the OS monthly is likely not enough. If you have a large percentage of your system drive containing flushed data I feel weekly balancing is required (possibly more). Obviously this depends on each use case. Those using large btrfs partitions will likely require balancing less often I feel.

This is only my subjective opinion.

2 Likes

Sorry, I’ve been away from home and haven’t been at a computer for the last week. So, I haven’t kept up with recent changes without my home computer.

1 Like

I have been testing implementing various hooks and services to advise users about Arch/Garuda maintenance tasks so that they are aware of issues and how to deal with them.

One of the features of the service I am working on is a check for any news releases since the last system update. The service will either give a popup notifying the user of new announcements or it may simply open the Arch news page in a browser.

The hooks and services will give a popup notification when specific events occur during the update process that may require attention. This includes such things as not having performed system updates for over for several weeks, pacnew files that require merging, and when a reboot should be performed after major updates.

Bare in mind these are intended for the benefit of newbies (not experienced Arch users). Of course the service can be modified or disabled entirely if you feel you do not require these kind of reminders. :wink:

2 Likes

probably almost everybody will disable it, no matter how experienced they are, lol. As newbies are usually unwilling to learn maintainence and only get annoyed about popups, and experienced users don't need it. But still, it might help to some open-minded people out there

Yes, I agree most will disable it, and when the Newb blows up his install because he didn't follow the recommendation of the Arch news announcement the distro can then say guess you shouldn't have disabled that popup Mr. Know it All.

:wink:

1 Like

Btw., I forgot to mention that I masked the systemd-swap.service as it caused problems for me (already earlier when I still ran pure Arch). The boot process mostly stuck , and I wasn't able to switch to a TTY with Ctrl-Alt-F3. I don't know if other users are also affected. Anyway, I'm using a slightly modified version of an approach which I found here. It looks like this:

This is /usr/local/bin/zram-on:

#!/bin/bash

# Disable zswap
echo 0 > /sys/module/zswap/parameters/enabled

modprobe zram

#zram0
echo zstd > /sys/block/zram0/comp_algorithm
echo 2G > /sys/block/zram0/disksize
echo 1G > /sys/block/zram0/mem_limit
mkswap --label zram0 /dev/zram0
swapon --priority 32767 /dev/zram0

#zram1
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram1/comp_algorithm
echo 2G > /sys/block/zram1/disksize
echo 1G > /sys/block/zram1/mem_limit
mkswap --label zram1 /dev/zram1
swapon --priority 32767 /dev/zram1

#zram2
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram2/comp_algorithm
echo 2G > /sys/block/zram2/disksize
echo 1G > /sys/block/zram2/mem_limit
mkswap --label zram2 /dev/zram2
swapon --priority 32767 /dev/zram2

#zram3
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram3/comp_algorithm
echo 2G > /sys/block/zram3/disksize
echo 1G > /sys/block/zram3/mem_limit
mkswap --label zram3 /dev/zram3
swapon --priority 32767 /dev/zram3

#zram4
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram4/comp_algorithm
echo 2G > /sys/block/zram4/disksize
echo 1G > /sys/block/zram4/mem_limit
mkswap --label zram4 /dev/zram4
swapon --priority 32767 /dev/zram4

#zram5
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram5/comp_algorithm
echo 2G > /sys/block/zram5/disksize
echo 1G > /sys/block/zram5/mem_limit
mkswap --label zram5 /dev/zram5
swapon --priority 32767 /dev/zram5

#zram6
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram6/comp_algorithm
echo 2G > /sys/block/zram6/disksize
echo 1G > /sys/block/zram6/mem_limit
mkswap --label zram6 /dev/zram6
swapon --priority 32767 /dev/zram6

#zram7
cat /sys/class/zram-control/hot_add
echo zstd > /sys/block/zram7/comp_algorithm
echo 2G > /sys/block/zram7/disksize
echo 1G > /sys/block/zram7/mem_limit
mkswap --label zram7 /dev/zram7
swapon --priority 32767 /dev/zram7

# /tmp on ZRAM
#cat /sys/class/zram-control/hot_add
#echo zstd > /sys/block/zram8/comp_algorithm
#echo 6G > /sys/block/zram8/disksize
#mkfs.ext4 /dev/zram8
#mount -t tmpfs -o size=75%,nr_inodes=524288 tmpfs /tmp
#mount /dev/zram8 /tmp
#chmod 1777 /tmp

#Virtual Memory Kernel Tweaks default for low-memory systems
#sysctl -w vm.swappiness=80
sysctl -w vm.swappiness=5
#sysctl -w vm.vfs_cache_pressure=500
sysctl -w vm.vfs_cache_pressure=75
sysctl -w vm.page-cluster=0
#sysctl -w vm.dirty_background_ratio=1
#sysctl -w vm.dirty_ratio=50
#sysctl -w vm.page-cluster=0

This is /usr/local/bin/zram-off:

#!/bin/bash
swapoff /dev/zram0
swapoff /dev/zram1
swapoff /dev/zram2
swapoff /dev/zram3
swapoff /dev/zram4
swapoff /dev/zram5
swapoff /dev/zram6
swapoff /dev/zram7

umount /dev/zram0
umount /dev/zram1
umount /dev/zram2
umount /dev/zram3
umount /dev/zram4
umount /dev/zram5
umount /dev/zram6
umount /dev/zram7
#umount /dev/zram8


echo 0 > /sys/class/zram-control/hot_remove
# Not required, but creating a blank uninitalzed drive
# after removing one may be desired
cat /sys/class/zram-control/hot_add

And this is /etc/systemd/system/zram-config.service:

[Unit]
Description=ZRAM Config (swap, /tmp, and kernel parameter tweaks)
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/zram-on
ExecStop=/usr/local/bin/zram-off
RemainAfterExit=yes
#TimeoutStopSec=600
#OOMScoreAdjust=-1000
#CPUSchedulingPolicy=fifo
#CPUSchedulingPriority=20
#CapabilityBoundingSet=CAP_SYS_ADMIN CAP_SYS_MODULE
#DeviceAllow=block-blkext r
#DeviceAllow=block-device-mapper r
#DeviceAllow=block-loop rw
#DeviceAllow=block-sd r
#DeviceAllow=block-virtblk rw
#DeviceAllow=block-zram rw
NoNewPrivileges=true
#PrivateNetwork=true
#ProtectClock=true
#ProtectControlGroups=true
ProtectHome=read-only
ProtectHostname=true
ProtectKernelLogs=true
#ProtectSystem=full
#RestrictNamespaces=true
#RestrictSUIDSGID=true
#SystemCallArchitectures=native
#SystemCallErrorNumber=EPERM
#SystemCallFilter=@system-service @swap @module

[Install]
WantedBy = multi-user.target

The systemd sandboxing needs some more finetuning ::

This approach works absolutely reliable for me. Might be interesting for othre users who have trouble with systemd-swap.

1 Like