How I installed every Garuda spin on one partition
Inspired by @dalto's post here, for the past few months I have been experimenting with adding multiple Linux installations to a single Btrfs partition using subvolumes. In this topic, I describe how I have installed every Garuda Raptor spin on one partition with this method, using the rEFInd Boot Manager to easily boot to any of the installations, with options for extra kernels and fallback images. In this setup, an option for booting to Grub is preserved as well for easily booting into Snapper snapshots with grub-btrfs
.
This topic is largely intended to showcase Btrfs subvolume multibooting and the rEFInd boot manager, but is written in the style of a how-to in case anyone would like to follow along, or adopt a few ideas into their own setup. A much simpler version of this guide, without the rEFInd setup and some of the other steps, has been posted on the Garuda Wiki here: Multiple installations on one partition | Garuda Linux wiki
Like all multibooting setups, this should be considered not offically supported by the Garuda team. Multibooting adds extra layers of complexity to a system that can make troubleshooting more difficult. Be warned that manipulating subvolumes and bootloader configurations like described in this topic can make your system unbootable if you make a mistake.
Why would anyone do this?
I'll admit, thirteen installations is a bit much. Keeping them all up to date is bound to grow a little too time consuming. Sooner or later I will let a few of them go to free up some space on the disk.
But that's just the thing: adding an installation I want to test out--or removing it when I am done--is trivial when the installations are contained in subvolumes. Adding partitions to the disk or resizing filesystems is not needed. You can set up a fresh installation in less than ten minutes, tinker around for an hour or two, and then blow away the subvolumes when you are done and it's like it was never there.
This provides a nice alternative to testing a distro in a virtual machine. Having the installation "on the metal" often yeilds a better performance, and eliminates the possiblity of unexpected behavior caused by virtualization.
This is also a great way to enjoy having multiple desktop environments. Installing one desktop environment on top of another can cause conflicts and breakages, many of which can be difficult to troubleshoot. Keeping them as separate installations allows you to configure each system as deeply as you wish without having to worry about breaking something on another desktop environment.
Thanks to subvolumes, it is easy to share files or directories between your systems. Multiple systems can share a common subvolume, as described in this topic, or If you need something from your other installation that isn't stored on a shared resource you can always just reach into its subvolume and grab it.
Above all, I found this project to be a fun way to learn more about the interesting and complex features of Btrfs.
Getting started
- A Btrfs filesystem is needed for this, obviously. One long, contiguous Btrfs partition on your disk is best.
- An EFI partition is needed as well, since we are using the rEFInd Boot Manager. Systems that boot in legacy (BIOS) mode without an EFI partition can still multiboot with subvolumes, but will need to set it up with Grub only which has some disadvantages (it is much more difficult to organize when you have a lot of installations up, especially the bootable snapshots).
- Encryption is a perfectly worthy consideration, and is absolutely possible with this kind of setup. However, an encrypted setup is beyond the scope of this topic.
Rename the default subvolumes
Create a mount point outside the top-level subvolumes. In my example here, my Btrfs partition is on nvme0n1p2
. Obviously change that to whatever your Btrfs partition happens to be.
sudo mkdir /mnt/top-level_subvolume
sudo mount -o subvolid=0 /dev/nvme0n1p2 /mnt/top-level_subvolume
cd /mnt/top-level_subvolume
Rename all the subvolumes. Replace gnome
, gnome_cache
, etc with whatever names you wish to use for your subvolumes.
sudo mv @ gnome
sudo mv @cache gnome_cache
sudo mv @home gnome_home
sudo mv @log gnome_log
sudo mv @root gnome_root
sudo mv @srv gnome_srv
sudo mv @tmp gnome_tmp
It is customary to mark these subvolumes with "@". This convention can be preserved or not depending on your preference. In this setup I have decided not to use "@" in the subvolume names.
Set up the shared subvolume
While you have the top-level subvolumes easily accessible (i.e. you are still in the /mnt/top-level_subvolume
directory), create one more top-level subvolume to use as a shared resource.
sudo btrfs subvolume create shared_subvolume
It doesn't have to be called "shared_subvolume
", it can be called whatever you want.
The shared subvolume is not needed, but it is handy so you can boot into any installation and pick up where you left off with your work.
There are many ways to create shared resources between installations using Btrfs subvolumes; this particular method is one that I personally find simple to use and understand. It uses only one subvolume and symlinks.
Create a mount point for the shared subvolume. It can be named whatever you like and doesn't have to be in the /mnt
directory if you don't want it to be, that is up to your preference.
sudo mkdir /mnt/share
Take ownership of it (obviously use your own username, if it is not jeremy
):
sudo chown -R jeremy:jeremy /mnt/share
Creating the subvolume and mountpoint is enough for now; we'll come back to it later.
Update /etc/fstab
It is important to get your /etc/fstab
updated before you reboot. Take extra care to ensure the entries are accurate.
Update the subvol=
values to the new names (i.e. [email protected]
should be changed to subvol=gnome
, and so on.) For example:
<device> <mount point> <type> <options>
UUID=xxxx-xxxx /boot/efi vfat defaults,noatime 0 2
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / btrfs subvol=/gnome,noatime,compress=zstd 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /home btrfs subvol=/gnome_home,noatime,compress=zstd 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /root btrfs subvol=/gnome_root,noatime,compress=zstd 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /srv btrfs subvol=/gnome_srv,noatime,compress=zstd 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /var/cache btrfs subvol=/gnome_cache,noatime,compress=zstd 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /var/log btrfs subvol=/gnome_log,noatime,compress=zstd 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /var/tmp btrfs subvol=/gnome_tmp,noatime,compress=zstd 0 0
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
Add a new line to /etc/fstab
to mount the shared resource.
<device> <mount point> <type> <options>
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /mnt/share btrfs subvol=/shared_subvolume,noatime,compress=zstd,discard=async,ssd 0 0
In the example above, I simply copied one of the other lines from the default entries for this disk and changed the mount point and subvolume description.
Re-mount the partitions. This will mount the shared subvolume as well, since it has been set up in fstab.
sudo systemctl daemon-reload
sudo mount -a
Update Grub
Change the name of the boot directory from "Garuda" to something else so it won't create a conflict when you install the next ISO. The next installation will also name the boot directory "Garuda", which will simply overwrite any directory with the same name (this would make our first installation unbootable).
sudo mv /boot/efi/EFI/Garuda /boot/efi/EFI/Gnome
Change the GRUB_DISTRIBUTOR=
line in /etc/default/grub
to match whatever you just named the boot directory in /boot/efi/EFI/
.
sudo micro /etc/default/grub
GRUB_DISTRIBUTOR="Gnome"
If this line does not match the name of a directory in /boot/efi/EFI
(the one we just changed above), when you run the grub-install
script it will automatically make a completely new directory.
Optionally, disable the Grub OS-prober. OS-prober will not be needed since we will be using rEFInd, and disabling it will make the Grub menu much less crowded when we do boot to Grub for restoring a snapshot.
Find the line that says GRUB_DISABLE_OS_PROBER=false
and comment it out by adding a #
in front of the line.
#GRUB_DISABLE_OS_PROBER=false
Next, run the Grub installation script.
sudo grub-install ...
I have put "...
" to mean "add whatever options are specifically relevant for your Grub installation". In some cases, grub-install
or grub-install --no-nvram
is enough. If you are not sure, refer to the document here: GRUB - ArchWiki
Regenerate the Grub configuration file.
sudo update-grub
It is fine to reboot here if you wish; at this point you should be all set to get back into your new installation with Grub.
Install rEFInd
Install the rEFInd package:
sudo pacman -S refind
Run the refind-install
script:
refind-install
Setting up the manual boot stanza
rEFInd has an automatic kernel discovery capability, and will automatically generate a config file stored in /boot
with the kernels which allows for a boot entry to be created without the nuisance of setting up a manual boot stanza. I wrote my first draft of this topic with the whole setup based on this feature, so setting up the manual boot stanza could be avoided--it's kind of clunky to set up and can make starting with rEFInd more challenging.
However, there are a few issues with the automatic process that made me reconsider. I won't bore you with the details, it is enough to know that for "reasons" the manual boot stanza is better for this kind of setup.
"No really, I must know the details!"
-
The automatic setup does not handle booting with multiple kernels well without additional configuration. When there are multiple kernels and multiple initramfs images in
/boot
it will sometimes pair a kernel with an incorrect image, and the system won't boot unless you pull up the extra boot options menu and select an entry that has a valid configuration. To address this, a value inrefind.conf
calledextra_kernel_version_strings
has to be set up with a comma-delimited list of strings to fascilitate the kernel detection process. This isn't a great hardship, however it started making the auto-detection method feel less like "easy-mode" and more like it was just complicated in a different way than setting up the manual boot stanzas. -
rEFInd does not look in non-standard root subvolumes (i.e. the root subvolume is named something other than
@
) by default. In the case of a boot directory in a non-standard root subvolume, such as all of the ones in this topic, the subvolume holding the kernel and initramfs image must be explicitly stated inrefind.conf
. Again, this isn't specifically difficult to do (uncomment thealso_scan_dirs
line and specify all subvolumes to look for kernels in asname_of_subvolume/boot
), but it was just one more factor contributing to making the "automatic" process just as complicated and clunky as setting up a proper boot stanza. -
The process for adding a custom icon to an auto-generated boot entry involves adding a .png to
/boot
which is has the same name as the kernel but with the .png extension (for examplevmlinuz-linux-zen.png
). This works fine, except if you are continuing to use Grub like in our case. Grub picks up the .png as a bootable entry for some reason, and I found it would even set the .png as the default boot option. So if you wanted to boot to Grub, you had to either just not choose the default option, or you had to change the default option in Garuda Boot Options or/etc/default/grub
so it wasn't pointing to the .png. It was just a little messy! There may be some way to correct this, but the workaround is unlikely to be simpler than just creating a manual boot stanza. -
Booting to Grub had to be a separate menu option with the automatic setup, and required setting up a stanza with all of the Grub entries in
refind.conf
. They were like mini-stanzas--much easier to set up than a normal boot stanza, but still. Ultimately, being able to have the Grub option in the submenu is much cleaner, and makes it easier to access the correct Grub menu when you need it.
In the end there were so many little gotchas and extra steps needed to preserve the "easy, automatic setup" method that it really wasn't that easy or automatic anymore.
Getting the boot stanza set up is a little tricky, but not overwhelmingly so. Additionally, when you are multibooting off of a single partition you can copy some of the values (volume
and root
, for example) from one stanza to the next as you set them up because they will be the same on every installation.
Here, I'll explain what the needed components of the stanza are and how you can find them. Additional resources for writing the stanzas can be found on the ArchWiki here, the rEFInd website here, or in refind.conf
itself--there are plenty of example stanzas and documentation about all the different available options written into the file.
Open refind.conf
in your editor. Since it is on the EFI partition, you will have to open it with sudo or as root.
sudo micro /boot/efi/EFI/refind/refind.conf
Scroll all the way down to the bottom of the file. You will see many boot stanza examples on the way down, with disabled
written in as the last option--this is a simple way to disable a boot stanza, instead of commenting out the whole thing line by line.
Find some space in the file to set up your stanza. It can be before or after the example stanzas (it doesn't matter).
The basic layout of the stanza looks like this (my comments are CAPITALIZED and enclosed in square brackets, the curly brackets are part of the stanza):
menuentry [NAME] {
icon [PATH TO ICON RELATIVE TO EFI PARTITION]
volume [FILESYSTEM LABEL OR PARTUUID OF THE PARTITION YOU ARE BOOTING TO]
loader [PATH TO THE KERNEL, STARTING WITH SUBVOLUME]
initrd [PATH TO THE INITRAMFS IMAGE]
graphics on
options [MUST INCLUDE "root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx rw rootflags=subvol=SUBVOLUME" FOLLOWED BY ANY NEEDED KERNEL PARAMETERS]
}
menuentry
The menuentry
can be named whatever you like, however if it contains a space (more than one word) you must enclose it in quotes (for example, "Garuda Gnome").
"Show me!"
A menuentry
value with no space:
menuentry Garuda {
...
A menuentry
value with a space:
menuentry "Garuda Gnome" {
...
icon
The icon
entry is the path to the icon relative to the EFI partition (i.e. /boot/efi
). So to use the Arch icon at /boot/efi/EFI/refind/icons/os_arch.png
, you write in the stanza /EFI/refind/icons/os_arch.png
.
There is a default icons directory at /boot/efi/EFI/refind/icons
, which has an assortment of .pngs for some popular Linux distros, or you can use a custom icon.
"Show me!"
The easiest way to set up your custom icon is to save it on the EFI partition. I recommend not saving it in /boot/efi/EFI/refind/icons
because the directory gets overwritten (rather, saved to a backup file) when the refind-install
script is run. You will have to restore the directory to get your icons back in that case.
Instead, either make your own directory or just store them in /boot/efi/EFI/refind
. For the latter case, you would describe the path in your stanza like so:
icon /EFI/refind/MY_CUSTOM_ICON.png
Again, the path is relative to the EFI partition (i.e. relative to /boot/efi/
in a default Garuda setup).
Alternatively, if you put the icon
entry after the volume
entry then it will be relative to the volume
entry instead of the EFI partition (if, for example, you wanted to store your icon on the Btrfs partition instead of the EFI partition.)
For example, this points to a .png in the boot directory (in this example, the root subvolume is named "@
"):
menuentry Garuda {
volume xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
icon /@/boot/MY_CUSTOM_ICON.png
...
volume
The volume
entry is the partition where the kernels and images are stored (i.e. the Btrfs partiton). You must describe the volume with either the partition name, the filesystem label, or the PARTUUID--not the filesystem UUID.
"Show me!"
Run sudo blkid -s PARTUUID -o value /dev/sdXY
, where sdXY
is the Btrfs partition.
sudo blkid -s PARTUUID -o value /dev/nvme0n1p2
e798748f-c287-43e6-b675-cf376345f211
Or, just run sudo blkid
and find the PARTUUID=
value for the Btrfs partition in the output.
Add that value to the volume
entry:
menuentry "Garuda Gnome" {
icon icon /EFI/refind/gnome_logo.png
volume e798748f-c287-43e6-b675-cf376345f211
...
If you labeled your filesystem in the installer (in the step where you set up the mount points for /boot/efi
and /
), you can use the label in the stanza instead of the PARTUUID if you want to.
loader
and initrd
The loader
points to the kernel and the initrd
entry points to the initial ramdisk. The path is relative to the root of the volume
entry, and needs to be preceded by the subvolume.
"Show me!"
Look in your /boot
directory and find the kernel and the initial ramdisk files:
Write the path to each file starting with the root subvolume. In this example, the root subvolume is named "gnome
":
menuentry "Garuda Gnome" {
icon /EFI/refind/gnome_logo.png
volume e798748f-c287-43e6-b675-cf376345f211
loader /gnome/boot/vmlinuz-linux-zen
initrd /gnome/boot/initramfs-linux-zen.img
...
graphics
The graphics
option needs to be set to on
if you wish for rEFInd to boot in graphics-mode, instead of just using a console. This will be needed for the Plymouth splash screen, and I recommend setting it if you want to boot to Grub as well. If you don't, Grub will still load but it will be in a low-resolution mode that can make it difficult to see all the options in the menu on certain displays.
If you have set graphics on
in your stanza but you are still not getting the splash screen, you may need to enable early kernel mode setting as well. See here for how to set that up on a dracut system, or here for a mkinitcpio system.
options
This one is a little tricky because there are a few components you need to put together to get the line right.
- Identify the root filesystem by UUID (you may copy from
/etc/fstab
orlsblk -f
) and specify read-write access (rw
), for example:
root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw
- Add
rootflags=subvol=*root_subvolume*
(as described here), for example:
rootflags=subvol=gnome
- Add your kernel parameters (you may copy from the
GRUB_CMDLINE_LINUX_DEFAULT=
line in/etc/default/grub
), for example:
quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 nvme.noacpi=1 nowatchdog
Combine these three components on one line enclosed in quotation marks ("..."), and that is your options
entry:
menuentry "Garuda Gnome" {
icon /EFI/refind/gnome_logo.png
volume e798748f-c287-43e6-b675-cf376345f211
loader /gnome/boot/vmlinuz-linux-zen
initrd /gnome/boot/initramfs-linux-zen.img
graphics on
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 nvme.noacpi=1 nowatchdog"
...
An easy way to get a pre-made options
entry is to grab it out of the auto-generated refind_linux.conf
file.
"Show me!"
When you run refind-install
, the script will generate a configuration file that can serve as a pre-made boot entry. This file is related to the "automatic" process referenced earlier in the topic. You can actually use it to boot if you want, you just need to add the root subvolume it is stored in to the also_scan_dirs
line in refind.conf
. Although we aren't covering that approach in this topic, you can still use this file to your advantage.
If you did not run refind-install
on the installation you are booted into, you can generate this file by running mkrlconf
.
Print the contents of the file:
cat /boot/refind_linux.conf
File: /boot/refind_linux.conf
"Boot with standard options" "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 nvme.noacpi=1 nowatchdog"
"Boot to single-user mode" "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 single nvme.noacpi=1 nowatchdog"
"Boot with minimal options" "ro root=/dev/nvme0n1p2"
That part after "Boot with standard options" is your whole options
line written out, based on whatever kernel parameters you are booted with when the refind-install
or mkrlconf
script generates the file.
To be clear, I am talking about this part of the file:
Copy the whole line, including the quotes, and paste into your boot stanza.
If you are using mkinitcpio to build your initramfs instead of dracut, you need to add
initrd=boot\cpu_manufacturer-ucode.img
to the options line as well in order to get your microcode loaded, see here: Microcode - ArchWiki
Submenu entries
Submenu entries are a special option that can be added to a manual boot stanza. When a menu entry is highlighted on the rEFInd boot screen and you press Enter, it will boot the main entry in the stanza. If you press Tab instead, it will bring up a menu with all of the submenu entries you have set up.
Here is an example of a submenu:
This is a quick and easy way to access different boot options like alternate kernels or a fallback image, or even booting to the Grub bootloader for booting directly into Btrfs snapshots.
Setting up the submenu entries is pretty simple once you have the boot stanza finished. Basically you just need override any lines in the stanza that are different for the submenu option.
To add a submenu entry for booting with the fallback image, the only line that needs to change is the initrd
line:
...
submenuentry "Zen fallback" {
initrd /gnome/boot/initramfs-linux-zen-fallback.img
}
...
If all of the other lines in the stanza (volume
, loader
, and options
) are going to remain the same for this boot option, then you are done! No need to specify any additional lines unless they deviate from the main stanza.
If you wanted your fallback boot entry to have different kernel parameters, you can add a new options line as well. For example, if you want your fallback entry to boot without the quiet boot kernel parameters (quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3
) so you can see the journal output at startup, simply re-do the options
line with those options omitted.
...
submenuentry "Zen fallback" {
initrd /gnome/boot/initramfs-linux-zen-fallback.img
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome nvme.noacpi=1 nowatchdog"
}
...
To create a submenu entry for an alternate kernel, specify override values for loader
and initrd
:
...
submenuentry "LTS kernel" {
loader /sway/boot/vmlinuz-linux-lts
initrd /sway/boot/initramfs-linux-lts.img
}
...
And finally, to create an entry to boot to Grub you need to override the volume
entry (since the target in this case is on the EFI partition, not the Btrfs partition which is the volume
specified in the main stanza) and also the loader
entry.
In the case of booting to Grub, the loader
does not point to a kernel, but rather it points to the grubx64.efi
file for that Grub installation. If you recall, back in the "Update Grub" section earlier in the topic, we renamed /boot/efi/EFI/Garuda
to something else (in the example it was changed to /boot/efi/EFI/Gnome
). That is the directory we need to point to in this submenu entry.
...
submenuentry Grub {
volume 15a71d3d-f2ef-4513-8e6b-a65458ffbb75
loader /EFI/Gnome/grubx64.efi
}
...
The volume
in the example is the PARTUUID of the EFI partition (or you may use a filesystem label if you made one). Check sudo blkid
to find the PARTUUID or the label if you have one.
The boot stanza is complete!
menuentry "Garuda Gnome" {
icon /EFI/refind/gnome_logo.png
volume e798748f-c287-43e6-b675-cf376345f211
loader /gnome/boot/vmlinuz-linux-zen
initrd /gnome/boot/initramfs-linux-zen.img
graphics on
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 nvme.noacpi=1 nowatchdog"
submenuentry "Zen fallback" {
initrd /gnome/boot/initramfs-linux-zen-fallback.img
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome nvme.noacpi=1 nowatchdog"
}
submenuentry "LTS kernel" {
loader /gnome/boot/vmlinuz-linux-lts
initrd /gnome/boot/initramfs-linux-lts.img
}
submenuentry "LTS fallback" {
loader /gnome/boot/vmlinuz-linux-lts
initrd /gnome/boot/initramfs-linux-lts-fallback.img
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=gnome nvme.noacpi=1 nowatchdog"
}
submenuentry Grub {
volume 15a71d3d-f2ef-4513-8e6b-a65458ffbb75
loader /EFI/Gnome/grubx64.efi
}
}
Clean up the Boot menu
After each install, rEFInd will automatically detect the grubx64.efi
file on the EFI partition and will add a boot entry for it (it boots to Grub). It typically gets a Tux icon, unless you install a distribution that is associated with an icon in /boot/efi/EFI/refind/icons
--in which case it will get whatever the icon is.
Once you have Grub set up as a submenu entry in your boot stanza, these auto-generated boot entries are no longer needed--they only add unnecessary clutter to the boot menu. You can hide them by pressing Delete and confirming on the prompt that you would like to hide that boot option.
This feature doesn't actually delete anything; you can retrieve the boot option any time you like by entering the "hidden tags" menu and choosing to restore it.
Install a rEFInd theme
Optionally you may install a theme to stylize your rEFInd boot menu however you wish. There are tons of themes available to choose from; check out some of the options available on GitHub here: refind-theme Β· GitHub Topics Β· GitHub
In the examples in this topic I am using the "Regular" theme, available here: GitHub - bobafetthotmail/refind-theme-regular
Set up the shared resources
The basic idea with the shared resources setup is to move something you want to share on all installations out of it's normal home into the shared subvolume, then replace it with a symlink. For example, to convert the Documents folder to a shared resource:
cd
mv Documents /mnt/share/
ln -s /mnt/share/Documents
The "default" directories in the home folder are an obvious choice for this kind of setup (Downloads, Documents, Pictures, et cetera...I typically do not bother using the Desktop directory, but if you do then obviously that could be included in this process):
Documents -> /mnt/share/Documents
Downloads -> /mnt/share/Downloads
Music -> /mnt/share/Music
Pictures -> /mnt/share/Pictures
Public -> /mnt/share/Public
Templates -> /mnt/share/Templates
Videos -> /mnt/share/Videos
This setup allows you to, for example, download a file in Gnome, then boot into KDE and the file will still be there in your Downloads folder.
This trick can be useful for other directories as well; for example, sharing ~/.ssh
will allow all of your installations to share the same ssh keys.
Adding certain config files to the shared subvolume can be useful too; for example, if you share ~/.config/Signal
it will not only sync all of your Signal-desktop instances, it also allows all of your installations to only count as a single device on your Signal account.
Be cautious! You can break things by "sharing" config files.
It may be tempting to just share your entire /home
subvolume between all your installations, but this can give you conflicting configuration files and break your setup. It is best to leave the dotfiles alone, unless you specifically want them shared.
This system works well for some things, and does not work for others. As an example, web browsers can be complicated to set up with shared configs this way because different desktop environments handle browsers in different ways (different methods for unlocking the keyring, for example, or storing browser data). If you wish to preserve the continuity of a browsing session between desktop environments, use the sync features built into the browser itself.
In general, it is easier, cleaner, and safer to just copy a file from one installation to the next when you set it up instead of moving the file to the shared subvolume and setting up symlinks.
Don't forget, you can always mount a neighboring subvolume and grab a copy of a file if you need something that isn't stored on the shared subvolume.
A general suggestion regarding preserving directory structure inside the shared subvolume
Consider preserving the directory structure of config files inside the shared subvolume. Symlinks can be set up from any directory, and if you end up setting up a lot of them it is not always obvious where they are supposed to go when getting ready to set up the symlinks on a fresh installation.
For example, you could have a directory called syncthing
in your shared subvolume, but without the context of where the directory was moved from it can be easy to forget where you are supposed to symlink it to. Was it ~/.config/syncthing
, or was it ~/.local/share/syncthing
? Where does that syncthing
symlink go?! With one or two symlinks outside of the home directory it might not be an issue, but with many you may end up needing a map.
To keep it straight, add the parent directories so you can remember where the symlinks should point.
mkdir /mnt/share/.config
mv ~/.config/syncthing /mnt/share/.config/syncthing
ln -s /mnt/share/.config/syncthing ~/.config/
Now the syncthing
symlink actually describes the correct path of the file.
~/.config/syncthing -> /mnt/share/.config/syncthing
When you are setting up a fresh distribution, you can review the shared directory and actually see where the symlinks need to be set up instead of just seeing a jumble of random files.
/mnt/share
βββ .config
β βββ syncthing
βββ Documents
βββ Downloads
βββ Music
...
Once you have finished setting up your shared resources, feel free to test things out to make sure everything is working as expected. Move some files around, restore a snapshot, whatever. If everything seems good, grab your USB stick and start with the next installation.
Add another installation
It's time to install another spin! Boot to the installer from a USB like you would normally do. Choose the manual partitioning option.
Select the EFI partition and click on Edit. Be very careful with the selections you make--you want to keep the contents of the partitions--do not format or you will lose the first installation.
EFI partition:
- Content: Keep
- Mount point:
/boot/efi
Next, select the Btrfs partition and click on Edit.
Btrfs partition:
- Content: Keep
- Mount point:
/
While you are adding the mount points, consider adding a filesystem label if you haven't already. You can use the filesystem label instead of the PARTUUID for the
volume
entry on your boot stanzas.
That's it! Proceed with the installation.
Once you are booted into the fresh installation, the initial setup is very similar to the first time around; I will summarize in brief anything already explained in more detail above, to have a short version easier to follow along.
Mount the top-level subvolume and rename the new subvolumes--the same process as the first distro, only this time obviously you should pick different names. Replace dr460nized
, dr460nized_cache
, etc with whatever names you wish to use for your subvolumes, and nvme0n1p2
with whatever your Btrfs partition is.
sudo mkdir /mnt/top-level_subvolume
sudo mount -o subvolid=0 /dev/nvme0n1p2 /mnt/top-level_subvolume
cd /mnt/top-level_subvolume
sudo mv @ dr460nized
sudo mv @cache dr460nized_cache
sudo mv @home dr460nized_home
sudo mv @log dr460nized_log
sudo mv @root dr460nized_root
sudo mv @srv dr460nized_srv
sudo mv @tmp dr460nized_tmp
Make your shared directory and take ownership of it (replace with your username).
sudo mkdir /mnt/share
sudo chown -R jeremy:jeremy /mnt/share
Edit your /etc/fstab
with the new subvolume names.
micro /etc/fstab
Don't forget to add in a line for your shared subvolume!
...
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /mnt/share btrfs subvol=/shared_subvolume,noatime,compress=zstd,discard=async,ssd 0 0
...
Re-mount your partitions.
sudo systemctl daemon-reload
sudo mount -a
Edit the name of the boot directory, and change the GRUB_DISTRIBUTOR
line in /etc/default/grub
. Disable OS prober by commenting the =false
line.
sudo mv /boot/efi/EFI/Garuda /boot/efi/EFI/Dr460nized
sudo micro /etc/default/grub
...
GRUB_DISTRIBUTOR="Dr460nized"
...
#GRUB_DISABLE_OS_PROBER=false
...
Optional: before regenerating the Grub configuration file, update
/etc/default/grub-btrfs/config
with the subvolumes to exclude when scanning for snapshots--see "Clean up the snapshots boot menu" below.
Reinstall Grub and regenerate the Grub configuration file.
sudo grub-install --no-nvram
sudo update-grub
Install rEFInd.
sudo pacman -S refind
Set rEFInd as the default bootloader.
sudo refind-mkdefault
"What's this refind-mkdefault? What happened to refind-install?"
Running the refind-install
script is only necessary on the first installation, to get everything set up on the EFI partition and make a NVRAM entry. On subsequent installations, running refind-mkdefault
is enough.
refind-mkdefault
is a script that checks if rEFInd is the default boot option, and changes it to the default if not--basically an alternative to messing around with efibootmgr
. It is handy because if you need to reinstall Grub for any reason it will re-set itself as the default bootloader, and this script will easily fix it for you.
An advanced use of this script would be adding it to a startup task or similar, as described in the man page:
"The intent is that refind-mkdefault can be called after booting via GRUB or some other means to restore rEFInd as the default boot program. It can also be placed in a startup and/or shutdown script to restore rEFInd to its default position automatically. Because it does not re-write the boot order if rEFInd is listed as the first boot entry, this practice should be low in risk."
There is no harm in running refind-install
on every installation (your configs will not be overwritten) but it is overkill and will also restore the icon library to default every time you run it. If you have custom icons you are saving in /boot/efi/EFI/refind/icons
, it will save the directory as a backup file and your icons will be gone from your boot screen until you restore it.
Set up your boot stanza in refind.conf
.
"Show me!"
sudo micro /boot/efi/EFI/refind/refind.conf
...
menuentry "Garuda Dr460nized" {
icon /EFI/refind/dr460nized_logo.png
volume Btrfs
loader /dr460nized/boot/vmlinuz-linux-zen
initrd /dr460nized/boot/initramfs-linux-zen.img
graphics on
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=dr460nized quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0 loglevel=3 nvme.noacpi=1 nowatchdog"
submenuentry "Zen fallback" {
initrd /dr460nized/boot/initramfs-linux-zen-fallback.img
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=dr460nized nvme.noacpi=1 nowatchdog"
}
submenuentry "LTS kernel" {
loader /dr460nized/boot/vmlinuz-linux-lts
initrd /dr460nized/boot/initramfs-linux-lts.img
}
submenuentry "LTS fallback" {
loader /dr460nized/boot/vmlinuz-linux-lts
initrd /dr460nized/boot/initramfs-linux-lts-fallback.img
options "root=UUID=5fa54f34-b5fc-40be-8092-8ba34ced9eba rw rootflags=subvol=dr460nized nvme.noacpi=1 nowatchdog"
}
submenuentry Grub {
volume EFI
loader /EFI/Dr460nized/grubx64.efi
}
}
Revisiting the shared subvolume
Now that the shared subvolume is already set up, instead of moving stuff into it you just need to set up the symlinks. On a new installation, the default home directories will be empty so you can just use rmdir
.
rmdir Documents/ Downloads/ Music/ Pictures/ Public/ Templates/ Videos/
Then set up the symlinks.
ln -s /mnt/share/Documents/ /mnt/share/Downloads/ /mnt/share/Music/ /mnt/share/Pictures/ /mnt/share/Public/ /mnt/share/Templates/ /mnt/share/Videos/ ~/
Directories that are not empty you can delete with rm -rf
. It should go without saying, but: please be cautious with this command. Once you have deleted the directories, set up the symlink to the shared resource.
rm -rf ~/.config/syncthing
ln -s /mnt/share/.config/syncthing ~/.config/
On a fresh installation, you may not even need to delete the config directories--check first to see if they have been set up yet.
That's it! Now you are multibooting two installations off of one partition with a shared subvolume between them.
The "Add another installation" section may be repeated for as many installations as needed.
Clean up the snapshots boot menu
Although at this point we have a separate Grub menu for each installation, and OS-prober has been disabled so there are not tons of boot options crowding the menu, there is still an issue with snapshots--namely, all snapshots on all subvolumes are detected and added to each Grub menu.
You may pull up the Cinnamon Grub menu to restore a snapshot, but the Cinnamon snapshots are all intermixed with snapshots from LXQT and Gnome and Sway, and depending on the snapshot limit set in /etc/default/grub-btrfs/config
(default is 50) there may not even be that many Cinnamon snapshots to begin with--the 50 snapshot limit gets eaten up by the snapshots from other installations!
To resolve this, open /etc/default/grub-btrfs/config
and find this line:
GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")
Here, you can specify a directory or subvolume which will be recursively ignored while scanning for snapshots. The Docker paths are included there by default; it is fine to leave them as they are or delete them if they are not needed, according to your preference.
What you want to do is list out all the root subvolumes (except the one your are booted into) and add them to this line. The root subvolumes are the ones that have been renamed from "@
" all the way in the beginning of this topic, in the "Rename the default subvolumes" section. If it helps to see them printed out, run something like this:
sudo btrfs subvolume list / | grep "level 5 "
Then, while you are in /etc/default/grub-btrfs/config
, there is another line in this file which should be changed:
GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@")
This is a similar idea, but it is a non-recursive path. Basically it keeps the actual system subvolume from showing up on the snapshot menu. This value should be changed from "@
" to whatever you have named the root subvolume.
"Show me!"
Let's say I am currently booted into a KDE installation, and the root subvolume is named kde
. I want to change @
to kde
in the SPECIFIC_PATH
option like this:
GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("kde")
That will stop the main system subvolume (which is not a snapshot) from showing up on the snapshot menu.
Next, I will search for any other root subvolumes on the system.
sudo btrfs subvolume list / | grep "level 5 "
ID 256 gen 80942 top level 5 path gnome
ID 257 gen 80942 top level 5 path gnome_home
ID 258 gen 80520 top level 5 path gnome_root
ID 259 gen 80392 top level 5 path gnome_srv
ID 260 gen 80941 top level 5 path gnome_cache
ID 261 gen 80942 top level 5 path gnome_log
ID 262 gen 80941 top level 5 path gnome_tmp
ID 621 gen 80780 top level 5 path shared_subvolume
ID 631 gen 79700 top level 5 path dr460nized
ID 632 gen 79700 top level 5 path dr460nized_home
ID 633 gen 79620 top level 5 path dr460nized_root
ID 634 gen 79491 top level 5 path dr460nized_srv
ID 635 gen 79700 top level 5 path dr460nized_cache
ID 636 gen 79700 top level 5 path dr460nized_log
ID 637 gen 79700 top level 5 path dr460nized_tmp
ID 256 gen 80942 top level 5 path xfce
ID 257 gen 80942 top level 5 path xfce_home
ID 258 gen 80520 top level 5 path xfce_root
ID 259 gen 80392 top level 5 path xfce_srv
ID 260 gen 80941 top level 5 path xfce_cache
ID 261 gen 80942 top level 5 path xfce_log
ID 262 gen 80941 top level 5 path xfce_tmp
ID 631 gen 79700 top level 5 path cinnamon
ID 632 gen 79700 top level 5 path cinnamon_home
ID 633 gen 79620 top level 5 path cinnamon_root
ID 634 gen 79491 top level 5 path cinnamon_srv
ID 635 gen 79700 top level 5 path cinnamon_cache
ID 636 gen 79700 top level 5 path cinnamon_log
ID 637 gen 79700 top level 5 path cinnamon_tmp
ID 256 gen 80942 top level 5 path kde
ID 257 gen 80942 top level 5 path kde_home
ID 258 gen 80520 top level 5 path kde_root
ID 259 gen 80392 top level 5 path kde_srv
ID 260 gen 80941 top level 5 path kde_cache
ID 261 gen 80942 top level 5 path kde_log
ID 262 gen 80941 top level 5 path kde_tmp
ID 631 gen 79700 top level 5 path wayfire
ID 632 gen 79700 top level 5 path wayfire_home
ID 633 gen 79620 top level 5 path wayfire_root
ID 634 gen 79491 top level 5 path wayfire_srv
ID 635 gen 79700 top level 5 path wayfire_cache
ID 636 gen 79700 top level 5 path wayfire_log
ID 637 gen 79700 top level 5 path wayfire_tmp
I can see besides kde
, the other root subvolumes are gnome
, dr460nized
, xfce
, cinnamon
, and wayfire
.
Add these root subvolumes to the PREFIX_PATH
option like this:
GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "gnome" "dr460nized" "xfce" "cinnamon" "wayfire")
(I left one of the Docker directories on there for the example. If you use Docker, you will have to decide what is the appropriate option to use in your case.)
After you have finished editing the file, run sudo update-grub
. Instead of 50 snapshots from all different distros, you should see only snapshots relevant to the system you are booted into.
You will have to set up this file on each installation you have. Optionally, consider setting this file up before setting up /etc/default/grub
so when you regenerate the Grub configuration it is not cluttered with extra snapshots right off the bat.
If you add any more installations after /etc/default/grub-btrfs/config
has already been set up, you will have to circle back and add the new root subvolume to the IGNORE_PREFIX_PATH
list.
Deleting an installation
Deleting an installation is a simple matter of deleting the subvolumes it is contained in. It can be a little tricky in the case of subvolumes that contain read-only subvolumes, such as with snapshots.
First, get back outside the top level subvolume.
sudo mount -o subvolid=0 /dev/nvme0n1p2 /mnt/top-level_subvolume
cd /mnt/top-level_subvolume/
In this example I am removing an XFCE installation. All of the subvolumes have "xfce" in the name:
ls | grep xfce
drwxr-xr-x - root xfce
drwxr-xr-x - root xfce_cache
drwxr-xr-x - root xfce_home
drwxr-xr-x - root xfce_log
drwxr-x--- - root xfce_root
drwxr-xr-x - root xfce_srv
drwxrwxrwt - root xfce_tmp
Subvolumes that do not contain read-only subvolumes can be removed with plain old rm -rf
. You will need to elevate, since these subvolumes are owned by root.
As always, be very cautious when running a command with
rm -rf
, especially when elevated. You can easily destroy your system by targeting the wrong subvolume or directory.
sudo rm -rf xfce_cache xfce_home xfce_log xfce_root xfce_srv xfce_tmp
If you try that on a subvolume that contains Btrfs snapshots, however, you will get a barrage of errors reading something like rm: cannot remove 'xfce/.snapshots/1/snapshot/path/to/file': Read-only file system
.
Read-only subvolumes can be deleted with btrfs subvolume delete
--but they cannot be deleted recursively, however:
sudo btrfs subvolume delete xfce
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce'
ERROR: Could not destroy subvolume/snapshot: Directory not empty
The path must be followed down, and the subvolumes deleted directly.
Let's see what subvolumes are inside xfce
:
sudo btrfs subvolume list -o xfce/ -t
ID gen top level path
-- --- --------- ----
953 83334 946 xfce/.snapshots
That is the .snapshots
subvolume--which is, of course, a parent subvolume. Like the root subvolume, It similarly cannot be directly deleted because it contains the read-only subvolumes. So, we follow the path deeper:
sudo btrfs subvolume list -o xfce/.snapshots -t
ID gen top level path
-- --- --------- ----
954 83176 953 xfce/.snapshots/1/snapshot
955 83183 953 xfce/.snapshots/2/snapshot
956 83189 953 xfce/.snapshots/3/snapshot
957 83190 953 xfce/.snapshots/4/snapshot
958 83219 953 xfce/.snapshots/5/snapshot
959 83220 953 xfce/.snapshots/6/snapshot
960 83233 953 xfce/.snapshots/7/snapshot
961 83234 953 xfce/.snapshots/8/snapshot
962 83236 953 xfce/.snapshots/9/snapshot
963 83237 953 xfce/.snapshots/10/snapshot
Bingo, there are the troubleshome subvolumes.
Mercifully, btrfs subvolume delete
supports globbing, so we can pass a wildcard for the numbered directory and delete them all at once:
sudo btrfs subvolume delete xfce/.snapshots/*/snapshot
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/1/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/2/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/3/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/4/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/5/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/6/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/7/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/8/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/9/snapshot'
Delete subvolume (no-commit): '/mnt/top-level_subvolume/xfce/.snapshots/10/snapshot'
With the read-only subvolumes out of the way, we can now take down the top-level subvolume:
sudo rm -rf xfce
All that's left is to clean up any lingering files on the EFI partition (/boot/efi/EFI/XFCE
, for example), and to delete the boot stanza from refind.conf
.
You can also use Btrfs Assistant to delete the subvolumes rather easily.
"Show me!"
Deleting the subvolumes in Btrfs Assistant is a simple matter of selecting the subvolumes and clicking on delete! Tick the "Include Timeshift and Snapper Snapshots" box if you would like those to be selectable from the menu as well.
Confirm when the prompt asks whether to delete subvolume metadata.
Subvolumes that contain nested subvolumes will initially fail to delete:
Deleting the nested subvolume will succeed, however, so you may simply re-select the subvolume that failed to delete and try again. In the typical case of a home subvolume (@
in this example) and a Snapper snapshots subvolume (@/.snapshots
) it will take three tries to delete all the subvolumes.
Additional considerations
A few worthy considerations that are beyond the scope of this already lengthy topic:
Deduplication
Deduplication is a process which involves identifying data blocks that are identical, but tracked separately, and combining them into an extent. Theoretically, with a multibooting setup like the one described in this topic, deduplication could free up a significant amount of disk space. This should be considered an advanced topic.
btrbk
This is a tool for easily backing up subvolumes to a separate filesystem--an external drive, for example. Consider making some btrbk
backups before you get too in the weeds with deduplication.
Custom Grub config
The setup described in this topic will work perfectly fine without the rEFInd Boot Manager, however the default Grub configuration will list every installation as "Garuda linux on nvme0n1p2", "Garuda Linux on nvme0n1p2", "Garuda LInux on nvme0n1p2", and so on. This makes it difficult to figure out which installation you want to boot to. If you want a pure Grub setup, it would be useful to make a custom Grub configuration so you can label each boot option however you wish.
refind-btrfs
Grub is fairly deeply integrated into some of the Garuda tooling, and not easy to remove without losing a lot of other stuff in the process. grub-btrfs
is also an incredibly effective and easy-to-use snapshot restoration application. For the purpose of this topic, I have left Grub and grub-btrfs
intact for these and other reasons.
However, booting into snapshots from rEFInd itself is also possible thanks to refind-btrfs
. It handles snapshots a little differently than grub-btrfs
(it maintains its own directory of read-write versions of snapshots for booting into) and lends itself to a different workflow (instead of "restoring" a snapshot and the snapshot automatically becoming the main system, the more obvious use case would be to boot into a snapshot for the purpose of repairing or rolling back the main system manually, then go back and boot to the main system), but it works great and is pretty simple to set up if you already have a manual boot stanza.
Closing thoughts
Hopefully you have found this topic useful or interesting. I am happy to read through questions or corrections in the comments, and don't forget there is a significantly abridged version of this topic available in the Garuda Wiki here: Multiple installations on one partition | Garuda Linux wiki.