I moved this discussion from a thread requesting assistance, as this was drifting off topic and of no real help to the member who was looking to repair their system.
Original topic:
1 Like
tbg nails it, I'd add one thing further, running LUKS ontop of btrfs is just asking for failure, and as PerlMonk observes, failure is what he got. I consider, and again, as Perlmonk notes re the tools 'being worked on' for btrfs, btrfs is in my opinion a beta filesystem which for reasons absolutely a mystery to me is being offered as a stable robust mature filesystem, which it is not. It's not even feature complete yet.
Keep in mind the real history: Oracle, before they bought Sun, started btrfs as a weak attempt to create their own ZFS type file system, but as soon as they bought Sun, development of btrfs was dropped at the corporate level, and after a while, it drifted more into what I'd consider the hobbiest development pool of developers.
If you read the btrfs docs, which I did while looking at extending the inxi --logical/-L feature to support btrfs, you'll see that it's not feature complete at all, and also never achieved maturity before leaving it's funded development at Oracle, I don't believe it was even half done at that point.
I dropped development of full btrfs support as soon as I saw how awful their tools were, it wasn't even a pale imititation of zfs tools, the stuff just is not done.
As far as I can tell, the entire snapshot thing is just a solution to a problem that doesn't exist on a real operating system, failed upgrades, I think that's what it's supposed to solve, but if your upgrades fail, you have an issue with your operating system and package management system, not your file system.
If you're going to run a file system and luks, then keep it simple, run a very stable, rock solid file system, like ext4, and run luks on top of it. When you start injecting more and more software layers between you and your data, you're just asking for failure at some point. I ran extensive vm tests on the most absurd combinations of scenarios when developing the --logical feature (lvm, mdraid, luks, stacked one on top of each other) and by the end of that, I was totally convinced to never do anything like that myself for real data.
I really wish distros would stop marketing btrfs as a robust production ready file system, it does too much, it's too complicated, it's never had the heavy duty development required, and, worst, it has no reason to exist beyond one simple fact: Sun adopted a poison pill defense when licensing zfs, they chose a license that was deliberately incompatible with native linux kernel distribution. That is literally the only reason for btrfs to exist, if that license were to change, there would be essentially zero reason to use btrfs beyond having fun with hobbiest stuff. And OpenZFS has removed even more of those obstacles, though the kernel guys can't directly implement it due to the license poison pill, but it can now be run reasonably solidly, but only with known supported kernels. That is the thing Sun was trying to achieve, no native out of the box linux kernel support for ZFS, and that achievement is still in effect.
Obviously, eventually, with enough users, and enough skilled developers, btrfs may one day become mature and robust enough for real production use, but to me, I'll know that day is nearing when their data and reporting tools stop sucking. That day is certainly not today, which is why you'll find inside of inxi a btrfs data stub, where I'd started initial testing, and gave up, somewhat horrified at the terrible job that had been done on the tools.
Not trying to poke any bears here, I just find it really odd that distros are shipping btrfs as if it were robust and ready for prime time when it quite clearly isn't. Filesystems take forever to get working, I've watched the microsoft corporation fail repeatedly delivering their still not there NTFS replacement, for example, making these things work and be reliable is HARD!!! And MS has 10s of thousands of paid developers, who can, and were, told to make it work, yet still failed to do so. it took forever for ext4 to become stable enough to use, and that was simply an extension of long time stable ext2 and ext3, more in the lines of an incremental development.
I was talking to another developer guy who is fairly close to the kernel guys, while doing the --logical feature, and he noted he has a friend who makes a good living recovering data from these types of failed software based storage systems. Just because something is called a filesystem doesn't mean it's a file system in the sense that fat32, ntfs, ext3/4, xfs, ufs, and so on, are file systems, once you get into something like zfs, you need heavy duty hardcore development dollars, Sun had the talent to do zfs (sun was one of the places to work in silicon valley at the time, which meant that they got the real talent) oracle didn't have that talent, and was and remains a crappy place to work, and came up with btrfs.
1 Like
Is there any evidence of this?
Since you don’t use Garuda yourself, I don’t know what your problem is. Nobody is forcing you to use Btrfs.
A “rolling” distribution will suffer a malfunction every now and then, so in order to be able to continue working until the problem is solved, there are the snapshots.
BTW, please tell us the real operating system.
4 Likes
OpenSuse is shipping btrfs as default file system, but I don't follow each distro closely. Of course, they were still shipping reiser3 after hans reiser was sitting in prison for murder, where he still remains, so maybe opensuse isn't the best example, lol.
No need to get defensive, this is just a general comment based on the highly predictable failure when combining luks and btrfs.
My rolling distribution has been rolling since... hmm, I think 2006 now, that's Debian testing in my case. Yes, now and then I have to fix breaks, but the key word there is fix. All my installs are rolling and none of them are breaking, they do need a fix occasionally, which I fix, then go on. That's probably more a reflection of the simple fact that debian apt packages are subject to the most strict packaging requirements of all the distros out there than anything else. But I talk to many Arch type users and I don't hear much about failures, Arch also has done a very good job stabilizing the package pool and making rolling release very practical to run long term. Real backups are the solution there, not snapshots imo.
Anyway, it's just something I've noticed for a while now, re btrfs, and I do have a perspective most people do not have because of the heavy testing and development stuff I do in inxi, and the various people I get to talk to during that process.
This wouldn't be about garuda anyway, this is about luks, current kernels, and the btrfs contained in that kernel, all of which I believe come from Arch? And Arch if I'm not mistaken takes some pride in not messing around much with the upstream source code. Just as, again if I'm not mistaken, Garuda takes some pride in not messing around too much with the Arch base.
I've done this stuff long enough not to confuse the source of packages in a distribution with the actual software packaged in those packages, this will be very unlikely to have anything to do with garuda or arch linux imo, it's an issue with the core software involved, not the distribution of that software, thus my generalized comment about that software, not the distro, which should not matter unless someone messed up the kernel configurations when compiling and packaging the kernel and its file systems, or something.
2 Likes
I'm not being defensive (I hope my translation program is not to blame for this.),
I'm just asking where Garuda Linux, or other distributions, advertise (marketing) the stability of Btrfs.
The nice thing about Linux is that there is something for everyone. 
1 Like
I've noticed an increased tendency of some distros to offer btrfs as if it were an equivalent robust tested solution to actually robust tested solutions like ext4. So it's definitely happening, not unlike when kde 4 was shipped as is by many distros despite kde devs stating categorically that kde 4.0 was NOT ready for prime time, and not to ship it as a kde replacement.
Basically the issue with btrfs is that it needs a lot of work, and any complex scenario, like running luks on btrfs, almost certainly has not had enough heavy testing to be stable or reliable, but most newbies or even somewhat experienced linux users are sort of being told that btrfs is stable and done, when it isn't. It's better than it was, certainly, and is now usable in some circumstances, like when you don't need the data on the file system, but every thread I read on btrfs which had actual sys admins etc talking about it, had much the same responses, it's failed repeatedly, don't use it, which was enough for me. That's under real production use, not single user desktops where it might work fine, but it's definitely not production ready. This was as of 2021, that's the last time I spent time researching it, but the results of the research were always the same from what I could see, users who were testing it seriously and heavily reported failures on a consistent basis.
I like the spirit of testing new things, that's to me where Garuda and Arch and EndeavorOS kind of shine, kind of the leading edge of software development by creating realworld users for quite current software, and thus, finding the issues with that software in a realworld setting, not a test environment. But my overall impression is that Arch kind of solved more or less the rolling failures issue by splitting the primary pool into something similar to what debian does, an unstable and a stable rolling release pool, leaving the unstable pool for developers and beta testers to run. At least, that's my understanding of how Arch pacman pools work.
In terms of snapshots, my suspicion is that they are more important for systems that use AUR, which as far as I can tell have no quality control at all, fairly extensively, than systems that use the primary repos of pacman. AUR would be similar only a bit more risky than Ubuntu's PPAs in terms of long term system stability. I think I understand that right, not positive.
But yes, btrfs is starting to appear, despite btrfs docs themselves saying it's not feature complete yet, and not having complete tools, as an option or a default, which is unfortunate, to me, linux has enough problems trying to retain its relatively stable and small user base, having file system failures just isn't helpful in that scenario, people who don't know better will tend to blame "linux" rather than the file system, then give up on using linux.
2 Likes
Nothing is secure.
What good is your nice ext4 if your hard drive becomes unusable?
It happened to me last year with ext4.
I'm not posting this for the first time.
We also share this on our homepage:
Garuda Linux does not imply that BTRFS snapshots are a full backup solution. If you wish to ensure your data's security you must implement your own full data backup regimen. Garuda is not responsible in any way or manner if a data loss occurs. The user is solely responsible for ensuring the safety of their own data. Likewise, Garuda cannot guarantee that Btrfs snapshots can recover your system to a functional state in the event of a serious system breakage.
4 Likes
I think your assessment is fairly accurate for a production environment. However,that is not what Garuda is trying to evolve into. We are strictly a platform for desktop users, not a corporate platform that requires an emphasis on security and rock solid reliability. While not being on the bleeding edge, Garuda strives to implement new technologies. The Devs here have a dedication to trying all the new "toys" coming down the pipeline.
Will this result in poor choices, sure, at times these tests will prove premature. However, the Devs here are very nimble and responsive when problems occur. I can truthfully say the responsiveness of the Garuda dev team is second to none. Issues are usually corrected in a lightning fast manner here in comparison to most other distros. Being a small distro has its advantages, because decesions can be implemented quickly unlike some behemoth mega corporations. Cough, cough, looking at you M$.
8 Likes
The hardware is a constant, the same for all file systems, the robustness of the file system, or file system stack, is a variable, thus, the more robust the file system, the more robust the data stability. Not complicated. I always chose maximum data integrity and stability, it's just something I don't to mess with, I'm happy letting others have failures, I think I waited many years after ext4 was done before switching to it, for example.
I'd for instance used mdraid for years in production, never realizing it doesn't have any actual data integrity protection, which meant, in the real world, if one disk got corrupted, mdraid would happily copy this corrupted data to the raid data, without any warning. ZFS doesn't have this issue, it does integrity checks.
I can understand the snapshots thing if the underlying OS is not actually stable in terms of upgrades, but that's the only scenario I can see it in, but my choice there is to use a stable system, I'm extremely conservative when it comes to data, might be related to my doing sys admin for years where data integrity was number one for my main client, so we were always heavily redundant, but that was all around backups, not snapshots, which aren't meaningful, as has been noted in this thread, as backups, more as restore points for the os.
It's good garuda makes this clear on home page, that's the right way to do it.
For the record, my clients, if they were using setups made by me, never had any data loss. I don't mean never as in a year or two, I mean never as in almost 2 decades. I never have had any either, but that's just my systems, which are all the same roughly, and aren't as representative since they are made to be reliable yet rolling. I attribute this never to extremely conservative selection of file systems, and extremely careful selection of hardware for storage, coupled with some awareness of when EOL of the storage hardware is.
You can see a lot of these features in sudo inxi -dpa by the way, and even more with sudo inxi -dpLRa if you use advanced software stuff like lvm, luks, mdraid, zfs, etc.
1 Like
tbg, I'm aware of the quality of garuda, that's one of the reasons I tend to track garuda re inxi reports and stuff. I only track a few distros nowadays, garuda is one of them. I like the clarity of mission and purpose that you note, that's important, as I indicated, to the overall health of the global linux ecosystem, without distributions like garuda, endeavoros, and many others over the years who pursued similar ideas, the stuff just doesn't work or develop long term.
4 Likes
This is completely not the case. While this is what many people use them for, that is not the reason for their existence. They exist to be able to access data from a prior point in time.
I am not sure whether Btrfs is “production ready” or not. That being said, it is being used by default in a number of places.
- Synology NAS devices
- Suse
- Fedora
- Garuda
4 Likes
Yeah, that's roughly what I figured, I used to do somewhat actively a backup utility tool, based on rsync, but for a while, it also supported rdiff-backup, so I got familiar with that notion, and the problems inherent in it. Apple uses something fairly similar that in their time machine software, which I believe is just taking delta diffs of data, something like that, which then creates all kinds of issues since the data grows endlessly until it fills its container, after which OSX starts complaining endlessly about full storage of the backup device, lol. Of course, that's apple, and their software is just not very good, but it is a real problem with that approach.
Your list confirms what I was saying, btrfs is shipping now as if it were production ready. Re synology, if I were to guess, I'd guess they have the features that btrfs supports very stripped down to the most robust, with a sort of protective envelope around it. This is however a choice probably dictated by the zfs license issue more than the robustness or readiness of btrfs, as a commercial entity, synology simply can't ship zfs legally I believe, similar to how you can't legally ship nvidia non free drivers with the kernel, it has to be implemented by a user initiated action of installing the driver onto the kernel, similar to how zfs when shipped with linux works. The license issues only kick in once distribution of the software happens, end users are totally free to do whatever they want. The NAS OSes I was testing recently worked that way, they supported zfs but you had to install it yourself using 3rd party sources.
I think the one example you listed there that's interesting in terms of how to do btrfs in a stable way is synology, I would actually be interested to see what they do with that, that would probably reveal fairly accurately where the actually stable features of btrfs end, and where the unstable parts begin.
However, you can be certain that choice was not made based on reliability, it was made because zfs isn't ship-able commercially with the linux kernel, that's all. One of the more successful 'free' software license poison pills ever done I'd say, but insiders state categorically, the ones who were in the room when the choice was made on license, that the license choice was made explicitly to block use in linux kernel.
So probably over time, and enough users, and enough users patient enough to explore how to make the filesystem fail, you'll get a reasonably good result. That's what happened to systemd too, it took about 10 years to get it stable enough to where it usually works, these things aren't fast. Wayland still isn't even remotely there as a mass market option either, and that's been over 10 years now. Improving, but not there yet.
I am fairly certain it is just btrfs on top of lvm.
I'm not up on the details, but that seems redundant. If memory serves, the guy I knew whose friend made a good living salvaging data from broken systems was saving data from failed LVM setups, I thought btrfs already, like zfs, spans drives etc. I know in my testing systems, I set up all those layers, but you'd never actually want to do that in the real world, the fewer you have, the less prone to failure it will be. But I stopped my btrfs research and testing after I concluded I was not going to try to add full support for it until the btrfs tools improved, significantly.
What's odd is that OpenBSD created a quite good, robust, advanced software raid with native encryption file system, softraid, and did it quite quickly, that was a very impressive system by the way, and was probably one of the reasons I dropped btrfs efforts, I had been working with LVM, which is pretty good re tools etc, mdraid, again, pretty good, ZFS, very good, softraid, very good, so I had something to really compare against when I looked at btrfs, which was the last in the sequence I looked at.
Usually from my perspective, when I hit stuff that is clearly not ready for prime time, like wayland and wayland compositors (finally support added in 3.3.13 inxi, and extended in 3.3.14 and 15), mir (died before I needed to deal with it), I tend to wait and see if it improves or not. That's where I personally am putting btrfs full support in my internal priority list, probably will happen in the future sometime if some things change and improve, otherwise not worth the effort. I guess after doing this stuff, I get a certain perspective. I waited about 5 years to add bluetooth support for inxi, for example, due to lack of decent tools, and deprecated primary tools, but in the end, added it, though it's still not perfect, but it's not bad.
But I would hope synology is not making the mistake of putting two software based block device system stacked ontop of each other, after my tests, that's something I'd never do, all that happens is that the failure risk increases I think by much more than a doubling, since you add in the risk of failures from unanticipated negative interactions between the two types of logical volume handling. All this to me highlights why ZFS should have been the single solution, since it always was built to do this stuff natively, and has only seen steady improvements and developments since it was released, and in particular, now that it's largely being done as OpenZFS, which kind of freed zfs from its BSD roots, to the degree that now I think FreeBSD ships the linux focused OpenZFS.
I read up a bit more on synology os, and would personally just avoid it, why use something that has no real documentation and where you are dependent on a corporation for everything, seems to be against the entire point of using free software and operating systems in the first place, just make a real server with a real distro, and call it good.
Core stuff just is hard, that's all, essentially trying to start from scratch, and then having to redo all the fixes for corner cases the previous generation of things handled, to me, if the core stuff doesn't come with excellent debuggers and developers, you will see huge delays before you hit stable and reliable, I saw that with systemd, and wayland in particular, which was designed as a protocol without any built in debuggers that are always there and reliable, a very bad idea for a new protocol, but that's how it goes. I view btrfs as quite similar to those two projects, will probably roughly work and be reliable one day for all use cases that matter, and will probably take a long time to get to that point. Won't ever happen if people don't use it in the wilds however, so it's good that more cutting edge distros like garuda promote it, that type of thing is helping wayland too today, which is why I finally added real wayland support to inxi.
Note wayland stuff doesn't really work in kde and gnome because I was unable to discover any cli tools to give wayland specific info, and there's no global tools yet for wayland, though they are being considered and maybe will appear in a year or two. Nor can inxi determine it's wayland if it's a root or sudo start, I still have not found a solution to that issue, but that's one I expect can be solved, though I'm not going to stress about it. btrfs can also resolve some of its issues by fixing its tools, and getting rid of the need for root to get simple data from the file system, zfs does that always, there's obviously no real reason for that to be not possible barring possible core design errors or something.
1 Like
Interesting thought. I will wait more time to let btrfs mature.
Me going back to ext4 under luks ... For a while ...
Thank you for comments & thoughts about a filesystem not so production ready.
BT 
The abundance of features in Btrfs that repel you so much are exactly the same features that make it an attractive and interesting file system to others. As for the complexity, that is the nature of any COW file system. To give ZFS a pass but say Btrfs is too complex is unfair.
One man’s meat is another man’s poison.
While I understand the appeal of running Debian, the most stable possible distro, on ext4, the most stable possible file system, it’s an awfully good thing that not everyone does–innovation in Linux would die on the vine. Not for nothing, but Debian isn’t exactly known for pioneering the next big idea.
If we all dialed our machines back to the most stable possible settings and never dared to run software that might crash or break, there would be no KDE, no Pipewire, no Wayland, and absolutely there would be no Arch Linux.
6 Likes
That would be a misunderstanding of how debian works, or at least an exaggeration. Debian Testing or Sid in most cases are quite similar to where Arch is, except they have a significantly different focus and emphasis, the entire function of the sid-unstable and testing pools being to test and stabilize packages for next stable. This leads to a freeze a while before next stable is released in sid and testing, which is actually the main difference between Arch packages and Debian sid/testing packages, otherwise there's not a huge degree of difference in the actively rolling release pools.
There is however one really big difference, the number of packages in the primary repos, and no AUR, because almost all the packages are packaged in the main repos by default. AUR is something I'm very skeptical about as an idea, it strikes me as asking for trouble, like using a random third party repo and expecting it to not break, I think that's my biggest issue with how arch does stuff. I faintly remember AUR being real packages back in the day, but I could be wrong, but I think it was. That shift I don't think was a great idea, but otherwise I like what arch does as a rule.
I used to do some distros as a core team member that did sid, which I now view as a real mistake, and I was actually happy to see Arch based distros largely take over that testing role, which I came to view as inappropriate for sid/unstable branch, which has a slightly different function than Arch based primary package pools. Very similar, but slightly different.
Basically the distros I was involved in were doing something quite similar to what Arch does, before Arch had gotten stable enough to really rely on over time, but that changed, and I'm glad it did. But except for the freeze period when next stable is being prepared, debian's rolling release pools aren't that different than Arch's, the main difference is that every few years there's that roughly 6-8 month freeze period.
I think as a direct result of my heavy involvement in those distros, which was all about stabilizing an unstable pool, which worked quite well with the right team, I more or less totally lost interest in that game, I do actual work on my system, my clients depend on me, it can't fail, and it doesn't, but it's rolling release, I just don't upgrade it that often, every few months maybe, because generally something or other breaks on upgrades, unless it doesn't, but you never know.
But no, I'm not interested in getting excitement from my distro and desktop and work stations and laptops, lol, I want the excitement to come from what the machines do, not the operating systems or desktops. That's also why I don't use kde anymore, it lost its charm after the second major breakage series with kde 5.x.
There's a certain I think mythical nature to your views of debian, they all get their source code from the same places upstream unless they are making some of it themselves. I like how Arch evolved over a few years to really be the best choice for never freezing rolling release, I think that was a niche we tried to fill, but it was really trying to cram a square peg into a round hole in the end, debian testing is a very nice compromise to me, rolling release, but very stable. The only cutting edge stuff I run is Liquorix kernels (aka the zen patchset), because damentz asks me to test new stuff in them, which I do now and then, so my kernels tend to be fairly up to date as a rule.
There's still no fully functional wayland sad to say, there is a wayland that often works for some users, and fails for others, which is improving slowly, very slowly, but fairly surely. But it's better, particularly if you use something really focused and dedicated like Sway, which is what I'd use if I was going to use wayland. But Arch isn't nearly as different from sid as you think, it's just more consistently rolling is all, and is more focused on that being its primary task, which I think after too many years of experience, is the right way to split the distro market, I was glad when most people who wanted full rolling release went to arch based distros, I thought it was the right thing to do, still do.
My rough understanding of Arch pool is that it is about halfway between sid and testing, more stable than sid, but fresher than testing, which due to certain requirements, doesn't always get all the packages from sid, if they aren't intended to enter next stable, for example. Plus of course never freezing for a next stable, which is really the main thing that actually impacts realworld users.
I'm in particular extremely happy to let other people lose their data on raw file systems, lol, that's something I have no interest in, I used to a long time ago run reiser 4, which absolutely comical to use, it would fail constantly, that was 'exciting', but not very interesting in terms of having a working desktop, so I lost interest in that type of thing fairly rapidly. I would however say I have never seen anyone who seems to know what they are doing on a more pro level say they have issues with zfs, but I have seen it repeatedly with btrfs, which to me just says it needs more work, how much more, I can't say, it will be the point where it's 'boring' I guess, which is what I want my file systems to be, extremely boring. Too much valuable data on my systems to play with interesting file systems, lol.
1 Like
I am not sure whether Btrfs is “production ready” or not. That being said, it is being used by default in a number of places.
Synology NAS devices Suse Fedora Garuda
No wonder I thought Btrfs was so mainstream… Suse, Fedora, Garuda are all OS’s that I started my desktop Linux journey with after my experience with Unraid for my homelab, which also recommended Btrfs for my cache pools when I was setting it up.
1 Like
Chris Mason is the founding developer of btrfs, which he began working on in 2007 while working at Oracle. This leads many people to believe that btrfs is an Oracle project—it is not. The project belonged to Mason, not to his employer, and it remains a community project unencumbered by corporate ownership to this day. In 2009, btrfs 1.0 was accepted into the mainline Linux kernel 2.6.29.
Personally the cutting edge features are what makes Btfs for me. Do I use it on my workstations absolutely. I also keep backups; since you know any file system can fail and lose data. Would I use Btfs on a corporate critical server; no but then again I wouldn't use Arch on that either. Its all about using whats best for ones different scenarios. There is no one size fits all.
6 Likes
btrfs is fine for many people who use more than four years.
The Debian wiki page shows the configuration recommendations for btrfs against some problems.
- Do not use Software RAID 0,5 and 6 except RAID 1, just use regular backup (There are btrfs backup tools).
- Do not use
btrfs filesyystem defrag - Do not use quotas/qgroups
- Do not enable mount -o discard, autodefrag in fstab
- Periodic trim is not required on SSD.
- Never run btrfs defrag against a child-subvolume (eg: snapshots) except subvolumes.
6 Likes