[btrfs] Docker and Subvolumes

[BrutalBirdie]

I use docker quite a lot for my work and other projects.
I was very happy when docker got the support for cgroups.

Now I started learning more about btrfs and noticed something.

╭─eha at eha-desktop in ⌁
╰─λ sudo btrfs subvolume list /

ID 256 gen 77576 top level 5 path @
ID 257 gen 77576 top level 5 path @home
ID 258 gen 77549 top level 5 path @root
ID 259 gen 77508 top level 5 path @srv
ID 260 gen 77574 top level 5 path @cache
ID 261 gen 77576 top level 5 path @log
ID 262 gen 77574 top level 5 path @tmp
ID 372 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b8172a6a23ff814b732780361aaa47f341e6c15943d5a0decb778fceec961134
ID 373 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/21539b56522dbf67ba30b03e28319868aa4a40eed9c880d483a955d1d263e4c3
ID 374 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b62214dfe9352307138900207f29f3fff8d829e10d1e9892dbb3a000ebaaf1c6
ID 375 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/92b3e4e08cf2e1627cb3d9ce816270b23b5fdf88ed55d21b8c8865abeeca5078
ID 376 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cf2181eadc29cd03dfe8d8c31d4f56600155ded1160b50868f3c50420220b20b
ID 377 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e03330bf1c0719a7aa16e82e3da3b45190abdb9b2ecbbde6d2581b22a01b3b67
ID 378 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/5a1f65319a165519c8354c4d968469366fd7aeab2e9f1baf41b131306d36ad74
ID 379 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/652fe6d76ca4d7a492e9b36ea97f6ac1c5ad7d991b3ad398f756e37c2645b433
ID 380 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c22e5b61698a32845f6325d996922947d906886d8bf12ad8b37e49957f4a9f85
ID 381 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b68b9a84c08c27cbdf403b6b5c98855ac09b523cf3c14e107991131ed4fb3116
ID 382 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1bc3714d23e8815c02e0df6e03cf39b82e6917f4573f71dd28c0c408fe7552c9
ID 383 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cad456c85d7a721bc91ae553a47c57cebd822e4f1246105755a2d6f1ed4c6b43
ID 384 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e5510d4006c41988f5748d51b0350f0b866ee2c9b1085ef3f1159f9880b1414f
ID 399 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/27fc7d15623d59d6e04121caa30e51b7f1c5fb5226fa268fbb688e4419575eaf
ID 405 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f2d091b6506a8d61fc3dcce324ca87fc3d3027b9e86ea26a2a9f284ddc792c5e
ID 408 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e51a1d0068adb3cc57b3ed823c4ba961affb60d11b1f0bac142f793a1ddf372c
ID 411 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/2fe50d7acd3a576fd00f38e2bb72934f22eb40f1267fcff5d3c41cbe4c604dd4
ID 414 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/625ba318bb1a4a605230e0cb6216873f4e43bf7cefc8a627259a424752ac45d0
ID 416 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c0dc64b576960cabf94b3d017dd197bc337eb0ff67a61cfca70635f9498f91d4
ID 419 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c53d17555899b8868f0250093b2ad380db3010f1379f3927f4512f46e87c1dda
ID 421 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/3fc2195387463a2a5c4e39dedd8ae72bd7d706ca6462ee5ab466a4fe84157e9d
ID 501 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/a97bccc1b48b8b7057e1076798bed746d311f073e8194dd3f5e53a814c2a5bb4
ID 507 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/ff885029234b631ce9beca14a16759963ce896380596731e2e700393f16aade8
ID 510 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/5d875abe41f851b011c2241ab5b2ba04cd6a371eed67e1115da1c0f2c8365972
ID 512 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/6eeed55c5b498f18c5579de251100f1dde1e0000bbbe6d68f33ba9ca97e50293
ID 514 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/d3f48e1d28912de41a512b09e094d2004569111ac3724cf94528c87e43149917
ID 516 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/ed8663f7b30f7597289a7b7a043acdce4d7a79abf574f1fcaa4c795b0c7a0132
ID 519 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/58b7d0228e56f8c7501b06de4364bb89117341268f917d136851e30f280025eb
ID 578 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/52323b22ee8c0c909bea7bb392f049756d3a86645598eea86ba6ae3fb81abe98
ID 580 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/9d45932d89816a39af103a7bce06eb3137fe3fdfdd87cf379953dd0df19a7bdc
ID 582 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1d4e57bb2ad4490953e8ac4d64d168f27e9ac99ae22319e6617c1f599313b451
ID 584 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/3cd098760e4dfdb13f1a359081591bc545c1bca29214777cb4f11dbde2219ac2
ID 587 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/985861c213b9e0defad0043f19c9b23a95446855a74dca6c4012885b189df3eb
ID 592 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/7adc20ea5c066c31e43a2df25120c33189ed70be13edd5951fc799aa29bb1295
ID 594 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/816efe3db4affbeaca6dab39680b04a8782a99b56dc7701bc89fce11109625e0
ID 597 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c77ab952936516811982e3487a34cf5287d10725bdc7e57258cd9875cf356768
ID 599 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c24dc8068b2358c35fe942d8f1189c401c525f6fa64d9292aae549c81d322ae2
ID 601 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/62d73f9574f9411ecc3479a4017e59d377dea2cb13e53625033bd4613c757fb0
ID 604 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/7bdb3c1368f49fb4ed53f79142c831a6f5529a8fd4248f94a12418c3761f9761
ID 609 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f72413767c05b3ff12d18216ef98d626f7e87b9ca29f7994fa85afc0f5f994f5
ID 611 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/bede1b1c57c4f8ac01262a83f82dc923be4f6a18daea6bb32e2439c6bd37f7ac
ID 614 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1f91b189b382247cb1ecd05a917e6934d44e888b0c5518e862d247861f9c116d
ID 616 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/4f0cc47b555660fd2f511915c6a61fab779b907957d0dddd41e2a99a38d1f55a
ID 617 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/95920bd7593e96ebfede7ff64dde747b6d8ea8061321a37b5218a027aad38fe9
ID 618 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c5ca1d82c50dac33e65d4318539b21aa1d49c4cd055c19a62d16035c08f07887
ID 619 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/fe09303af58e12fe14d1878f0f766bb54a1fbdbf79e6a69e3bfb228d218a5bc3
ID 620 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/ea05bf24ca78c34696546b672118e7780440ae1f5b666fb580c65dbd2039732b
ID 621 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cd212ba040cc7d4c60ecf8216cffce17cf0300918a3bea7de9cce22a02216976
ID 622 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/8115af94301f81462f14d8580c0a82c77b2fccbe64181e9a8c0dcbb6e2ff4c69
ID 623 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/11823f81bb7d348940340ac8ac5680f35571732056b4cd84acc18704342910c9
ID 624 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/a1cede4a10673ffb32ac63778257580dfb6efc74d97e8ec4fa6d60b1a3e2b2e2
ID 625 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/dea9fa4d3b42ca9d67890746b06e4c004c9fa540de827e9227268ab31b9c5abe
ID 626 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/608f25215103e88188c8d3fffb6a5fe283867b24821d2d2b3bbe2db0435e9730
ID 627 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/be3cfbc16a96407442d4342c58c5fe2a8ffe4ffdea688837c75f273cd8582929
ID 628 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/8f43d755bfe6485250fdb9dba308b11ef912168802119d3cbc34ab5f7d553ae5
ID 629 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/4b37cae7ab0cc8a75c6570da1d2813602418b359c2e0806a3107e364a48d90c7
ID 636 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/da164612e6df7ad0a8e433925e6b66987f7b4cbae814ec2196abf3858ed23401-init
ID 638 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/12984d3c75fe310f7b6647cc01fdcefd5b5ad55c439863449114ff41163a07e6-init
ID 642 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/6e9aebca3d94b171608c375390c00f0d2d3b3bdb92b842c52d4015506e4628e4
ID 677 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/d710da053cd0afb5ba036b6e338c3bbe6f314b20c35fb2cbef09b2627a4e0936
ID 702 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/75f4f0d441df218e827228fb8240b308d2d7cc9bfc4e76aab411acde7dc9aaca
ID 704 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/089d311937ce0b4ae79b453bc3ad33e17b735c683a24e2416b3f7588981acf90
ID 707 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/bdcc37f365757f4e745cfa71a7bd96bda7737042cfc708e82518605150378efb
ID 714 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/0f030f1f82e9af85b60bee6abd9fd7a91dfc7217eb07fb20f0e23b698170a549
ID 716 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/69bc13319282714fdfe4d8d89412eb821d2706a2ea03ca0ff6fcaf7eb65a597e
ID 723 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f389d7b65e23350dd78db04db7a1d97c6545ba7e97551d5d21e3cd1a7bd574a5
ID 728 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/71931762d5ae9e5d4acfdbcd41dce368020080e1fc50c4464991de04c1599499
ID 731 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e965d25857a2686531d7aed564bd9ba56c7e359ebc3848451f87eb4ada2a4667
ID 734 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/047bd2383854892ac6b3d68e551e01982cb55711fa65314457183967b8ee64e4
ID 736 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/3a59d913e23cd54a5f7800c1adb56511a785b336e4ff52c2c3d5725a253feee4
ID 739 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/32222c311b098c107914499f93356df5f26de9be9e11763c12ab0b4278c8b8a9
ID 744 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/9553f199986b0882d54cc259267932c6077bbf3285380d183620af354948c663
ID 746 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/ea0321a50d6292d94e41fa8dc1baaa8413079bd2afd68345e59aadc66f68e3e1
ID 749 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1fa2d7df6916332ba6ba588a9c64de80c7b08e3c43a91d59b2373409f5167b95
ID 754 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/2b13d1609eac07f0dfd70ae2ce7c3418bee4655c2114310b0ae97b494a12df0c
ID 768 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/85611102c273e8d32a340ac16a5e35817a34a23a2525b46c53f40ab26eea7a23-init
ID 772 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/0489c9d16f0f65b76523c692bfb7b3ae5df075600ad04ed217669f19dc58cf2f-init
ID 774 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/769fcf4ba0d0bb56b5766d485d916d4d513f54310d13bdc058b68d4c2067999c-init
ID 780 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/a32668d9aceb6901b8b25337a9e35374ba3becd0016c2c13e51b906bff9b7632-init
ID 784 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/67368fe6dace2ba69df91d220393b6a902194a2b7864f9056ea21e4e07911660
ID 785 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/91403a76c494f07ef1fdd3a1dbaf7df2d4e69937ac3e0081d30868189000ef60-init
ID 791 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e4b1aff4c3857a376fd84d6a233d95168bc197787ee543e21686aedc1f8eb897
ID 792 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/8c24fecebd91fcd8b618247bc1736c23be160142ce4ebb5712a9a7352afebf98-init
ID 796 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1308e3d7bdb5b1f677f3f6eda67ad8a704f8762c4493b9f7f6f9bd79e2789143-init
ID 800 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/dc38bbcd9d8175e01124d2a6e1f96ab40a1ebcb807c9d69db076a893a1d8dbab
ID 803 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f65c1c776fef47d882ea652f54cb55a80f30279c1b163cf84ebe3b6fe30c23e3
ID 805 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/674de21aa0958e7607ff883cd23b08b62bdfc8b6d133d2ccd4462d8b3519e801
ID 809 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/90b2476a75bb1f078b54970128690f39b5dad53eac09b6c123c937dbc14d7a1b-init
ID 813 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/08759d405c1cd1cf93fab22929566e833960ff0f8563b2fc29f1b9a5c72fb82d-init
ID 815 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/5511185d4d14151c43f24df854080ddc73f3a7f825f1a23a98325eb0c9189043-init
ID 818 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/240436f58142f44e6db21021cd81d7a1e103c9c3858ce159b3917dfda30d05ee
ID 822 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/57a734220bceb9b31e8d08c0113cb6c21372c29efbe6d344fa80b075886e5035-init
ID 825 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/625b11b50cad02d48ca051ed303c0e58d7426582d6da17996e78035199c68a08
ID 829 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/045122535b3c1f0dd4b19c1b8b624fb51d2ec72ed9905e8190f01ec93c3ed819-init
ID 835 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/020eeda372efc5b3f10eb2e623717cefc51fde9a7d835c394e0ecce2bd139501
ID 837 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b51a9ec4f1a220573150d862bf036978cf96dcf524e12d932cf3fe4594c895d1
ID 841 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/8a5bdcf95032e2dd4cbcf055c6f133c14bee260ce1d82a4c26528d22cea233ef-init
ID 844 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/8559328d21f71604bfd84ffa268ae55078f0979b1103bca8fb1753091180eed0-init
ID 848 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/d7fa80cac5a35ba7ca2b2895bc4ff8991c8e4273c9d5769d1a548ef02ae59fd8
ID 850 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b56ed9d40d0672155188544f8e1a71603d237292f528cefc07129f232b4e60cc
ID 854 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/4ed4a95d5b4cb1ebf1f95d77517c300f0248db4b15e2903c3d135571c2592c9a-init
ID 858 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/de614ac127ed99b9978a363450e7ab4e1d859e9d7ec48fd3b5e0d7cd5b678aed
ID 860 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/bcdb74f7e1b3be08688438230051d872ad5fd345cca27d6cb824fba852882d5f
ID 864 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/00390c2f032d04686dc8f2aedff575e17f5d49516dfd71823be674ee21222c94-init
ID 866 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/9e2bf37c1d9e64e9d6aa4e7e68f65e7f5743950fd5b18c6273ef5816f92dd3a1-init
ID 869 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1ee9e5a075eb02e781c83c11b9c0aef7a9404ea60b5cb3157c9950fb855533bb
ID 873 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f39d0904574f194d26e02558de454192d562a9f05f2a84136a218cfc31511cf7-init
ID 876 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/87066d5a51409444c270d6c909af7a362338c578039bc98092f2abacfda54676
ID 880 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b5b70b8d11e47acd101c9820e86d51d1b7992f584095bd4db5f6d59108616844-init
ID 884 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1d300600ce60fd33025903162a577f42252774b1857350ec0a11b81ea68e44ec
ID 886 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f419a6d2a0777ac0e085ac93654b5b8941ca33fad26bec53d88ea915a7b27f7b
ID 890 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/295f9c2062289c3bc519bd705ce34b6f9986f6f8d25d6acc675aff6c36aab5af-init
ID 893 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/6709e8218779e7aa266254134e2abb404cd5f18689f0730edede3eee84711bb2
ID 897 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/6f87fa635affc8664830a8174b6c323a54c3e3ba2844ff17e81bb06a7247a794-init
ID 899 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/bc20e1c993c08358bb37a21ebe16d31494ffc9c280048e5af4cb58d8faa2d56b-init
ID 901 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/98ba0808807dca0a28d9adbb4db3462529914775ae61392195215c7d8c90d334-init
ID 903 gen 77543 top level 256 path var/lib/docker/btrfs/subvolumes/ffb9ee30d2c6e668ee45d58d44a38dbd712fc14d07cc6049efbf015ee2a2053d-init
ID 905 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/7b0febffff980fb41e29a281b88c4640afc6f94bd57ff7149b6c2368aa777af2-init
ID 913 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/deceb9f3e5d756ef19f12dbd1dd8188c6d21bf682225009fa2f379ee0b22e23d
ID 915 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e1a86dff117de6e86f07ead320d714e374459841eca23a6f2c6ef9ec6f9d00cc
ID 919 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/0d7a5c67a90e8f52f13cfb2251bbd9ba5e1eee93573d39f0af9ecb3a3a31ff44-init
ID 921 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/61d7aec07008a36d219d6524576c8cc907e01c8f011566734faa5b15a8f84632-init
ID 924 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/59fd14a19b8ed76a1dec2199b3c589bbd5d74935f29a991c347e6eb22657acc3
ID 928 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/62e177f8411651981ce630bd7d40006b238a1f55ad2d6f1c08221ca5243f2106-init
ID 930 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/a606217506d50ec8596dc0d31e7865247a33bad8909dd223d4a43c9a884688a4-init
ID 934 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/b563534eed3a6407db58e4fe9636c6c57733ffbb901cd0f5e02ba52cd6f816bb
ID 936 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/115ed740ca5083349c1e061d7a3de38f5266036962c879e241b7c2fa1087dced
ID 940 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/11acaa402118b0bfcfd0ca4009fa04cbe7af21d1de45defc5c9c9f012d0d579c-init
ID 944 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/5f88f9c71cf32a9c83452e947a8c7cbb0532edc84bfbbb136e74c2f333201977
ID 946 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/2b31fcbdb965d00a5d00f9ceba259e442c09e06ce3bbcee878d2fbcfe39b6975
ID 1027 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e0ae6a95703c61cbe40c38987fd1cc1b6e1a86a669a8612f8a85424772f83f9d
ID 1070 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/3079410ed22f4db2bc72d0849a53a2795d28c0884633df799678060bc91198b2
ID 1077 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/39dfc7404a07c6aef4e4e5a3bceb2a3f2cec28e959c56ad91523b5926af58f5a
ID 1102 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f4974a53aef5935d499d9f34615cb1edccdce58d278d3fc79569e27d1a404ea7
ID 1104 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/57ea8acae3a062d8da520fd26cc64f9ac07bd2f9ecdb09ba0836488bce082828
ID 1106 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/a933bca655fa5fd5d5a27ad6c8ea9c47f227b9b1befd0d514daccdc9e905e824
ID 1108 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cd475af38c23c2fdd572e42b6b2331ee0349c1e8a1775387b4e134589f0deb4b
ID 1111 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/0ceba61c0db0d3dd55cee839ed578bc216eb499dc946c083dec6ea7ee274e8af
ID 1114 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/64f1b81840ed107f52a22e12bf34a8905e7fb2a97a51d38780b5cc864ec1a1a8
ID 1170 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/5f27c16a515072571b41d63269cd330d0a3da4d9b95a6f81fd8119c7cfe06f51
ID 1172 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/9998cafe236f69ac6ff7eb05aaf2a8de76077ce239d8ccc35167a44a4f40731e
ID 1263 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/9139de4cfe9807b8605ea49f263fdacfd610281a480dd7830e524dee64da1652
ID 1265 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/35dd76b4e6b802a6e884ef852f21259664ebf6ae4910df2a3970ecfebe355251
ID 1268 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/eaa4763c291fd8945ed9d282dcbad7c798cc6bd72ab26f7036e768806a8450db
ID 1271 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/1145d39f3c96d19821cf398b55d8df5469627bde81623b2a9cc0492c146722aa
ID 1278 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/0d5cc15adfbbf4352a96c53dbdf70d52b188bcc9bdbd4b3c9f228277b5e6777c
ID 1280 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/c0dd2291176f6b6c54e46cdb497bee88d1a500922ea28d733aa52ecf35afb89d
ID 1282 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/a8e1bda0ccf7b4dc2c90b0cf38ec46c692760dba00248b975f89314b0bb7c5cb
ID 1285 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/4151b0ea202bf0d4fbaa26f2c4096acdb2c6c9772a1e93d0afd0e79343dcf5d1
ID 1288 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f8c0129fd600bcb53ca5ddc32a7eafe63660925d006d7e9de101869be8dbab0a
ID 1297 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/47a6f48655bc3bd452b50ed0151a066edcf3644b7acc7d21d011d40fdbb3bf58
ID 1299 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cb6c62d362c0dc24434a12fb0fd028e58dc2a14156c534d41ef63586b9318e1c
ID 1301 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/75079e4afac231cb2f35fadfffc1b2fb5af193bca283a46caed7c31b0dd7c704
ID 1304 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/5bf54beff970eeb2181f7e9ebf9c91b8cc443d08beb021493b1f0ec6a917dc67
ID 1307 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/4de003b5c96f4f40d4981af2dcf8de799777361af4f88eda9ce14dd11288ca2e
ID 1311 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/da61cc7719632034db83d37ec8fc2500e6fabfcfb75edbf43fec4f55d78be1f0-init
ID 1471 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cca5331a5e625965b4deecff14e43660ce3ad6de936a7d24efde7d3a23210aa1
ID 1473 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/d8eb9a13d3cc675ed4315cc05f0e1df7e4af230617a38d29294a3812cc009680
ID 1490 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/ccda6a980587cb53ef53be0fa2c0b331b5cc99d54fb9d1a145168816c8da47b8
ID 1493 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/14b980e8bc224e5c0716e456e4933648bc2d7fd90976b3c70a9156d9488481e3
ID 1496 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/4b9d4f06f52aefd4f43c46233805947d33bdb81244c2fc4271e9b0acf18723ac
ID 1511 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f9abd00f867dc272b94d73b949d668385dc40c0cc7bc0c9b492d6486667a8308
ID 1513 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/cd08312d1306a7428eef0ed83718d716dfaa220c4280ad977e67bcbd82305e04
ID 1516 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/e316aa9e6ff6f4bd5cf5c35f27c04fbd0614603bc8db3b8d311f529e63fdd259
ID 1521 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/2484e9a164cb8ea71b60b4ae44ceee47e1f0ae356a94f4ad93579d88e11c09fa
ID 1524 gen 77537 top level 256 path var/lib/docker/btrfs/subvolumes/f29c1d4201f431bd4d3fa9e56265ffac3525b18b4f547cf0c0871a1796fa68ca
ID 1529 gen 77543 top level 5 path timeshift-btrfs/snapshots/2021-02-19_16-06-14/@
ID 1530 gen 77543 top level 5 path timeshift-btrfs/snapshots/2021-02-19_16-20-17/@
ID 1531 gen 77543 top level 5 path timeshift-btrfs/snapshots/2021-02-19_16-39-01/@
ID 1532 gen 77543 top level 5 path timeshift-btrfs/snapshots/2021-02-20_01-21-06/@
ID 1533 gen 77543 top level 5 path timeshift-btrfs/snapshots/2021-02-20_01-22-59/@
ID 1686 gen 77558 top level 256 path data/eha_data

uhm...

Lets check some facts about my local docker

╭─eha at eha-desktop in ⌁
╰─λ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

╭─eha at eha-desktop in ⌁
╰─λ docker volume list
DRIVER    VOLUME NAME

I try to keep my docker pretty clean, just the images I build and no stale stuff.

But from the above output of sudo btrfs subvolume list / it looks like the btrfs subvolumes of the docker volumes got never deleted?
I did not create any btrfs subvolume for docker.
I did not make any fancy docker config.

I am running Docker version 20.10.3, build 48d30b5b32 as plain as it is from pacman.

I also started searching about this and found this:

Docker gradually exhausts disk space on BTRFS

and

docker does not remove btrfs subvolumes when destroying container

moby/moby/blob/620339f166984540f15aadef2348646eee9a5b42/contrib/nuke-graph-directory.sh#L52-L62

This look promising.

Is this a known thing? Could this also cause performance issues?

[BrutalBirdie]

Oh no..

[root@eha-desktop subvolumes]# ls -lah /var/lib/docker/btrfs/subvolumes/f29c1d4201f431bd4d3fa9e56265ffac3525b18b4f547cf0c0871a1796fa68ca/app/code/
total 12K
drwxr-xr-x 1 root root  52 13. Feb 16:06 .
drwxr-xr-x 1 root root   8 11. Feb 16:49 ..
-rw-r--r-- 1 root root  90 13. Feb 15:03 .env
-rwxr-xr-x 1 root root 682 13. Feb 16:27 start.sh
-rwxr-xr-x 1 root root 834 13. Feb 16:05 vhserver.sh
drwxr-xr-x 1 root root  36 11. Feb 16:50 web

The data of the docker volumes got never deleted..

To be sure:

╭─eha at eha-desktop in ⌁
╰─λ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

[root@eha-desktop docker]# du -h -d 1 btrfs/
409G	btrfs/subvolumes
409G	btrfs/

Holy...

Yep

This could be a reason why I have some performance problems since I did some development with docker.
I just assume this could be part of the problem since my system freezes happend pretty much after I started my docker work again.

---

In true docker fashion

docker-compose up -d - Docker Scream

Need to edit this video to also include diskspace

[tbg]

Use disk imaging, wipe disk clean, restore image and perform updates and reimage once a month.

[BrutalBirdie]

Sorry do what?
I will do some research about your suggested solution since I have no clue what I am supposed to do.

[tbg]

Disk imaging is full disk backup.

[BrutalBirdie]

Like clonzilla?
This sounds like awfully lot of work.
I think it's save to assume I am not the only person using docker with Garuda and vice versa not the only one with this behavior.

Also how is a full Disk Backup and restore supposed to help with the undeleted btrfs subvolumes?

[BrutalBirdie]

Okay so here is what I did now:

sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/*

Delete all subvolumes in /var/lib/docker/btrfs/subvolumes/*

Check subvolumes:

sudo btrfs subvolume list /
ID 256 gen 78735 top level 5 path @
ID 257 gen 78735 top level 5 path @home
ID 258 gen 78718 top level 5 path @root
ID 259 gen 77508 top level 5 path @srv
ID 260 gen 78726 top level 5 path @cache
ID 261 gen 78735 top level 5 path @log
ID 262 gen 78714 top level 5 path @tmp
ID 1530 gen 78423 top level 5 path timeshift-btrfs/snapshots/2021-02-19_16-20-17/@
ID 1531 gen 78423 top level 5 path timeshift-btrfs/snapshots/2021-02-19_16-39-01/@
ID 1532 gen 78423 top level 5 path timeshift-btrfs/snapshots/2021-02-20_01-21-06/@
ID 1533 gen 78423 top level 5 path timeshift-btrfs/snapshots/2021-02-20_01-22-59/@
ID 1767 gen 78423 top level 5 path timeshift-btrfs/snapshots/2021-02-21_01-32-54/@

Looks better, feels better.

Lets run a scrub

sudo btrfs scrub status /

If you want to see what happens you can do

# -d is for --differences to show which text changed in the watched command
sudo watch -d btrfs scrub status /
UUID:             d3c75445-c596-456b-b7f9-a7bc48e9126b
Scrub started:    Sun Feb 21 13:07:23 2021
Status:           running
Duration:         0:00:05
Time left:        0:00:59
ETA:              Sun Feb 21 13:08:31 2021
Total to scrub:   135.83GiB
Bytes scrubbed:   10.52GiB  (7.74%)
Rate:             2.10GiB/s
Error summary:    no errors found

[tbg]

@BrutalBirdie

Like clonzilla?
This sounds like awfully lot of work.
I think it’s save to assume I am not the only person using docker with Garuda and vice versa not the only one with this behavior.

Also how is a full Disk Backup and restore supposed to help with the undeleted btrfs subvolumes?

Yes, like clonezilla, but there is simpler GUI programs available with zero learning curve.

It helps because you make the base image without the docker volumes contained in the images. When you wipe and restore you are then starting with a clean slate and the latent docker images are not taking up room on your btrfs partition.

Yes it is extra work, but all methods of backup are extra work. Timeshift and btrfs snapshots are not a valid backup solution. It also only backs up your system partition by default (not your personal data). Timeshift snapshots are only an emergency system restore feature in case you run into a bad update and need to restore your system. It is by no means even remotely close to a proper backup regimen, as your personal data is not protected.

Any backup stored on the same drive it is duplicating is simply a disaster waiting to happen if a hardware failure occurs. To be considered a proper backup, the backed up data must be stored on a separate drive (preferably external). Many would argue this is not even a secure backup without a separate redundant backup stored on another computer, offsite, or in the cloud.

[BrutalBirdie]

But I never asked about anything related to backup in my initial question ???

I am really confused, what backups have to do with docker volumes being btrfs subvolumes which do not get deleted by a docker system prune aka. orphaned data which will never be accessed again by either the user or docker it self.
In my case 409 GB of orphaned data which will fill up the drive fast if not recognized.

Why suggest a full disk backup and restore for a problem where I want to get rid of data?
This makes absolutely no sense to me.
I believe there might be a misunderstanding here..

[tbg]

Have you ever watched a science fiction movie related to time travel. You go back to before the event occurred and erase the event from existence.

That's the concept here. Your restored image does not contain the space wasting images taking up space on your system (in the future). When you restore the image from the past it is pristine and untainted by future system problems you wish to erase.

I really can't explain this in any more layman's terms. I feel I've exhausted my input on this topic. I thought I'd make a simple suggestion and it's turned into a position paper on theoretical time travel.

Good luck with your issue.

[jonathon]

@BrutalBirdie

docker volumes being btrfs subvolumes which do not get deleted by a docker system prune

This sounds like a Docker issue?

[BrutalBirdie]

This is indeed a docker problem, and that since 2016! (or even longer)
I can't believe they would add btrfs support without "garbage collection".
I wonder if podman has the same issues..

This issue explains very good what can happen with this behavior.

Possible disk space exhausted when running Docker on BTRFS by read-only traversing into /var/lib/docker/btrfs

[BrutalBirdie]

After some troubleshooting and ranting in this topic (sorry for that):

Temporary system hangs/freezes when updating

hello alll. I have 4 core intel N4200 cpu, 512GB ssd and 4GB of ram. system is unresponsive when I am running updates. Cursor moves but let's say the website ain't scrolling/ keyboard typing Ram is not full. I have tried muqss and bmq kernels, but not much difference. It can be related to the disk activity. which kernel can help. I am not taxing it too much. For me 4GB of ram IS enough. was enough in Manjaro for years and with core 2 duo. Please, any ideas guys?

I have decided to reinstall Garuda since I believe docker played a big role in why my system got the problems described in the topic linked above.
(ps: I can believe what ever I want, only time time and knowledge can prove me wrong!)

But I need docker for my work! What to do?
Simple… disable the btrfs support for docker.

For example I just build my app:

╭─eha@eha in repo: valheim-gameserver-app on  master [!] took 40ms
╰─λ sudo docker images
REPOSITORY                           TAG       IMAGE ID       CREATED              SIZE
dr.cloudron.dev/valheim.server.app   0.0.13    7e703242b8ed   About a minute ago   2.24GB
cloudron/base                        <none>    75ad8e394e18   3 weeks ago          2.18GB

and

╭─eha@eha in ~ took 9ms
╰─λ sudo btrfs subvolume list /
ID 256 gen 858 top level 5 path @
ID 257 gen 858 top level 5 path @home
ID 258 gen 294 top level 5 path @root
ID 259 gen 57 top level 5 path @srv
ID 260 gen 316 top level 5 path @cache
ID 261 gen 858 top level 5 path @log
ID 262 gen 247 top level 5 path @tmp
ID 282 gen 70 top level 5 path timeshift-btrfs/snapshots/2021-03-06_00-37-17/@

No crazy subvolumes like I had before.
How did I do this?

First read about docker storage driver to see which storage driver is the correct for you.
I’ve chosen overlay2 since it’s the default and recommended one.

Docker storage drivers

Edit /usr/lib/systemd/system/docker.service
Add the --storage-driver overlay2

ExecStart=/usr/bin/dockerd --storage-driver overlay2 -H fd://

Or the better way:

@jonathon

The other way (that won’t be overwritten on a package update) would be to edit /etc/docker/daemon.json:

{
  "storage-driver": "overlay2"
}

[petsam]

@BrutalBirdie

How did I do this?

First read about this:

It would help us focus correctly and to better understand what you are suggesting as a solution to which of the problems, if you explained in more detail.
Thanks for your work!

[jonathon]

@BrutalBirdie

Edit /usr/lib/systemd/system/docker.service

The other way (that won’t be overwritten on a package update) would be to edit /etc/docker/daemon.json:

{
  "storage-driver": "overlay2"
}

Use the OverlayFS storage driver

[petsam]

@jonathon

The other way (that won’t be overwritten on a package update)

Also with a drop-in, but my question is how do you decide about your configuration? What info is checked from current config and what is considered a better option and why?

There is a btrfs storage driver in docker docs, which requires good setup.

[jonathon]

@petsam

how do you decide about your configuration?

It’s all in the linked Docker documentation.

[petsam]

@jonathon

It’s all in the linked Docker documentation.

I have read enough of that, as you have assumed.
I was referring to OP and how he decided the suggested solution, which is implied as a solution to other readers. I disagree with the copy-paste suggestions, without any explanation, as I guess most (if not all) Arch users do .

[BrutalBirdie]

@petsam
I have decided by reading the documentation
Here are some points which made me chose overlay2 as storage driver.

Note : The btrfs storage driver is only supported on Docker Engine - Community on Ubuntu or Debian.

So Arch is kinda excluded from the official docker support but still the default for btrfs systems.
aka. Hey we don’t support this but use it as default anyway, but don’t complain if there are issues, we told you its not supported.

Also this section of the doc is very interesting to read.

Btrfs and Docker performance

There are several factors that influence Docker’s performance under the btrfs storage driver.

Note: Many of these factors are mitigated by using Docker volumes for write-heavy workloads, rather than relying on storing data in the container’s writable layer. However, in the case of Btrfs, Docker volumes still suffer from these draw-backs unless /var/lib/docker/volumes/ is not backed by Btrfs.

• Page caching. Btrfs does not support page cache sharing. This means that each process accessing the same file copies the file into the Docker hosts’s memory. As a result, the btrfs driver may not be the best choice high-density use cases such as PaaS.
• Small writes. Containers performing lots of small writes (this usage pattern matches what happens when you start and stop many containers in a short period of time, as well) can lead to poor use of Btrfs chunks. This can prematurely fill the Btrfs filesystem and lead to out-of-space conditions on your Docker host. Use btrfs filesys show to closely monitor the amount of free space on your Btrfs device.
• Sequential writes. Btrfs uses a journaling technique when writing to disk. This can impact the performance of sequential writes, reducing performance by up to 50%.
• Fragmentation. Fragmentation is a natural byproduct of copy-on-write filesystems like Btrfs. Many small random writes can compound this issue. Fragmentation can manifest as CPU spikes when using SSDs or head thrashing when using spinning disks. Either of these issues can harm performance.If your Linux kernel version is 3.9 or higher, you can enable the autodefrag feature when mounting a Btrfs volume. Test this feature on your own workloads before deploying it into production, as some tests have shown a negative impact on performance.
• SSD performance: Btrfs includes native optimizations for SSD media. To enable these features, mount the Btrfs filesystem with the -o ssd mount option. These optimizations include enhanced SSD write performance by avoiding optimization such as seek optimizations which do not apply to solid-state media.
• Balance Btrfs filesystems often: Use operating system utilities such as a cron job to balance the Btrfs filesystem regularly, during non-peak hours. This reclaims unallocated blocks and helps to prevent the filesystem from filling up unnecessarily. You cannot rebalance a totally full Btrfs filesystem unless you add additional physical block devices to the filesystem. See the BTRFS Wiki.
• Use fast storage: Solid-state drives (SSDs) provide faster reads and writes than spinning disks.
• Use volumes for write-heavy workloads: Volumes provide the best and most predictable performance for write-heavy workloads. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. Volumes have other benefits, such as allowing you to share data among containers and persisting even when no running container is using them.

All of this lead me to the conclusion that overlay2 is better in the garuda use-case.
Because if I understood the docker-doc correctly it’s recommended to have a separate btrfs drive for the /var/lib/docker folder.
I don’t have or want to use another disk for that.
And the described non cleanup of docker and btrfs subvolumes.

[btrfs] Docker and Subvolumes

Oh no.. [root@eha-desktop subvolumes]# ls -lah /var/lib/docker/btrfs/subvolumes/f29c1d4201f431bd4d3fa9e56265ffac3525b18b4f547cf0c0871a1796fa68ca/app/code/ total 12K drwxr-xr-x 1 root root 52 13. Feb 16:06 . drwxr-xr-x 1 root root 8 11. Feb 16:49 .. -rw-r--r-- 1 root root 90 13. Feb 15:03 .env -rwxr-xr-x 1 root root 682 13. Feb 16:27 start.sh -rwxr-xr-x 1 root root 834 13. Feb 16:05 vhserver.sh drwxr-xr-x 1 root root 36 11. Feb 16:50 web The data of the docker volumes got never deleted.. T…

[petsam]

@BrutalBirdie

it’s recommended to have a separate btrfs drive for the /var/lib/docker folder.

It looks like it, but I think it maybe a little general, talking about storage drivers. In case you don’t actually use btrfs on your system and you may decide to use btrfs for docker volumes etc. then you need a separate mount to use btrfs.

Maybe an alternative usage would be to exclude docker images folder from auto-snapshots, if there is such an option (timeshift, grub-btrfs). IDK really.

Anyway, good decision, just make it stick from updates as suggested from @jonathon .