Super Long Delay to Display GRUB - Garuda Gnome 230501 ISO

Issues & Assistance Unsupported Software (AUR & Other)
1

Hello all!

Some really weird happened when trying to install Garuda Gnome using the latest ISO as of now (garuda-gnome-linux-zen-230501.iso) and I would like to share the experience and ask if anyone knows what exactly happened and maybe how to fix it.

I have a small convertible (Atomx5, 4GB RAM,128GB eMMC) that had Win10 and some other distro installed, and using rEFInd as boot manager. Everything was fine, I booted the Garuda ISO a few times to test how it would run on the device, no problems. I was planning to replace the Linux distro with another one.

But then I tried to install an Ubuntu based distro, and it was not able to finish the installation when it tried to install the bootloader. I then decided to try installing Garuda Gnome (garuda-gnome-linux-zen-230501.iso) but it would just hang after choosing the USB device to boot.

I thought maybe the ISO was corrupted, checked the SHA256 sum, burned the ISO again, same thing. Tried to boot from Ventoy, same problem. All other distros, including Ventoy, display the GRUB right away, just not Garuda. I even re-formatted the partitions meant for Linux, removed all other EFI boot entries besides Win10, same problem... I also went to the BIOS, loaded the default settings, no deal.

Then I gave up and let the computer showing the black screen and went to search for info online, and after some minutes the Garuda GRUB menu was displayed! So something is making GRUB hang for a very long time, and only on this computer (I tested the same USB stick with another computer and it was fine).

Everything else in the computer is fine, I installed Ubuntu fine, Windows boots fine, no errors, it is just this version of GRUB in this image that has this issue. If I wait long enough I can then boot the Live ISO fine too.

Anyone else experienced something similar? Any ideas how to troubleshoot/fix this?

Regards

2

You are exactly describing the symptoms of eMMC failure. As eMMC boot media dies, sometimes it will boot normally, sometimes the boot will fail, and sometimes it will boot very slowly.

eMMC chips are not rated for many read/write cycles, and on low-end hardware often they will start to die before even the writes they are spec’d for. Unfortunately they are often soldered to the motherboard and replacing them is difficult or impossible.

If you can get an SSD in that thing you can probably get some more life out of it. Otherwise, I hate to say so but it won’t be long till it’s trash.

6 Likes
3

Thank you for the reply. Actually the disk is fine, as I said previously, booting any installed OS works fine, no other issues regarding the internal eMMC in my tests.

I did a deep dive in the GRUB world and I have found the cause for the long delay:

  1. grub.cfg in the Garuda ISO does a search for /etc/issue to detect if there is a Linux installation;

  2. GRUB then hangs when the file cannot be find in the internal disk (mmcblk0) because eMMC has two block devices (disks) that can be used to store bootloaders, like Uboot, and when GRUB tries to search each one of them it hangs for several seconds.

So it is actually a GRUB bug, that happens when grub.cfg searches for a file that cannot be found in the internal disk and the device has eMMC. It is more evident in low power devices like this Atom convertible, of course.

By the other hand, actually the search (located at the bottom of grub.cfg) is somewhat useless, because the only place where is it used the logic is broken because it does detect if the variable passed is empty and the entry for UEFI does not work at all because it has syntax errors (maybe it was intentional to disable the entry since the logic is broken).

I looked at Garuda Gitlab and could not find where grub.cfg is loaded from, if someone can point out I may open an issue there to either fix the logic to make the search worth the trouble or just remove the unnecessary code.

Cheers!

4

I’m not so sure this theory checks out, or even makes sense. Still, if you would like to examine grub.cfg for yourself you can find it easily enough in /boot/grub.cfg.

Keep in mind, typically editing this file is not effective because it is automatically generated by grub-mkconfig using the settings described in /etc/default/grub, and the templates in the /etc/grub.d directory. The suggested way to amend grub.cfg is by making edits to those files, then regenerate the grub configuration file (sudo update-grub). Any direct edits you make to /grub.cfg will be automatically overwritten when grub-mkconfig runs again.

Garuda does not ship a custom Grub, it is stock from upstream. If you found a bug in Grub like you say, the appropriate place to announce your discovery would be on the Grub bug tracker: https://www.gnu.org/software/grub/grub-bugs.html. Like I said, I do not think that is a likely cause for the issue you have described, but I wish you good luck as you investigate. :male_detective:

2 Likes
5

Hmm, I meant the grub.cfg that is shipped in the Garuda ISOs... I will report the bug to GRUB itself, but both things are separated:

  1. GRUB causes the delay because of the (unused) eMMC partitions
  2. GRUB config in the Garuda ISO searches for a file that triggers the bug, but the search is unnecessary, and there are other problems in the same grub.cfg.
6

This file does not exist in the ISO. It doesn’t get generated until grub-mkconfig is run for the first time.

This still does not make sense to me. What are you basing this theory on?

Grub doesn’t go walking through /etc, or any other part of your system, looking for files during the boot process. It has a very small grouping of resources in the /boot directory and on the EFI partition that it uses for booting the system, and that’s it. Any other system files Grub interacts with at other times (i.e., when you are not actively booting the system) are not directly involved with the boot process at all.

Grub also does not care how many used or unused partitions you have. You can have a thousand partitions and Grub will not be affected in any way. Grub only looks where it has been explicitly instructed to in the grub.cfg, and only for the specific files that are explicitly referenced in the file.

Again, grub.cfg does not exist on the ISO, so it cannot be initiating lengthy file searches, triggering mysterious bugs, or otherwise creating problems of any kind.

3 Likes
7

You (and everyone else who "hearted" your previous comments) can check it for yourself:

sudo mount ~/Downloads/garuda-gnome-linux-zen-230501.iso /mnt
ls -al /mnt/boot/grub/grub.cfg
grep search /mnt/boot/grub/grub.cfg

I know it stopped when trying to access the hidden eMMC partitions because I have enabled debug in GRUB while booting, and it stopped exactly when it tried to access a sector in those paritions (hd2,0 and hd3,0 in this case).

Again, GRUB in the Gnome ISO searches for a file /etc/issue to detect if there is a Linux installation in any of the disks (what can be verified using the grep command above, for instance), to show an option in the GRUB menu to quit (and just boot normally), which code is also broken when booting on UEFI machines:

function grub_efi_installed {
    if is_efi; then
        menu "" {true}
        menu --class=garuda "Installed EFI-System on $installed" {exit}
    fi
}

There is no menu command in GRUB, the correct would be menuentry.

I do not understand the problem here, this should be something very simple to check... I even do not understand why the thread was moved from where it should be, because this is a problem about the installation ISO, where it was initially open.

Seriously... This not the first time I simply try to report a problem in Garuda and the reaction is the same, unfortunately. What a disappointment.

Regards

8

This is not the real grub.cfg, this is a temporary file to get to the live environment. The real grub.cfg, which is not created until the installation runs, is much longer and contains more specific information. You will notice once you are booted to the ISO, /boot/grub/grub.cfg no longer exists (it hasn’t been generated for the first time yet).

This is not something that happens during the boot process. Grub searches for other installations when it is generating the configuration file, not when the system is booting.

What you are looking at is a shell script, not a “command in GRUB”.

This is the only topic you have created, unless you have another account. :eyes:

Just for the sake of argument, let’s assume there is a bug with Grub that is causing your issue. I still don’t think this is the correct community for addressing that, because Garuda Linux doesn’t ship a custom version of Grub. It is stock from the Arch repos.

You keep saying there is a problem with the Garuda ISO that needs to be fixed, but I am struggling to see how that can be the case. If there were an issue with Grub, it would be up to the Grub developers to fix it, or if it was a problem with how it was packaged the Arch package maintainers would need to fix it.

I hope you do not take my reaction to your theories as hostility; I haven’t intended to attack you. I encourage you to keep searching for your solution. Perhaps someone more knowledgeable than me will chime in with additional information that will be more useful for you. Hopefully no hard feelings, good luck.

7 Likes
9

Well, I give up. I will simply list here some things and let the thread close (in the wrong place btw), so if someone else search for a similar problem the this person can find some answers:

  1. garuda-gnome-linux-zen-230501.iso (and probably other Garuda ISOs) is a hybrid image, this means it has a ISO9660 file system and a FAT partition that is used to boot when the ISO is "burned" to a USB stick;
  2. Not as a surprise, this FAT partition only has GRUB, than then looks for the ISO9660 file system to boot
  3. As most common Linux boots using GRUB, it looks for a file called grub.cfg to load its configuration, do other tasks and then display the normal boot menu with all the options, etc
  4. In the case of the Garuda Gnome ISO above, the file is /boot/grub/grub.cfg inside the ISO9660 file system. Yes, the ISO uses GRUB to boot (at least on UEFI machines) and, yes again, it loads a normal grub.cfg file when booting
  5. The grub.cfg in the Garuda Gnome ISO above has this as one of the last lines:
search --no-floppy --file /etc/issue --set installed

This is used to search if there is a installed Linux distribution in one of the disks (If a file /etc/issue exists in one of the disks, then most probably this disk has a Linux distribution installed on it)

  1. The command above then set the name of the disk (using GRUB codes, of course, as again it is run inside grub.cfg and not a regular shell script) that has the file, if any, in the variable "installed"

  2. This variable is then used when the subroutine below is called (again, this is inside grub,cfg, it is not a regular shell script):

function grub_efi_installed {
    if is_efi; then
        menu "" {true}
        menu --class=garuda "Installed EFI-System on $installed" {exit}
    fi
}

But this function is not actually displayed, it simply causes an error because there is no menu in GRUB commands, the correct command would be menuentry, so basically the use of the search command previously is mostly useless, since the only place it would be used actually just causes an error because the code is broken (bad syntax).

The delay I found in my system is indeed a bug in GRUB, and it was detect simply by enabling debug and following the steps. Regardless, if the line with search is removed from the grub.cfg used in the Garuda ISO or one if the internal disks indeed has a file /etc/issue then there is no delay.

So, in resume , there are two problems reported here:

  1. GRUB stops for a very long time when searching for files if the device has internal eMMC storage;

  2. grub.cfg is garuda-gnome-linux-zen-230501.iso has some broken code, and it has a useless search command that can cause long delays on some systems because of the GRUB problem above, so removing this useless call to search would prevent this unnecessay long delay.

Arigatou, sayonara

10

This topic was automatically closed after 14 days. New replies are no longer allowed.