Garuda Update change request

rinchen · April 24, 2023, 1:43am

Hi,

Garuda update's rate-mirror line is

rate-mirrors --allow-root --save=$MIRRORLIST_TEMP arch --max-delay=21600 > /dev/null

Would it be possible to add --concurrency=16 to that so it speeds things up?

anon72786180 · April 24, 2023, 1:53am

That would be nice. I second the motion.

SGS · April 24, 2023, 6:06am

Is it ok like that @TNE ?

rinchen · April 24, 2023, 1:23pm

Thanks for the PR. I only have a work gitlab account. Guess I need to create one for home so I can do this myself in the future. That way everyone can git blame me when stuff breaks.

rinchen · April 24, 2023, 2:27pm

The other tweak I made locally to improve performance was to increase parallel downloads in pacman.conf to 10.

westandskif · April 24, 2023, 3:09pm

hi @SGS ,
concurrency is a generic option, so it should go before arch subcommand as follows:
rate-mirrors --allow-root --save=$MIRRORLIST_TEMP --concurrency=16 arch --max-delay=21600 > /dev/null

e.g. rate-mirrors --help shows generic options, while rate-mirrors arch --help arch specific ones.

rinchen · April 24, 2023, 3:55pm

I suspect that concurrency and pacmac changes would be applicable to everyone except those on sub-200mb speeds. I wonder if it would make more sense to include two (or three) buttons on the app launcher. e.g.

Slow:
concurrency = 4
parallel = 2

Default:
concurrency = 8
parallel = 5

Fast:
concurrency = 16
parallel = 10

edu4rdshl · April 24, 2023, 11:19pm

Such things need to be tested first, or, at least, discussed/adapted. I'm pretty convinced that the bast majority of users use slow connections, like, 50mbps or less.

I do not see any reason to bump it to 16 when 8 is already a good number of concurrent connections and safe for everyone.

SGS · April 24, 2023, 11:30pm

That's why I didn't put it into production. (No bump pkgver)

rinchen · April 24, 2023, 11:31pm

Here is my data for what it is worth.

time rate-mirrors --allow-root --concurrency=8 --disable-comments arch --max-delay=21600
Executed in   35.79 secs

time rate-mirrors --allow-root --concurrency=16 --disable-comments arch --max-delay=21600
Executed in   22.03 secs

BluishHumility · April 25, 2023, 12:17am

This detail is very significant if changing the default is being considered.

In cases where the user’s connection is the bottleneck (not the mirrors), adding additional concurrencies may do more harm than good, because the results of the “dirty check” will be skewed. The fastest of sixteen mirrors is meaningless if all sixteen mirrors are being throttled.

This discussion seems relevant: rate-mirrors: Everyday-use client-side map-aware tool / Community Contributions / Arch Linux Forums

“Dirty check”: 8 mirrors are tested concurrently. My assumption is that a fast mirror should be notable even in such conditions. Also the tool waits until a speed is ±stable for every mirror.

Hm, Xyne, the Reflector creator, writes in above mentioned thread:

I completely agree that parallel speed testing makes absolutely no sense if the user connection is the bottleneck.
…
I’ll change the default number of threads to 1 but leave the option for those with better connections.

I tend to agree with him. Even the results of the “dirty check” might be bogus if the available bandwidth is not sufficient which means that the pre-selected mirrors in that first step might not necessarily be the fastest.

It may be best to leave the default as-is, and allow opting in for additional mirror concurrencies to be a special “hack” for those who benefit from a generous internet bandwidth.

edu4rdshl · April 25, 2023, 9:23pm

@SGS noted.

I agree with you, @BluishHumility.

@rinchen what about proposing to the rate-mirrors devs to support a config file so that anyone can set whatever they want? That would be nice, though.

SGS · April 25, 2023, 9:46pm

https://gitlab.com/garuda-linux/packages/stable-pkgbuilds/garuda-update/-/commit/ba03628d5b43098297d28b12725afeed2070d720

rinchen · April 25, 2023, 11:34pm

Done.

lum1nuss · April 26, 2023, 4:14am

Wait its possible for people can get 50mbps? I thought the max was 9mbps at 2AM when everyone’s asleep…
oh wait im Australian, ofc our internet is that slow

rinchen · April 26, 2023, 5:35pm

rate-mirrors will gain a config file.

github.com/westandskif/rate-mirrors

Feature request: config file

opened 11:34PM - 25 Apr 23 UTC

rinchen

Hi, I've run into a situation where perhaps a configuration file for rate-mir…rors would be helpful. The current currency default is 8 and that works well for users on slower internet connections. For those on higher speed connections, a setting of 16 works much better. Several Arch-based distributions use rate-mirrors as part of their update scripts. Some use the default concurrency setting and some override it. It would be nice to have a configuration file to set the concurrency parameter in so that these tools can be extended to allow end-users to specify their internet speeds. These tools could then be extended to support the pacman.conf parallel option as well. Here is an example: ``` Slow: concurrency = 4 parallel = 2 Default: concurrency = 8 parallel = 5 Fast: concurrency = 16 parallel = 10 ``` What are your thoughts on adding this feature into rate-mirrors? Thanks.

system · April 28, 2023, 5:36pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.

rinchen · July 24, 2023, 4:49pm

A config file was not able to be added but command line parameters were. Also, the default concurrency was raised. Github comment.

rinchen · July 24, 2023, 4:49pm

This is an update to the locked conversation: Garuda Update change request

A config file was not able to be added but command line parameters were. Also, the default concurrency was raised. Github comment.