repo-mirrorlist quality control?

Ralf Corsepius rc040203 at freenet.de
Wed Feb 25 16:47:47 UTC 2015


On 02/25/2015 04:46 PM, Kevin Fenzi wrote:
> On Wed, 25 Feb 2015 16:29:14 +0100
> Ralf Corsepius <rc040203 at freenet.de> wrote:
>
>>> If so, then you have cached metadata thats older than a few days.
>> No.
>>
>>> If not, then it's indeed those mirrors that are out of sync.
>>
>> These mirrors are out of sync. Whether you like it or not,
>> mirrormanager is dysfunctional.
>
> That is not very helpfull.

The user-side of this problem is quite simple:

The metalinks/mirrorlists as being used in yum or mock configs are 
pointing to mirrors, which are broken or out of sync. I.e. the 
metalinks/mirrorlist are incorrect.


> The problem is that managing mirrors is not a simple problem,
I don't know how fedora mirror works, esp. whether they are polling or 
whether they served with pushes.

In another (much smaller) project, when pushing updates, we had removed 
all mirrors from our mirrorlists and had let the server poll 
"repodata/repomd.xml" from the mirrors to re-adde a mirror to the 
mirrorlists if this file matched. Of course, this is a primitive 
heuristic, but it had worked quite well for us.

>> It is a provable matter of fact that it points users (yum, mock, ...)
>> to broken and out of sync mirrors.
>
> There will be such times, sure.
Here (Germany), in recent times, they happen almost daily. My guess is 
the "fedora flavors", the launch of f22 and the mass-rebuild on rawhide 
are showing their nasty side.

> I don't see any way of eliminating
> that. We can reduce it as much as we can with the resources we have.

IMO, the issues yum/mock/dnf have, partially need to be worked-around by 
heuristics in yum/mock/dnf.

I sometimes observe yum and mock seemingly to lock up to dead mirrors 
(downloaded metadata is older than previous one) and not to try a more 
recent mirror.

I also occasionally see consecutive yum/mock runs to re-iterate and fail 
over apparently the same mirrorlists (I feel, it retries by priority and 
therefore always stumbles of the same broken/dead mirrors).

Having a past of scientific work on Genetic Algorithms, my first try 
would be to introduce some "randomness" into (mirror) selection - It 
would help to breakout of deterministic causes :-)

> There are some things we can do:
>
> * If there's a mirror that is reporting that it's up to date, but is
>    not, we can remove it from the list and ask mirror admins to check
>    it.
>
> * In mirrorlists and metalinks we provide a list of mirrors. If some
>    are out of sync, yum/dnf should move on to the next and retry.
My feel is, this currently doesn't work. The worst cases seem to be yum 
alternating between several high-priorized broken/dead mirrors.

> It
>    sounds like you might see some cases where your metadata is updated
>    locally and all mirrors fail? Please report such bugs.
> https://fedorahosted.org/mirrormanager/
Yes, this had happened for a longer period (>24 hours) some time last 
week (IIRC, Thursday) and for a shorter period (2-4 hours) yesterday.

My guess is, all EU f22/rawhide mirrors were out of sync during these times.

> * We can try and urge more mirrors to sync based on fedmsg (using
>    last-sync) instead of just randomly N times a day. They would then
>    reduce load on master mirrors and get content faster:
> https://fedoraproject.org/wiki/Infrastructure/Mirroring#Mirror_Frequency
>
> * We can finish deploying our mirrormanager2 re-write. It doesn't add
>    many/any new features, but it moves mirrormanager to a codebase we
>    can work on more easily and bugfix/add features to.
> https://github.com/fedora-infra/mirrormanager2
Sorry, lack of knowledge on these details :(

> Constructive bugs, ideas or patches welcome.
In this case, I am mostly a "p***ed user", who is struggling ;)

Ralf




More information about the devel mailing list