repo-mirrorlist quality control?

[Kevin Fenzi]

On Wed, 25 Feb 2015 16:29:14 +0100
Ralf Corsepius <rc040203 at freenet.de> wrote:

> > If so, then you have cached metadata thats older than a few days.
> No.
> 
> > If not, then it's indeed those mirrors that are out of sync.
> 
> These mirrors are out of sync. Whether you like it or not,
> mirrormanager is dysfunctional.

That is not very helpfull. 

The problem is that managing mirrors is not a simple problem, it's a
very complex one. You have to see when repos change on the master side,
you have to reply to lots and lots and lots of requests from users for
that data. You have to try and check mirrors to see they are up to
date, but you can't check every single file on every single mirror, so
you check what you can and hope it matches up. You also cannot check
every mirror all the time, you have to stagger things. You also cannot
run any code on any mirror beyond simple scripts that mirrors choose to
run or not. Also some mirrors are up to date on some content and not
others, so you have to adjust for that (ie, some don't carry isos or
debuginfo, but otherwise all other packages). 

If you have a simple solution to this problem, I look forward to your
proof of concept. 

> It is a provable matter of fact that it points users (yum, mock, ...)
> to broken and out of sync mirrors.

There will be such times, sure. I don't see any way of eliminating
that. We can reduce it as much as we can with the resources we have. 

There are some things we can do: 

* If there's a mirror that is reporting that it's up to date, but is
  not, we can remove it from the list and ask mirror admins to check
  it. 

* In mirrorlists and metalinks we provide a list of mirrors. If some
  are out of sync, yum/dnf should move on to the next and retry. It
  sounds like you might see some cases where your metadata is updated
  locally and all mirrors fail? Please report such bugs. 
https://fedorahosted.org/mirrormanager/

* We can try and urge more mirrors to sync based on fedmsg (using
  last-sync) instead of just randomly N times a day. They would then
  reduce load on master mirrors and get content faster: 
https://fedoraproject.org/wiki/Infrastructure/Mirroring#Mirror_Frequency

* We can finish deploying our mirrormanager2 re-write. It doesn't add
  many/any new features, but it moves mirrormanager to a codebase we
  can work on more easily and bugfix/add features to. 
https://github.com/fedora-infra/mirrormanager2

Constructive bugs, ideas or patches welcome. 

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20150225/c8cff250/attachment.sig>