Hello,
What is going on with fedora extras metadata lately? Every time I run yum I need to download more then 30megs of metadata before yum finds a mirror with a valid checksum. It always happens when yum needs to import the filelists.xml.gz file. It's about 3.2MB in size and it fails most of the times on +/- 10 mirrors, that's 32MB of data that is wasted. Last time I updated the 14th mirror was succesfull. It also takes a lot of time for updates to finish.
filelists.xml.gz 100% |=========================| 3.2 MB 00:30 http://sunsite.mff.cuni.cz/pub/fedora-extras/development/i386/repodata/filel...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.1 MB 00:38 http://mirrors.kernel.org/fedora/extras/development/i386/repodata/filelists....: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 01:44 http://mirror.hiwaay.net/redhat/fedora/linux/extras/development/i386/repodat...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:30 http://ftp-stud.fht-esslingen.de/pub/Mirrors/fedora.redhat.com/linux/extras/...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:48 http://ftp-stud.fht-esslingen.de/pub/fedora/linux/extras/development/i386/re...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:36 http://download.fedora.redhat.com/pub/fedora/linux/extras/development/i386/r...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:33 http://fr2.rpmfind.net/linux/fedora/extras/development/i386/repodata/filelis...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:36 http://ftp.chg.ru/pub/Linux/fedora/linux/extras/development/i386/repodata/fi...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:17 http://mirror.switch.ch/ftp/mirror/fedora/linux/extras/development/i386/repo...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:15 ftp://fedora.bu.edu/fedora/extras/development/i386/repodata/filelists.xml.gz: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 01:26 http://mirror.linux.duke.edu/pub/fedora/linux/extras/development/i386/repoda...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.1 MB 00:13 http://ftp.lug.ro/fedora/linux/extras/development/i386/repodata/filelists.xm...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.2 MB 00:21 http://mirror.web-ster.com/fedora/extras/development/i386/repodata/filelists...: [Errno -1] Metadata file does not match checksum Trying other mirror. filelists.xml.gz 100% |=========================| 3.0 MB 00:17 extras-dev: ################################################## 4225/4225
greetings,
Bart
On Thu, 15 Mar 2007 23:13:41 +0100, Bart Vanbrabant wrote:
Hello,
What is going on with fedora extras metadata lately?
Nothing. It's just the mirrors that choke on daily updates and don't sync safely and frequently enough.
http://mirrors.kernel.org/fedora/extras/development/i386/repodata/ still contains metadata from March 11th, for instance.
The Extras master site is never in an inconsistent state, because normal use of the script(e) we use doesn't allow that. That is, if something goes wrong during createrepo, we get a fatal error, and any incomplete changes are not published automatically.
And for tomorrow, there are many updates in the queue again already:
$ lspen fedora-development-extras: ccache-2.4-8.fc7 gaim-meanwhile-2.0.0-0.6.beta6.fc7 gconfmm26-2.18.0-1.fc7 gdal-1.4.0-15.fc7 glibmm24-2.12.7-1.fc7 glom-1.4.0-1.fc7 gnome-vfsmm26-2.18.0-1.fc7 gtk-murrine-engine-0.51-2.fc7 gtkmm24-2.10.8-1.fc7 hawknl-1.68-1.fc7 java-1.5.0-gcj-1.5.0.0-1.fc7 jd-1.8.8-0.1.cvs070315.fc7 libfreebob-1.0.3-1.fc7 libgnomemm26-2.18.0-1 libgnomeuimm26-2.18.0-1 linux_logo-4.16-1.fc7 maven-wagon-1.0-0.1.a5.3jpp.1.fc7 opensc-0.11.2-0.2.pre6.fc7 plexus-compiler-1.5.2-2jpp.2.fc7 poker-engine-1.0.24-1.fc7 poker-network-1.0.36-1.fc7 poker2d-1.0.36-1.fc7 qt4-4.2.3-3.fc7 rhino-1.6-0.1.r5.1jpp.2.fc7 scorchwentbonkers-1.1-2.fc7 xine-plugin-1.0-3.fc7 xtide-2.9.1-1.fc7
fedora-6-extras: gdal-1.4.0-15.fc6 gtk-murrine-engine-0.51-2.fc6 hawknl-1.68-1.fc6 linux_logo-4.16-1.fc6 poker-engine-1.0.24-1.fc6 poker-network-1.0.36-1.fc6 poker2d-1.0.36-1.fc6 qt4-4.2.3-3.fc6.1 scorchwentbonkers-1.1-2.fc6 xine-plugin-1.0-3.fc6 xtide-2.9.1-1.fc6
fedora-5-extras: gtk-murrine-engine-0.51-2.fc5 linux_logo-4.16-1.fc5 poker-engine-1.0.24-1.fc5 poker-network-1.0.36-1.fc5 poker2d-1.0.36-1.fc5 qt4-4.2.3-3.fc5 xtide-2.9.1-1.fc5
Michael Schwendt wrote:
On Thu, 15 Mar 2007 23:13:41 +0100, Bart Vanbrabant wrote:
Hello,
What is going on with fedora extras metadata lately?
Nothing. It's just the mirrors that choke on daily updates and don't sync safely and frequently enough.
http://mirrors.kernel.org/fedora/extras/development/i386/repodata/ still contains metadata from March 11th, for instance.
Ok, thanks.
Bart
But perhaps we should move repoview out of the repodata directory into parent dir. That would get rid of thousands of html files in the repodata tree and reduce the time mirrors spend in that tree.
Michael Schwendt wrote:
What is going on with fedora extras metadata lately?
Nothing. It's just the mirrors that choke on daily updates and don't sync safely and frequently enough.
This seems to be happening more often that we could hope for. Is there a documented way to set up mirroring, to ENSURE that the mirrors are in a consistent state? If not - and I believe this has been brought up earlier (by myself). I really think we could do with a simple timestamping mirror handshake mechanism, kinda like what debian does. This would allow mirrors to monitor for a special file and when that file is available, we know the mirror is in a consistent state (as consistent as it's master - which can also be tracked in this way). Mirrors could easily add a few lines to their scripts to honor this kind of thing, without the need for special mirroring tools.
Just a suggestion
/Thomas
Thomas M Steenholdt wrote:
( ... ) Mirrors could easily add a few lines to their scripts to honor this kind of thing, without the need for special mirroring tools.
Yum could easily use the very same technique to make a quick *validation* of a mirror before it goes on to download the large files... This should be done in a plugin though, since this is not related directly to yum.
/Thomas
On Fri, 16 Mar 2007 08:17:47 +0100, Thomas M Steenholdt wrote:
Michael Schwendt wrote:
What is going on with fedora extras metadata lately?
Nothing. It's just the mirrors that choke on daily updates and don't sync safely and frequently enough.
This seems to be happening more often that we could hope for.
As mentioned in the thread "repoview in our repositories", I believe repoview may be one culprit, since it's located *inside* the repodata directory and potentially increases the time a mirror spends in that tree.
Repoview was put in place at the beginning of Fedora Extras and has been kept in pretty much a "nobody cares about it" state. Meanwhile it results in thousands of html files. More than 16,000 pages per dist release! And until recently several thousands more for the debuginfo repos. Too many of the pages change when we publish updated packages. It's not 1:1, but 1:N (one updated package => many updated pages), see the other thread for more details. And this is the size of repoview for extras devel:
24M ./SRPMS/repodata/repoview 39M ./i386/repodata/repoview 38M ./ppc/repoview 41M ./x86_64/repodata/repoview
Is there a documented way to set up mirroring, to ENSURE that the mirrors are in a consistent state?
Not that I know of, as we don't do anything like clearing and setting a flag file before and after the sync.
So, theoretically it can happen that a mirror is downloading while we copy new packages and new metadata to Red Hat. And if that happens while a mirror is choking on thousands of repoview pages inside the repodata dir, this increases the window during which downloads can become inconsistent.
Mirrors that don't sync daily are hit hard apparently:
development/i386/repodata/ mirrors.kernel.org 2007-03-11 11:44 repomd.xml
If not - and I believe this has been brought up earlier (by myself). I really think we could do with a simple timestamping mirror handshake mechanism, kinda like what debian does. This would allow mirrors to monitor for a special file and when that file is available, we know the mirror is in a consistent state (as consistent as it's master - which can also be tracked in this way). Mirrors could easily add a few lines to their scripts to honor this kind of thing, without the need for special mirroring tools.
Just a suggestion
/Thomas
What does the algorithm look like?
"Handshake" implies bidirectional communication, which is not available. We only have "slave retrieves from master" or "slave retrieves from slave" relationships. Mutual exclusion is not possible either. Flag files don't make the mirroring an atomic operation. The master can still break the scheme and push updates while a mirror is downloading in believe that the previously checked flag file was up-to-date.
Michael Schwendt wrote:
/Thomas
What does the algorithm look like?
"Handshake" implies bidirectional communication, which is not available. We only have "slave retrieves from master" or "slave retrieves from slave" relationships. Mutual exclusion is not possible either. Flag files don't make the mirroring an atomic operation. The master can still break the scheme and push updates while a mirror is downloading in believe that the previously checked flag file was up-to-date.
IIRC, something like echoing the output of `date` into af file named "mirror.wherever.com" in a special dir, upon successful synchronization from upstream mirror. That same file would be deleted when starting the next sync (there are tricks you can do, like download new packages to a temporary dir before deleting old packages and moving stuff in place, to shorten that period as much as possible). The point would be that if I'm syncing from mirror.somehost.com, I can check if the file mirror.somehost.com exists before even trying. If it exists, the mirror is consistent, but could still be stale. Staleness would be evident from the timestamp.
Also, since we would sync the special directory with the mirror.somehost.com file, we can even track problems to watever upstream mirror host and check how old this particular set of packages has been on their way from the main site.
Something along those lines.
See http://www.debian.org/mirror/ftpmirror for more info on how the debian project does it, and perhaps we can be inspired by that. search for project/trace on that site, since that's where they'll throw the timestamp file.
/Thomas
On Fri, Mar 16, 2007 at 08:17:47AM +0100, Thomas M Steenholdt wrote:
Michael Schwendt wrote:
What is going on with fedora extras metadata lately?
Nothing. It's just the mirrors that choke on daily updates and don't sync safely and frequently enough.
This seems to be happening more often that we could hope for. Is there a documented way to set up mirroring, to ENSURE that the mirrors are in a consistent state? If not - and I believe this has been brought up earlier (by myself). I really think we could do with a simple timestamping mirror handshake mechanism, kinda like what debian does. This would allow mirrors to monitor for a special file and when that file is available, we know the mirror is in a consistent state (as consistent as it's master - which can also be tracked in this way). Mirrors could easily add a few lines to their scripts to honor this kind of thing, without the need for special mirroring tools.
Just a suggestion
Matt Domsch is working on just such a tool and is looking to have it in place for F7 release, afaik. The tool is Mirror Manager.
http://fedoraproject.org/wiki/Infrastructure/RFR/MirrorManager
He has a beta out there, iirc. If you have python/turbogears experience, I think he could use help on it.
-- Michael
Michael E Brown wrote:
Matt Domsch is working on just such a tool and is looking to have it in place for F7 release, afaik. The tool is Mirror Manager.
This looks like a very competent tool indeed and there's no doubt that it will be very useful for a lot of cases. However I have no idea how the mirror validation in the package will work, I just hope it will be implemented in a way that will be usable without special tools. Having a way to validate a mirror from within the ftp directory listing is very valuable - especially to mirror scripts etc.
It looks to me like the MirrorManager will notify the main site that the sync has completed. This is useful, but probably not to other mirror sites (or we may need specialized tools to perform the check). I could easily be mistaken here, though!
/Thomas
On Fri, Mar 16, 2007 at 10:01:57PM +0100, Thomas M Steenholdt wrote:
Michael E Brown wrote:
Matt Domsch is working on just such a tool and is looking to have it in place for F7 release, afaik. The tool is Mirror Manager.
This looks like a very competent tool indeed and there's no doubt that it will be very useful for a lot of cases. However I have no idea how the mirror validation in the package will work, I just hope it will be implemented in a way that will be usable without special tools. Having a way to validate a mirror from within the ftp directory listing is very valuable - especially to mirror scripts etc.
It looks to me like the MirrorManager will notify the main site that the sync has completed. This is useful, but probably not to other mirror sites (or we may need specialized tools to perform the check). I could easily be mistaken here, though!
First, the problems with the current mirroring is really twofold:
1) the storage array backing the master rsync servers is undergoing some serious stress. This is causing it to be very slow for the master rsync servers that serve the data, thus the global mirror servers pulling from it are seeing very slow syncs. Red Hat I/T is working on it. (It isn't helping that the RHEL5 floodgates opened on Wednesday either - that just added stress to an already stressed set of people and colo networks).
2) the global mirrors aren't being notified when content has changed on the master, such that they should start a new rsync run. mirrormanager takes a per-host email address, which the master sign-and-push scripts will eventually send an email to when the content changes. As it stands, syncing every 6 hours when nothing has changed doesn't make any sense.
Now, mirrormanager has 2 methods by which it can know a give mirror host is up-to-date. First is a new report_mirrors script that uploads directory data back to the database from the mirror itself. Not everyone will want to run that, and there's always the 'trust but verify' model, so I've also got a (fast?) crawler that crawls each host using HTTP HEADs and keepalives or FTP DIR calls looking for content it should be carrying compared to the master list, and tracking presence and up-to-date-ness on a per-directory level. Those that aren't up2date get dropped from the appropriate per-directory lists (e.g. the repodata dirs) in real time.
That's the idea. A lot of the code is implemented, there's more to go. If you're good with python, turbogears, and the like, I'm sure I could put you to work on it. Drop me a note.
Thanks, Matt
Matt Domsch wrote:
First, the problems with the current mirroring is really twofold:
( ... snip ... )
- the global mirrors aren't being notified when content has changed on the master, such that they should start a new rsync run. mirrormanager takes a per-host email address, which the master sign-and-push scripts will eventually send an email to when the content changes. As it stands, syncing every 6 hours when nothing has changed doesn't make any sense.
Now, mirrormanager has 2 methods by which it can know a give mirror host is up-to-date. First is a new report_mirrors script that uploads directory data back to the database from the mirror itself. Not everyone will want to run that, and there's always the 'trust but verify' model, so I've also got a (fast?) crawler that crawls each host using HTTP HEADs and keepalives or FTP DIR calls looking for content it should be carrying compared to the master list, and tracking presence and up-to-date-ness on a per-directory level. Those that aren't up2date get dropped from the appropriate per-directory lists (e.g. the repodata dirs) in real time.
Sounds good, but it still seems like we need special tools to determine on the fly, with standard tools, if a mirror is in a sync'ed state. If I need to create a local mirror of fedora X with updates, with standard tools, I have to figure out when to mirror by trial and error and have no way to check before the sync, if the mirror is in the middle of syncing itself.
The mirrormanager you're working on will be useful for monitoring and flagging stale mirrors. It could work well for yum too, but for the basic mirroring, we still lack a simple mechanism to allow us to track the state of upstream mirrors. Mirroring a few gigs of inconsistent data makes no sense at all and this can be implemented very very easily. It's all a matter of documenting "HOWTO setup a fedora mirror".
That's the idea. A lot of the code is implemented, there's more to go. If you're good with python, turbogears, and the like, I'm sure I could put you to work on it. Drop me a note.
Never worked with turbogears, sorry ;-)
/Thomas
-
Bart Vanbrabant -
Matt Domsch -
Michael E Brown -
Michael Schwendt -
Thomas M Steenholdt