repomd.xml timestamp checking for http://mirrors.fp.org/

[Florian La Roche]

Hello,

If you look at the below data, it seems that some hosts with older
repomd.xml data are still in the mirrorlist and not removed. Is the
data not checked further to leave out older mirrors until now?

regards,

Florian La Roche



Getting mirrorlist from updates-released-debug-fc5-DE-ppc.txt.
cacheLocal: looking at url: updates-released-debug-fc5-DE-ppc.txt
Reading yum repository
http://ftp.uni-koeln.de/mirrors/fedora/linux/core/updates/5/ppc/debug.
cacheLocal: looking at url:
http://ftp.uni-koeln.de/mirrors/fedora/linux/core/updates/5/ppc/debug/rep...
2007/04/24
Reading yum repository
http://ftp.uni-erlangen.de/pub/Linux/MIRROR.fedora/core/updates/5/ppc/debug.
cacheLocal: looking at url:
http://ftp.uni-erlangen.de/pub/Linux/MIRROR.fedora/core/updates/5/ppc/deb...
2007/05/14
Reading yum repository
http://ftp.uni-muenster.de/pub/linux/distributions/fedora/linux/core/upda....
cacheLocal: looking at url:
http://ftp.uni-muenster.de/pub/linux/distributions/fedora/linux/core/upda...
2007/05/14
Reading yum repository http://ftp.stw-bonn.de/pub/fedora/linux/core/updates/5/ppc/debug.
cacheLocal: looking at url:
http://ftp.stw-bonn.de/pub/fedora/linux/core/updates/5/ppc/debug/repodata...
2007/05/14
Reading yum repository ftp://ftp.fu-berlin.de/linux/fedora/core/updates/5/ppc/debug.
cacheLocal: looking at url:
ftp://ftp.fu-berlin.de/linux/fedora/core/updates/5/ppc/debug/repodata/repomd.xml
2007/05/14
Reading yum repository
http://wftp.tu-chemnitz.de/pub/linux/fedora-core/updates/5/ppc/debug.
cacheLocal: looking at url:
http://wftp.tu-chemnitz.de/pub/linux/fedora-core/updates/5/ppc/debug/repo...
2007/05/14
Reading yum repository
ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/fedora-core/updates/5/ppc/debug.
cacheLocal: looking at url:
ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/fedora-core/updates/5/ppc/debug/repodata/repomd.xml
2007/05/08
Reading yum repository
http://ftp-stud.hs-esslingen.de/pub/fedora/linux/core/updates/5/ppc/debug.
cacheLocal: looking at url:
http://ftp-stud.hs-esslingen.de/pub/fedora/linux/core/updates/5/ppc/debug...
2007/05/14
Reading yum repository
http://ftp.uni-bayreuth.de/linux/fedora/linux/core/updates/5/ppc/debug.
cacheLocal: looking at url:
http://ftp.uni-bayreuth.de/linux/fedora/linux/core/updates/5/ppc/debug/re...
2007/05/14

[Matt Domsch]

On Tue, May 15, 2007 at 04:08:40PM +0200, Florian La Roche wrote:
>  Hello,
>  
>  If you look at the below data, it seems that some hosts with older
>  repomd.xml data are still in the mirrorlist and not removed. Is the
>  data not checked further to leave out older mirrors until now? 
No, I see the problem.  Thanks for the report.  I knew this was going
to bite me sooner or later, and have plans to fix it, but hadn't had
time to implement it...
 

>  Getting mirrorlist from updates-released-debug-fc5-DE-ppc.txt.
>  cacheLocal: looking at url: updates-released-debug-fc5-DE-ppc.txt
>  Reading yum repository
> http://ftp.uni-koeln.de/mirrors/fedora/linux/core/updates/5/ppc/debug.
>  cacheLocal: looking at url:
> http://ftp.uni-koeln.de/mirrors/fedora/linux/core/updates/5/ppc/debug/rep...
>  2007/04/24 
This host was crawled today, and that the repodata directory isn't
up-to-date, but the crawler thinks that the updates/5/ppc/debug
directory *is* up-to-date.   But it shouldn't.

up2date   path  lastCrawled date

True updates/5/ppc 2007-05-16 07:43:17
True updates/5/ppc/debug 2007-05-16 07:43:10
False updates/5/ppc/debug/repodata 2007-05-16 07:43:02
False updates/5/ppc/repodata 2007-05-16 07:43:22

The problem is, the database doesn't track every single file in a
directory full of RPMs or HTML files.  That would have exploded the
size of the database. So, I only look at 10 files in such directories
(and all files in all other directories).  It's supposed to be "the
ten most recent files", but that part of the logic is what's broken.
It uses the first ten as reported by the dict, which isn't right.

I'll have to fix that to solve this.  It's just a matter of adding
rsync directory list parsing into the update-master-directory-list
program, shouldn't be hard, just needs a little time and a little
testing...

Thanks,
Matt

-- 
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com

[Matt Domsch]

On Wed, May 16, 2007 at 09:14:17AM -0500, Matt Domsch wrote:
>  On Tue, May 15, 2007 at 04:08:40PM +0200, Florian La Roche wrote:
>  > Hello,
>  > 
>  > If you look at the below data, it seems that some hosts with older
>  > repomd.xml data are still in the mirrorlist and not removed. Is the
>  > data not checked further to leave out older mirrors until now?
>  
>  No, I see the problem.  Thanks for the report.  I knew this was going
>  to bite me sooner or later, and have plans to fix it, but hadn't had
>  time to implement it... 
And now I have...

This should be fixed now, so future crawls will pick it up correctly.

Thanks,
Matt

-- 
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com