MirrorManager (was Re: Improving availability and guaranteeing integrity in ISO downloads)

[Anthony Bryan]

Hello, this is a repost of a message to fedora-devel, as this seems to
be the place for it.

>  From: Jesse Keating <jkeating(a)redhat.com&gt;
>
>  On Friday 08 June 2007 14:24:47 Anthony Bryan wrote:
>  > I was hoping Fedora could investigate using Metalinks for their ISO
>  > downloads. Metalink is an XML format for listing all the ways you can
>  > get a file or collection of files (mirrors + their location, rsync,
>  > p2p) along with checksums to automatically repair a file in case of
>  > error, signatures, language, OS/arch, and other metadata. It's mainly
>  > used for large files like ISOs, where errors can be very frustrating.
>  >
>  > It's supported by about 20 programs on unix, mac, and win, including
>  > aria2 (already in the Fedora repos). It's used by openSUSE,
>  > OpenOffice.org, cURL, and many other distributions.
>  >
>  > Here's a screenshot of a Metalink download in the DownThemAll Firefox
>  > extension (nightly build). What you don't see are all the mirrors and
>  > checksums.
>  > http://code.downthemall.net/maierman/metaselect4.png
>  >
>  > http://en.wikipedia.org/wiki/Metalink
>
>  This is something interesting, and I wonder if we could make use of
>  MirrorManager ( https://hosted.fedoraproject.org/projects/mirrormanager ) to
>  have dynamic .metalink files created with updated mirror readiness info.
>  Certainly something that looks worth looking into. 
That would be quite nice, no one else has dynamic .metalinks on a
large scale. When I got the F7 ISO, I noticed it would fit in well w/
the download pages which tell which mirrors have which releases. I
think it would make things less frustrating for end users trying to
get things, and hopefully create less strain on mirrors. Certain
metalink clients will download from domestic mirrors first, if country
info is in there, which should hopefully be more efficient for
everyone.

What can we do to make this happen? Is this the type of thing that's
easier for the maintainer of MirrorManager to add, or should we supply
a patch?

Here's the current tools people have done, if that helps -
http://www.metalinker.org/implementation.html#generate

Metalink Editor - in Python, GUI
cURL - they use a short Perl script that makes them based on location.
Simba/RoPkg::Metalink - Perl
Bouncer - there's a patch for it.
Metalink tools - CLI, C++

--
(( Anthony Bryan
 )) Metalink [ http://www.metalinker.org ]

[Matt Domsch]

On Sat, Jun 09, 2007 at 04:22:24AM -0400, Anthony Bryan wrote:
>  Hello, this is a repost of a message to fedora-devel, as this seems
> to
>  be the place for it.
>  
>  >From: Jesse Keating <jkeating(a)redhat.com&gt;
>  >
>  >On Friday 08 June 2007 14:24:47 Anthony Bryan wrote:
>  >> I was hoping Fedora could investigate using Metalinks for their ISO
>  >> downloads. Metalink is an XML format for listing all the ways you can
>  >> get a file or collection of files (mirrors + their location, rsync,
>  >> p2p) along with checksums to automatically repair a file in case of
>  >> error, signatures, language, OS/arch, and other metadata. It's mainly
>  >> used for large files like ISOs, where errors can be very frustrating.
>  >>
>  >> It's supported by about 20 programs on unix, mac, and win, including
>  >> aria2 (already in the Fedora repos). It's used by openSUSE,
>  >> OpenOffice.org, cURL, and many other distributions.
>  >>
>  >> Here's a screenshot of a Metalink download in the DownThemAll Firefox
>  >> extension (nightly build). What you don't see are all the mirrors and
>  >> checksums.
>  >> http://code.downthemall.net/maierman/metaselect4.png
>  >>
>  >> http://en.wikipedia.org/wiki/Metalink
>  >
>  >This is something interesting, and I wonder if we could make use of
>  >MirrorManager ( https://hosted.fedoraproject.org/projects/mirrormanager ) 
>  >to
>  >have dynamic .metalink files created with updated mirror readiness info.
>  >Certainly something that looks worth looking into.
>  
>  That would be quite nice, no one else has dynamic .metalinks on a
>  large scale. When I got the F7 ISO, I noticed it would fit in well w/
>  the download pages which tell which mirrors have which releases. I
>  think it would make things less frustrating for end users trying to
>  get things, and hopefully create less strain on mirrors. Certain
>  metalink clients will download from domestic mirrors first, if country
>  info is in there, which should hopefully be more efficient for
>  everyone.
>  
>  What can we do to make this happen? Is this the type of thing that's
>  easier for the maintainer of MirrorManager to add, or should we supply
>  a patch? 
I saw some articles on metalink and spent time time looking at it a
few weeks ago.  Good stuff.

Patches are certainly welcome.  What I picture is a new application
'generate_metalinks' along the lines of the generate_mirrorlists
application within mirrormanager that connects to the database, finds
what it wants, and generates static .metalink text files.  You'd
probably only want to metalink the .ISOs, of which there are plenty.
No sense making .metalink files for every file in the (presently)
660GB file system.

Then we also need to generate static web pages that include HTML links
to all the .metalink files.  And link to those from somewhere in the
mirrors.fedoraproject.org/ namespace, maybe
mirrors.fp.o/metalink/index.html plus all the .metalink files in a dir
there.

Then we add it into the update-static-content script that runs at the
top of the hour, so they're dynamically generated.

 
>  Here's the current tools people have done, if that helps -
>  http://www.metalinker.org/implementation.html#generate
>  
>  Metalink Editor - in Python, GUI
>  cURL - they use a short Perl script that makes them based on location.
>  Simba/RoPkg::Metalink - Perl
>  Bouncer - there's a patch for it.
>  Metalink tools - CLI, C++ 
As mm is written in python + TurboGears, it would be great if the
metalink creator tool was also in python. :-)

Source to mm is available here:
https://hosted.fedoraproject.org/projects/mirrormanager

Thanks,
Matt

-- 
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com