Official mirror requirements

[Rahul Sundaram]

Hi

What are the requirements other than bandwidth for the official mirrors 
listed at http://fedora.redhat.com/Download/mirrors.html. In particular 
do we require them to carry the complete copy including source images 
and packages? If not we should probably enforce that or list the mirrors 
which only have binary packages as partial mirrors. We can run some 
routine checks in a period basis automatically to cross verify this.

Rahul

[Mike McGrath]

On 12/27/06, Rahul Sundaram  wrote:
> Hi
>
> What are the requirements other than bandwidth for the official mirrors
> listed at http://fedora.redhat.com/Download/mirrors.html. In particular
> do we require them to carry the complete copy including source images
> and packages? If not we should probably enforce that or list the mirrors
> which only have binary packages as partial mirrors. We can run some
> routine checks in a period basis automatically to cross verify this.
>
> Rahul

I don't think there are bandwith requirements though we do regularly
test the mirrors and if they are overloaded would time out.  I think
they can be somewhat selective (as in not a full copy of all of
rawhide, testing, updates, etc) though it should be a requirement.

We're currently re-writing a new mirror system that can address some
of this stuff.  But monitoring an entire mirror is difficult because
you'd basically have to download it and md5sum everything which is
horribly inefficient.  Our checks are much simpler right now.

-Mike

[Matt Domsch]

On Tue, Jan 23, 2007 at 10:27:34AM -0600, Mike McGrath wrote:
> On 12/27/06, Rahul Sundaram  wrote:
> > Hi
> >
> > What are the requirements other than bandwidth for the official mirrors
> > listed at http://fedora.redhat.com/Download/mirrors.html. In particular
> > do we require them to carry the complete copy including source images
> > and packages? If not we should probably enforce that or list the mirrors
> > which only have binary packages as partial mirrors. We can run some
> > routine checks in a period basis automatically to cross verify this.
> >
> > Rahul
>
> I don't think there are bandwith requirements though we do regularly
> test the mirrors and if they are overloaded would time out.  I think
> they can be somewhat selective (as in not a full copy of all of
> rawhide, testing, updates, etc) though it should be a requirement.
>
> We're currently re-writing a new mirror system that can address some
> of this stuff.  But monitoring an entire mirror is difficult because
> you'd basically have to download it and md5sum everything which is
> horribly inefficient.  Our checks are much simpler right now.

I'm working on it... :-)

We don't force everyone to carry everything.  Most folks don't carry
Extras right now.  Most folks carry only i386 and x86_64.  A few only
carry updates.  And *everyone* uses a different URL to get to their
content.  Maintaining it manually is impossible anymore - hence the
need for some software.

See http://fedoraproject.org/wiki/Infrastructure/MirrorManagement for
what I've got in mind, a little of which already works.

-Matt

[Rahul Sundaram]

Matt Domsch wrote:
> I'm working on it... :-)
>
> We don't force everyone to carry everything.  Most folks don't carry
> Extras right now.  Most folks carry only i386 and x86_64.  A few only
> carry updates.  And *everyone* uses a different URL to get to their
> content.  Maintaining it manually is impossible anymore - hence the
> need for some software.
>
> See http://fedoraproject.org/wiki/Infrastructure/MirrorManagement for
> what I've got in mind, a little of which already works.

Thanks. The original mail was send by me long back before I subscribed 
to this list and has been struck on the moderation queue for a while I 
guess. I did find this page and added a note on source packages. It 
would very useful if the individual mirrors had a note clearly marking 
the type of access (http/ftp/rynsc), repositories (core/extras), 
architecture, source and binary packages and other details so users can 
quickly understand which mirror carry the content they want and how to 
access them.

Rahul

[Matt Domsch]

On Tue, Jan 23, 2007 at 11:24:36PM +0530, Rahul Sundaram wrote:
> Matt Domsch wrote:
> > I'm working on it... :-)
> >
> > We don't force everyone to carry everything.  Most folks don't carry
> > Extras right now.  Most folks carry only i386 and x86_64.  A few only
> > carry updates.  And *everyone* uses a different URL to get to their
> > content.  Maintaining it manually is impossible anymore - hence the
> > need for some software.
> >
> > See http://fedoraproject.org/wiki/Infrastructure/MirrorManagement for
> > what I've got in mind, a little of which already works.
>
> Thanks. The original mail was send by me long back before I subscribed 
> to this list and has been struck on the moderation queue for a while I 
> guess. I did find this page and added a note on source packages. It 
> would very useful if the individual mirrors had a note clearly marking 
> the type of access (http/ftp/rynsc), repositories (core/extras), 
> architecture, source and binary packages and other details so users can 
> quickly understand which mirror carry the content they want and how to 
> access them.

Absolutely.  And GeoIP, and ...

The current mirror list is a mess.  There are folks on it that aren't
mirrors anymore.  Some of the URLs are wrong.  I bet it's missing some
mirrors too.  It really only lists Core, not Extras or anything else.
When a release happens, everyone manually sends a "I'm synced" message
which someone has to collect and include in the release announcement.
All in all, a good opportunity for some nice software.

Turns out, Debian's got something, though I haven't found the source
for it yet.  But they do a lot of what we would want, plus
ftp.<country>.debian.org DNS maintenance which we don't have to do
right now because we're using GeoIP to build the per-country yum lists.