Bit flip

[Matt Domsch]

On Tue, Apr 28, 2009 at 10:18:47AM -0500, Mike McGrath wrote:
> On Tue, 28 Apr 2009, Seth Vidal wrote:
> 
> >
> >
> > On Tue, 28 Apr 2009, Mike McGrath wrote:
> >
> > >
> > > I'm thinking more like hours.  From the last release I monitored:
> > >
> > > http://mmcgrath.fedorapeople.org/alphaMirrorRediness.html
> > >
> > > We didn't hit 80% success rate until 3 and a half hours after the launch.
> > > I think mdomsch made some good changes since this graph that will make the
> > > outcome better, but I still think that initial launch isn't right.
> > >
> > > Perhaps we should wait to send the announcement out until a certain % of
> > > mirrors are ready?  We currently don't check that prior to announce.
> > >
> >
> > I went by Jesse's wording of "make sure at least a few mirrors are synced" and
> > you said on irc that 3 mirrors in the US were synced.
> >
> > That's why I sent out the announcement then.
> >
> > That may be where the mixup was.
> >
> 
> Synced but if you clicked on those mirrors they threw a forbidden message.


Backwards math:

Time to generate and publish updated mirrorlist: 20 minutes
Time to crawl all mirrors: ~2 hours
Time for most mirrors to rsync and catch the bitflip: 6 hours
Time to sync bitflip to all three netapps: < 30 minutes (maybe <<).
Time to do bitflip: 1 minute

So Mike is correct, that if we bitflip 2.5+ hours ahead of 1400 UTC,
that by 1400 UTC we'll know which mirrors have actually picked up the
bitflip and are live.  If we bitflip a lot earlier, say 6.5 hours
ahead, we can have nearly all the mirrors with the right permissions,
and showing right on the mirrorlist.

MM is also "bad" in that it doesn't know the directory permissions on
every directory that a report_mirror-using mirror has.  So we're
showing mirrors in the publiclist, and returning them to clients on
with mirrorlist/d.fp.o when we don't know that their permissions are
correct.  Adding that knowledge to report_mirror and MM would prevent
us from returning pre-bitflip mirrors to clients.  That's a good bit
of work I haven't scoped.











-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux