About MirrorManager2

[Pierre-Yves Chibon]

On Mon, Feb 23, 2015 at 04:42:46PM -0500, David Gay wrote:
> ----- Original Message -----
> > From: "Pierre-Yves Chibon" <pingou at pingoured.fr>
> > To: "Fedora Infrastructure" <infrastructure at lists.fedoraproject.org>
> > Sent: Wednesday, February 11, 2015 5:10:01 AM
> > Subject: About MirrorManager2
> 
> [snip]
> 
> > Before we move MirrorManager2 to prod, here is what I think would be nice to
> > do/have:
> > 
> > - Pickle validation
> >   - Figure a way to validate a pickle after its creation and before moving it
> >   to
> >     the mirrorlist boxes
> > - Find out if we can improve our tests some more
> >   (to improve our confidence that we're ready)
> > - Engage the mirror mailing list and try to get them to react on the coming
> >   changes
> > 
> > The pickle validation might also be an interesting idea to check if there is
> > a
> > difference between the pickle generated by prod and the one generated in stg.
> > 
> >  
> > Finally, at DevConf we have been speaking quite a bit with Dennis around
> > updates and MirrorManager and here is some of the ideas we spoke about:
> >  
> > - Be able to run the UMDL script on only a part of the tree
> >   (ie: be able to say, we updated f21-updates and we only update this part)
> > - Crawl the mirrors for only a part of the tree
> >   (This goes together with updating only part of the tree via UMDL)
> > - Consider if we should/could drop the content of the host_category_dir table
> >   before running the crawler
> > - Mirror versioning:
> >   - run UMDL, detect changes, increase master mirror's version by 1
> >   - run the crawler, check for the changes, align that mirror's version with
> >   the
> >     master mirror one
> >   - be able to see the difference between two versions
> >   - be able to crawl a mirror only for the difference between the version it
> >   is
> >     at and the version the master mirror is at
> >   note: we might still want to run a full crawl once in a while (daily?
> >   bi-daily?)
> >  
> > This list of ideas is more a long term todo list, not something we would want
> > to have working for pushing MirrorManager2 to prod.
> > 
> 
> I'd like to help move this along to production. I have very limited knowledge of the MM codebase, but I did attend the MM FAD, so I have at least a vague understanding of how it works. Are there any changes to these lists since last week? What might be productive to work on? Like I said, my knowledge is limited, but pickle validation sounds like a pretty general Python task. Maybe I could look into that?

None of these things have progressed over the last week.
Clearly the three points at the top have the most priority for MM2 (imo) and the
pickle validation might be tricky (how to test it) but is something I really
think I want to have before we switch.

The points listed below are things we may want to work on in time, but clearly
not something we want to wait for to release MM2 in prod.

> Any direction and info on some task(s) I could try my hand at is appreciated. :)

I think the starting point for this is to take a couple of pickle files
generated by MM1 currently.
Then we need to check their structure and eventually their content.
I have never looked at what the pickle file has, so I can't really say what to
expect.

Ideally, we want to develop the tests against the pickle generated by MM1, then
run them against the pickle generated by MM2.
Eventually, I think it would be a good idea to integrate the tests in the
publication workflow so that we only push to the mirrorlist servers the pickle
that we validated.

Pierre