About MirrorManager2

[Stephen John Smoogen]

On 11 February 2015 at 06:10, Pierre-Yves Chibon <pingou at pingoured.fr>
wrote:

> Hi everyone,
>
> We currently have MirrorManager2 running on staging. It's apparently not
> 100%
> set-up since we get emails once in a while that one of the crons failed
> (iirc among other we need to finish configuring fedmsg).
>
> Other that this, MirrorManager2 is currently in a decent shape I think.
> However, we really need to make sure nothing broke in the re-write and we
> want to make sure we won't break it in the future.
> To try to ensure that last part, I have try to write some tests for the UI
> but
> also for the backend part (all the different scripts).
> The pull-request is opened for review:
> https://github.com/fedora-infra/mirrormanager2/pull/14
>
> I have also been trying to capitalize on the knowledge we acquired during
> the
> FAD by starting to write down how mirrormanager works in the documentation:
> https://github.com/fedora-infra/mirrormanager2/compare/tests...doc
> (pull-request to be opened once the tests branch is merged)
> I would appreciate if those that were at the FAD could go through this
> branch/changes and adjust as needed.
> I have been thinking about asking Matt to do the review so that we can
> adjust
> and improve the documentation.
>
>
> Before we move MirrorManager2 to prod, here is what I think would be nice
> to
> do/have:
>
> - Pickle validation
>   - Figure a way to validate a pickle after its creation and before moving
> it to
>     the mirrorlist boxes
> - Find out if we can improve our tests some more
>   (to improve our confidence that we're ready)
> - Engage the mirror mailing list and try to get them to react on the coming
>   changes
>
> The pickle validation might also be an interesting idea to check if there
> is a
> difference between the pickle generated by prod and the one generated in
> stg.
>
>
> Finally, at DevConf we have been speaking quite a bit with Dennis around
> updates and MirrorManager and here is some of the ideas we spoke about:
>
> - Be able to run the UMDL script on only a part of the tree
>   (ie: be able to say, we updated f21-updates and we only update this part)
> - Crawl the mirrors for only a part of the tree
>   (This goes together with updating only part of the tree via UMDL)
>

For a dumb optimization (which may be there laready) I would only crawl the
trees which we know change "hourly" (updates/X/, development/X/) and only
scan releases etc daily or weekly because it should not change that much.


> - Consider if we should/could drop the content of the host_category_dir
> table
>   before running the crawler
> - Mirror versioning:
>   - run UMDL, detect changes, increase master mirror's version by 1
>   - run the crawler, check for the changes, align that mirror's version
> with the
>     master mirror one
>   - be able to see the difference between two versions
>   - be able to crawl a mirror only for the difference between the version
> it is
>     at and the version the master mirror is at
>   note: we might still want to run a full crawl once in a while (daily?
>   bi-daily?)
>
>
Those last couple of steps would be very useful in cutting down load from
mirrors running rsync against enchillada but only needing a couple of
packages in updates since the last time they did it.


> This list of ideas is more a long term todo list, not something we would
> want to
> have working for pushing MirrorManager2 to prod.
>
>
> Thoughts? Agreements? Disagreements?
>
> Thanks,
> Pierre
>
> _______________________________________________
> infrastructure mailing list
> infrastructure at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure
>



-- 
Stephen J Smoogen.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150211/39ed07bd/attachment.html>