[release] MirrorManager2: 0.2.0

[Adrian Reber]

On Fri, Jun 05, 2015 at 05:09:06PM +0200, Pierre-Yves Chibon wrote:
> Good morning everyone,
> 
> Here is another release of the day, MirrorManager2: 0.2.0
> 
> And here is the changelog:
> * Fri Jun 05 2015 Pierre-Yves Chibon <pingou at pingoured.fr> - 0.2.0-1
> - Update to 0.2.0
> - Include the background header file in MM2 itself (Adrian Reber)
> - Support always update hosts which are unreachable in the crawler (Adrian
>   Reber)
> - Adjust the spec file to the systemd packaging guidelines for Fedora
> - Multiple improvements to the crawler, including a start of a canary mode
>   (Adrian Reber)
> - Offer possibility to sort by product, bringing back MM1 behavior (Adrian
>   Reber)
> - Couple of UI fixes about who is allowed to access what
> - Fix peer ASNs (in the same spirit, who can access)
> - Create noauthed master for mirror publiclist so that it can be cached in
>   memcachd (Patrick Uiterwijk)

I think this should read varnish and not memcachd. I think it still uses
the 'default' cache age of 120 seconds. The publiclist used to be
created once or twice a day so that we could easily keep the cache much
longer than the current 120 seconds. I think 12 hours would be no
problem, as this list is 'only' used as an overview and independent of
the results of the actual yum/dny mirrorlist/metalink.

> - Fix the report_mirror to correctly catch the xmlrpclib.ProtocolError
> - Add a new utility script to upgrade repo from -alpha or -beta to release
> - Adjust the logrotate configuration to fix the permission denied error
> - Create 2 API endpoints, one for zodbot's .mirroradmin and one for nagios
> 
> Note: this is currently only in *staging*, we did not feel like pushing this to
> prod on a Friday afternoon :)
> 
> We'll likely push on Monday though, assuming nothing burns down over the
> week-end.

I tested the new canary and repodata crawl mode and I was able to crawl
all mirrors in a much short time:

$ time mm2_crawler --repodata -c ~/mirrormanager2.cfg --threads 75 --timeout 30
real    36m38.639s
user    27m20.689s
sys     3m13.829s

$ time mm2_crawler --canary -c ~/mirrormanager2.cfg --threads 75 --timeout 30
real    6m33.624s
user    0m16.061s
sys     0m2.279s

Canary mode only checks if at least one base URL of all categories on
the mirror works. If no URL works the whole mirror is marked as not
being update. This also means that the mirror will not be re-enabled
until a full crawl or successful report_mirror run. This behaviour that
canary mode disables the mirror until a full crawl or report_mirror run
could be changed but would require database changes and therefore we
decided to support this form of canary mode as a first step.

Repodata mode crawls all repodata directories and only marks those
directories as being up to date or not. A mirror will not be completely
disabled in repodata mode in contrast to canary mode.

As canary mode seems to finish in under 10 minutes we can now think
about using full crawls and canary or repodata mode in combination to
get better overview about the availability and the status of the
mirrors.

		Adrian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 811 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150607/d9b4b1d4/attachment.sig>