MM2 status

[Adrian Reber]

On Tue, Apr 21, 2015 at 07:33:27AM +0200, Pierre-Yves Chibon wrote:
> > > > ! Crons: iirc there are still a couple of crons that are not running
> > > > properly and returning errors
> > 
> > Which ones?
> 
> From this night:
> ----
> Subject: Cron <root at mm-crawler01> /usr/bin/mm2_crawler --threads 50 > /dev/null
> 2>&1
> 
> /bin/sh: line 1: 18523 Killed                  /usr/bin/mm2_crawler --threads 50
> > /dev/null 2>&1

This is OOM. One crawler did not finish within 12 hours. The second
crawler started and Linux killed one of them. The atomic tree is really
killing the crawler with its over 370000 files. If a mirror has http as the
only protocol and is geographically far and slow it is not possible to
crawl in a reasonable time. The atomic tree needs a different setup/rule
how to be crawled.

> ----
> Subject: Cron <root at mm-backend01> cd /usr/share/mirrormanager2 && /usr/bin/mm2_get_internet2_netblocks
>         /var/lib/mirrormanager/internet2_netblocks.txt
> 
> gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file
> short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.

[...]

> gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file
> short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
> unable to retrieve netblock list.

I think I see these also on my MM1 installation. Probably easy to fix.

> These are the only two I can see right now.
> 
> > > So, that would be another, higher level test: 
> > > 
> > > Make staging scan directories for latest content. 
> > > Make staging make a pkl based on that. 
> > > Check it against the prod one
> > 
> > I am not sure if the resulting pickles can be that easily compared as
> > there have been changes all around MM2 which might return slightly
> > different results which are nevertheless both valid results.
> 
> I don't think we changed the pickles themselves, for example mirrorlist barely
> changed and is currently working with MM1's pickle.

I was talking about the content of the pickles. As the crawler is making
slightly different decisions and umdl maybe also I do not think we can
compare the content of the pickles.

		Adrian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 811 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150421/9e598bf2/attachment.sig>