Re: Freeze Break Request - reduce number of auto-disabled mirrors

[Ralph Bean]

On Wed, May 13, 2015 at 02:17:24PM +0200, Adrian Reber wrote:
>  The new MM2 crawler disables mirrors which have failed to be
>  successfully crawled for 4 consecutive crawls. This seems to be a good
>  idea to reduce the total number of crawls by removing mirrors which are
>  just too slow. Unfortunately the current default timeout of 2 hours is
>  not enough. Especially for mirrors which mirror more than one category
>  as the timeout is per host and not per category. The problem is also not
>  network bound but it seems to be related to two crawlers updating the
>  directories of all mirrors on the same database at the same time. To
>  workaround this timeout problem I am now starting the crawler on the
>  second crawler 3 hours later and I have also increased the timeout from
>  2 hours to 3 hours. Additionally a small fix is included to also crawl
>  the last mirror in the database which was ignored until now.
>  After this is applied I would also re-enable the auto-disabled hosts
>  in the database.
>  
>  Can I get two +1 for these changes? 
This looks good and sane to me.  +1.

If things go wrong, we can easily back out by just reverting the
commits.