Freeze Break Request: add robots.txt to public facing taskotron instances

[Pierre-Yves Chibon]

On Wed, Aug 05, 2015 at 11:53:49AM -0600, Tim Flink wrote:
> For some reason, yahoo's bot has been hitting our taskotron instance
> really hard all of a sudden - slowing it down enough to trigger nagios
> alerts twice this week.
> 
> There is a robots.txt served by the buildmaster instance but since that
> isn't at the webroot in our setup, bots may not read it.
> 
> I'd like to add a robots.txt to the public facing taskotron instances
> so that we can avoid this in the future. Rendered content of the
> robots.txt is at the end of this email.
> 
> I've already deployed the file to taskotron-dev which can be seen at:
> http://taskotron-dev.fedoraproject.org/robots.txt
> 
> Proposed content would be at:
> http://taskotron.stg.fedoraproject.org/robots.txt
> http://taskotron.fedoraproject.org/robots.txt
> 
> Tim
> 
> 
> User-agent: *
> Disallow: /taskmaster/waterfall
> Disallow: /taskmaster/builders
> Disallow: /taskmaster/changes
> Disallow: /taskmaster/buildslaves
> Disallow: /taskmaster/schedulers
> Disallow: /taskmaster/one_line_per_build
> Disallow: /taskmaster/grid
> Disallow: /taskmaster/tgrid
> Disallow: /taskmaster/json

+1 for me


Pierre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150805/fd1596de/attachment-0001.sig>