Freeze Break Request: add robots.txt to public facing taskotron instances

[Tim Flink]

For some reason, yahoo's bot has been hitting our taskotron instance
really hard all of a sudden - slowing it down enough to trigger nagios
alerts twice this week.

There is a robots.txt served by the buildmaster instance but since that
isn't at the webroot in our setup, bots may not read it.

I'd like to add a robots.txt to the public facing taskotron instances
so that we can avoid this in the future. Rendered content of the
robots.txt is at the end of this email.

I've already deployed the file to taskotron-dev which can be seen at:
http://taskotron-dev.fedoraproject.org/robots.txt

Proposed content would be at:
http://taskotron.stg.fedoraproject.org/robots.txt
http://taskotron.fedoraproject.org/robots.txt

Tim


User-agent: *
Disallow: /taskmaster/waterfall
Disallow: /taskmaster/builders
Disallow: /taskmaster/changes
Disallow: /taskmaster/buildslaves
Disallow: /taskmaster/schedulers
Disallow: /taskmaster/one_line_per_build
Disallow: /taskmaster/grid
Disallow: /taskmaster/tgrid
Disallow: /taskmaster/json
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150805/a42a8f40/attachment-0001.sig>