<div dir="ltr"><br><div>+100 from me.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 16 April 2015 at 14:23, Patrick Uiterwijk <span dir="ltr"><<a href="mailto:puiterwijk@redhat.com" target="_blank">puiterwijk@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">+1 from me, though you might need to add a <Directory /var/www/html> with allow from all.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
----- Original Message -----<br>
> ok. Next thing... I notice that some queries hitting particularly hard<br>
> are from spiders.<br>
><br>
> Back in the puppet days, koji disallowed all spidering.<br>
><br>
> I never have seen useful search results from it.<br>
><br>
> So, I'd like to block all the spiders and see if that helps out load<br>
> issues.<br>
><br>
> robots.txt would have:<br>
><br>
> User-agent: *<br>
> Disallow: /<br>
><br>
> +1s?<br>
><br>
> kevin<br>
> --<br>
> diff --git a/roles/koji_hub/files/kojiweb.conf<br>
> b/roles/koji_hub/files/kojiweb.conf index 86abd2e..8334274 100644<br>
> --- a/roles/koji_hub/files/kojiweb.conf<br>
> +++ b/roles/koji_hub/files/kojiweb.conf<br>
> @@ -6,6 +6,8 @@ KeepAlive On<br>
> Alias /koji "/usr/share/koji-web/scripts/wsgi_publisher.py"<br>
> #(configuration goes in /etc/kojiweb/web.conf)<br>
><br>
> +Alias /robots.txt /var/www/html/robots.txt<br>
> +<br>
> <Directory "/usr/share/koji-web/scripts/"><br>
> Options ExecCGI<br>
> SetHandler wsgi-script<br>
> diff --git a/roles/koji_hub/tasks/main.yml<br>
> b/roles/koji_hub/tasks/main.yml index 0b6cd82..a5fc795 100644<br>
> --- a/roles/koji_hub/tasks/main.yml<br>
> +++ b/roles/koji_hub/tasks/main.yml<br>
> @@ -171,6 +171,13 @@<br>
> notify: restart httpd<br>
> when: env != "staging"<br>
><br>
> +- name: koji robots.txt config<br>
> + copy: src=robots.txt dest=/var/www/html/robots.txt<br>
> + tags:<br>
> + - config<br>
> + - koji_hub<br>
> + notify: restart httpd<br>
> +<br>
> - name: kojira log dir<br>
> file: dest=/var/log/kojira owner=root group=root mode=0750<br>
> state=directory tags:<br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> infrastructure mailing list<br>
> <a href="mailto:infrastructure@lists.fedoraproject.org">infrastructure@lists.fedoraproject.org</a><br>
> <a href="https://admin.fedoraproject.org/mailman/listinfo/infrastructure" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/infrastructure</a><br>
_______________________________________________<br>
infrastructure mailing list<br>
<a href="mailto:infrastructure@lists.fedoraproject.org">infrastructure@lists.fedoraproject.org</a><br>
<a href="https://admin.fedoraproject.org/mailman/listinfo/infrastructure" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/infrastructure</a></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Stephen J Smoogen.<br><br></div></div>
</div>