[change req] Allow fedorahosted robots.txt to only crawl /wiki/*

[Ricky Elrod]

We've been seeing load spikes on hostedXX, following
df7e8578432b224d9576dc8359f0729763861526. This semi-reverts that commit
and only allows /wiki/* to be crawled.

diff --git a/configs/web/fedorahosted.org/fedorahosted-robots.txt
b/configs/web/fedorahosted.org/fedorahosted-robots.txt
index cd572f8..7782677 100644
--- a/configs/web/fedorahosted.org/fedorahosted-robots.txt
+++ b/configs/web/fedorahosted.org/fedorahosted-robots.txt
@@ -1,5 +1,5 @@
 User-agent: *
-Disallow: /*/browser
-Disallow: /*/search
+Allow: /wiki/*
+Disallow: /
 user-agent: AhrefsBot
 disallow: /

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20121230/3fe1df84/attachment.sig>