Why is fedoraproject.org only indexed by Google?

[Kevin Fenzi]

On Tue, 27 Jan 2015 10:29:23 -0600
Michael Cronenworth <mike at cchtml.com> wrote:

> On 01/26/2015 09:46 AM, Kevin Fenzi wrote:
> > I think we added the Crawl-delay several years ago when we were
> > having storage issues. We could definitely try removing it and see
> > if things improve.
> 
> 10 seconds may be on the high side, but you may still want to keep it
> and lower it to 1 to 5 seconds. Bing Bot hits my sites like it's
> running a benchmark.

Ok. I changed it to 1. Should sync out over the next little while... 

We can see if it makes any difference. 

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20150128/e543ae14/attachment.sig>