Why is fedoraproject.org only indexed by Google?

[Kevin Fenzi]

On Mon, 26 Jan 2015 07:53:27 -0700
Brandon Vincent <Brandon.Vincent at asu.edu> wrote:

> On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer at redhat.com>
> wrote:
> > Any idea why?
> 
> https://lists.fedoraproject.org/robots.txt
> 
> User-agent: *
> Crawl-delay: 10
> 
> From Bing, "This means the higher your crawl delay is, the fewer pages
> BingBot will crawl. As crawling fewer pages may result in getting less
> content indexed, we usually do not recommend it, although we also
> understand that different web sites may have different bandwidth
> constraints." [1].
> 
> [1]
> http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/

Not sure that explains why there are no results at all though. 

I think we added the Crawl-delay several years ago when we were having
storage issues. We could definitely try removing it and see if things
improve. 

kevin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20150126/e39b107d/attachment.sig>