Why is fedoraproject.org only indexed by Google?

[Petr Spacek]

On 26.1.2015 16:46, Kevin Fenzi wrote:
> On Mon, 26 Jan 2015 07:53:27 -0700 Brandon Vincent
> <Brandon.Vincent at asu.edu> wrote:
> 
>> On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer at redhat.com> 
>> wrote:
>>> Any idea why?
>> 
>> https://lists.fedoraproject.org/robots.txt
>> 
>> User-agent: * Crawl-delay: 10
>> 
>> From Bing, "This means the higher your crawl delay is, the fewer pages 
>> BingBot will crawl. As crawling fewer pages may result in getting less 
>> content indexed, we usually do not recommend it, although we also 
>> understand that different web sites may have different bandwidth 
>> constraints." [1].
>> 
>> [1] 
>> http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/
>
>> 
> Not sure that explains why there are no results at all though.
> 
> I think we added the Crawl-delay several years ago when we were having 
> storage issues. We could definitely try removing it and see if things 
> improve.

Yes please. We definitely should get archives indexed by search engines!

-- 
Petr Spacek  @  Red Hat