Why is fedoraproject.org only indexed by Google?

Brandon Vincent Brandon.Vincent at asu.edu
Mon Jan 26 14:53:27 UTC 2015


On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer at redhat.com> wrote:
> Any idea why?

https://lists.fedoraproject.org/robots.txt

User-agent: *
Crawl-delay: 10

>From Bing, "This means the higher your crawl delay is, the fewer pages
BingBot will crawl. As crawling fewer pages may result in getting less
content indexed, we usually do not recommend it, although we also
understand that different web sites may have different bandwidth
constraints." [1].

[1] http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/

Brandon Vincent


More information about the devel mailing list