docs.fp.o Search [Was: Re: CMS]

Eric "Sparks" Christensen sparks at fedoraproject.org
Sat Jun 5 09:55:05 UTC 2010


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/05/2010 03:14 AM, Ruediger Landmann wrote:
> One of the new features of Publican 2.0 that I haven't mentioned yet is 
> that it creates an XML sitemap for search engine bots to crawl. You can 
> find d.fp.o.'s sitemap here:
> 
> http://docs.fedoraproject.org/Sitemap

Awesome.

> 
> I've fed this to Google, Yahoo, and Bing, and they're all slowly 
> re-indexing the site. The map now contains a little over 2,000 URLs and 
> at the time of writing, Google has crawled about 350 of them.

I know that Google has some algorithm that figures out how often your
site changes and then crawls more or less frequently.  Not sure if we
could work with Google on scheduling this more or less around the time
of a release.  Of course I'm guessing that we won't have this big
re-structuring next time, either.

> 
> The dilemma we face is the decision of when to turn off the 404 
> redirect. For the sake of all the existing links scattered around the 
> net (both on the Fedora Project site and off it), we'd want to postpone 
> this as far as possible. On the other hand, any bot attempting to verify 
> that link gets a page served up and probably concludes that the link is 
> valid; I suspect that if these links 404ed, they'd start to evaporate 
> from search results.

Isn't there a type of redirect (302?) that tells you that you are being
redirected so you don't think the URL is valid?

> 
> Given that existing links around the net are pointing to (at most 
> recent) the F12 versions of docs, there will be no need to keep the 404 
> redirect in place past October; however, if we want to start allowing 
> dead links to 404 out rather than poison search results, maybe we should 
> bring that date forward? The sooner we do this, the sooner search will 
> start working properly...

Yeah, the sooner the better, IMO.

> 
> Cheers
> Rudi
> 

- --Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJMCh74AAoJEDbiLlqcYamx570P/igKV+7uS9AlIYWIYbSgTANw
scl/oVSzcIvj3A6O9ddFgPZLBg7fHkXXCh8TtExttDC71Xl+nromHQH4B07H35+g
/35qnojQF1RZhDcBhzYbvlCbT5wNvJZ3Tg1/9yGixfgArPuwfB5UlDKWHhbPeMBF
15Hz2vEd140NCDcHZphZy8CZgvmFD655GfkM6bZ3J0CXfFTxgv20qI7AdjT+H/Fu
D31VZxnxNsIB4TwozgWkzAdyFRQ5DVJjsrYihUt+D3RVDXZ9U9eFaTU7gtNv17sz
bpkSati7R6pJ40wpHG5tYnlLSaSEmk0qIkyzopl90ZQ80WS7aeS9fdHZGvk9YZqY
YFk3fY/pH7KGzSIxN48y1Ai3RSR2XGsBoM70TVsHzaL498UY2gu8NzgsfLKsz5Qo
4NQJQJhm3+t2G4ONuAlA2l6fglppyRqp65lJNPTp/iNPnx+y9VcwL7hcZL8wUSE+
jZfSgVz04xXlxrK04wfrTOcSOxRmf678fMdQSLuOG/WXgrWlZ8gaAwa+z8vcCo/U
H/xab3DqQ2zV562fkRnj5ALhksbz0hW6ieK1y7cAMCR2CK6o0ZyFCjfQS3OFF6ZC
rVimBtAHrkM6mQEJutzcUeIJk/H5iFT9hol06YQ64Ftq4SZX47IKCit8CSBKgvvb
yivlbYiORTmnfv69cd4U
=B189
-----END PGP SIGNATURE-----


More information about the docs mailing list