bedouglas at earthlink.net
Thu Jun 22 16:32:25 UTC 2006
this question is for the wget guru!
i'm testing wget on a test site.. i'm using the recursive function of wget
to crawl through a portion of the site...
it appears that wget is hitting a link within the crawl that's causing it to
begin to crawl through the section of the site again...
i know wget isn't as robust as nutch, but can someone tell me if wget keeps
a track of the URLs that it's bben through so it doesn't repeat/get stuck in
a never ending processs...
i haven't run across anything in the docs that seems to speak to this
More information about the users