robots.txt

[JD]

SOme websites which provide files for download, refuse wget
They send a file robots.txt instead.
In that file I see:
User-agent: *
Disallow: /
Allow: /lang/*
Allow: /$

So, I ask if anyone knows how to use wget to get those files without the use
of FF ?

[Dave Stevens]

Quoting jd1008 :
> SOme websites which provide files for download, refuse wget
> They send a file robots.txt instead.
> In that file I see:
> User-agent: *
> Disallow: /
> Allow: /lang/*
> Allow: /$
>
> So, I ask if anyone knows how to use wget to get those files without the use
> of FF ?

wget -e robots=off --wait 0.25 http://your.site.here

like this?

d
> -- 
> users mailing list
> [e-mail]
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org