[gopher] Gopher Proxy & Archive robots.txt

Thu Dec 8 16:07:34 UTC 2011

Very nice!

On 12/08/2011 08:38 AM, Wolfgang F. wrote:
> The gopher proxy & archive ( 
> http://gopher-project:gpa-test-675@gpa.internetearth.com/ ) should now 
> pay attention to robots.txt exclusion rules. I haven't tested this 
> extensively, but it seems to work. Does anybody know of any gopher 
> pages that have been blocked by robots.txt so I can test?
>
> Also, the spider will begin crawling gopherspace shortly.
>
>
> _______________________________________________
> Gopher-Project mailing list
> Gopher-Project at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/gopher-project/attachments/20111208/caa37ae8/attachment.html>