[gopher] Gopher Proxy & Archive robots.txt

Thu Dec 8 14:38:40 UTC 2011

The gopher proxy & archive (
http://gopher-project:gpa-test-675@gpa.internetearth.com/ ) should now pay
attention to robots.txt exclusion rules. I haven't tested this extensively,
but it seems to work. Does anybody know of any gopher pages that have been
blocked by robots.txt so I can test?

Also, the spider will begin crawling gopherspace shortly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/gopher-project/attachments/20111208/4edc563d/attachment.html>