[gopher] Spidering the gopherspace

[James Mills]

On Mon, Dec 29, 2014 at 9:42 AM, Cameron Kaiser <spectre at floodgap.com>
wrote:
>
> FWIW, I throttled several minutes between requests to the same IP (or would
> find another to visit in the meantime) and I always honour robots.txt if it
> can be fetched (and cache it).
>
> However, since V-2 only fetches menus and has a well-known reverse DNS, I
> imagine sites are a little friendlier to me.
>

Good points for anyone wanting to experiment with
crawling and full-text search engines (i.e: me).

I'll try to keep this in mind :)

@Cameron: How do you presently find new Gopher servers? Manually via email
and through discovery of other Gopher servers?

cheers
James


James Mills / prologic

E: prologic at shortcircuit.net.au
W: prologic.shortcircuit.net.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/gopher-project/attachments/20141229/15384c60/attachment.html>