[gopher] Spidering teh gopherspace

[John Goerzen]

Kim Holviala wrote:
> Yeah, me again.
> 
> Since I got the server done I promised to start writing clients. But 
> instead, I got sidetracked and decided to code a search engine first...
> 
> I got a basic spider up and running (pure POSIX C again) in a couple of 
> hours and succesfully tested it against my own server. Which then first 
> throttled me (1-second delay before reply), and after I cleared the 
> sessions inetd decided that it had enough and kicked me totally out.

You may be interested in

http://git.complete.org/z-old/gopherbot/

though my goal was somewhat different; I aimed to archive all of
gopherspace, not search it.  I believe it was actually successful; I
burned some DVDs and shared them with a few people on this mailing list
at the time.  IIRC it was about 5 DVDs.

-- John