[gopher] Spidering the gopherspace

[Cameron Kaiser]

Mateusz,

> I am spidering the gopherspace since a few days, collecting things for a 
> new project I will publish soon.
> 
> If you see 178.170.108.36 hammering your gopher server, that would be me.
> 
> Do not hesitate complaining if you notice I'm too hard on you, I will 
> try to spare your server then (I already see that sdf.org have 
> blacklisted me).

FWIW, I throttled several minutes between requests to the same IP (or would
find another to visit in the meantime) and I always honour robots.txt if it
can be fetched (and cache it).

However, since V-2 only fetches menus and has a well-known reverse DNS, I
imagine sites are a little friendlier to me.

-- 
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser at floodgap.com
-- All things are possible, except skiing through a revolving door. -----------