[gopher] Spidering the gopherspace

[Mateusz Viste]

Hi,

Indeed, they use Gophernicus (at least that's what they say on the main 
page), but the issue I have is not an applicative blacklist, since they 
don't even answer with a SYN/ACK to me anymore. So it's rather some kind 
of ACL on a firewall... I guess I must have been appearing too much in 
their logs.

cheers,
Mateusz



On 12/29/2014 09:31 AM, Kim Holviala wrote:
>> On 28 Dec 2014, at 21:52, Mateusz Viste <mateusz at viste.fr> wrote:
>>
>> I am spidering the gopherspace since a few days, collecting things for a new project I will publish soon.
>>
>> If you see 178.170.108.36 hammering your gopher server, that would be me.
>>
>> Do not hesitate complaining if you notice I'm too hard on you, I will try to spare your server then (I already see that sdf.org have blacklisted me).
>
> As far as I remember SDF is running my Gophernicus which has automatic throttling. The default settings are 4096 hits or 4 megabytes of transfer until you are throttled, and throttling in this case means adding 1 second to each request before answering. If you do 8192 hits you’ll get a 2-second punishment etc etc. Session expires in 30 minutes, so to get around the limitations just crawl ~4000 requests and stop for half an hour.
>
> It’s possible that SDF has different settings, but those are the defaults. Gophernicus doesn’t implement complete blacklisting but it’s of course possible to do that on kernel level.
>
>
>
> - Kim