[gopher] Whole lotta hits

[Jacob Dahl Pind]

On Tue, 13 Sep 2011, Jeff W wrote:

> Anyone else seeing unusually large numbers of hits on their servers
> today?  Normally I get like a dozen or so; yesterday: 491, today:160.
> Most are from various eastern european countries or indirect via what
> looks to be some sort of WWW-gopher gateway at bytemark7.spotonnet.co.uk.
>> From the log it looks like intentional hammering to try to crash the
> server (it didn't).
>
> Jeff

I think it migth be baidubot, its quiet aggressiv in its crawling this 
days, it crawlers my http sites every day, with 30-40sec between access.
But google and bing are also crawling through the gopherproxy.org at 
bytemark7.spotonnet.co.uk.

As we cant ratelimit, or any other robots.txt things when those web 
crawlers are hitting us through those proxies, I have opped for simply 
redirecting access from those proxies to another selector.

is in the TELEFISK_FILTER ifdef in this patch

gopher://telefisk.org/0/gopher/gophernicus-1.2-beta-telefisk.diff

Regards

--
Jacob Dahl Pind | telefisk.org | fidonet 2:230/38.8