[gopher] gopher proxies

[Jacob Dahl Pind]

On Tue, 23 Jul 2013, Cameron Kaiser wrote:

> I think these are good thoughts to consider, but the problem is that most of
> the "overly permissive" proxies are on servers run by folks who either aren't
> on these lists, or disagree with the premise that the content should not be
> indexed. Realistically, asking them to do more work won't fly, unless there
> were a concerted cooperative effort to simply drop requests from them on the
> floor and refuse to serve their proxies data until changes were made.

had hoped if there was a sample implementation that parsed, check and 
generated that unifed robots.txt, it would be easy enough for them to 
implement it, but you are right, in the end we are at their mercy and 
blocking is the only option.

Have been using the redirection code I made, that servering up nonsense to
them, I even though about serving them piratebay data, but that would at best
get them delisted from google and bing but I doubt its ether of those that is
causing those none stop crawer storms.

I see the same behaviour on http from:

user-agent: Lightspeed
user-agent: SISTRIX Crawler
user-agent: Baiduspider
user-agent: YandexBot
user-agent: Ezooms
user-agent: Exabot
user-agent: AhrefsBot


--
Jacob Dahl Pind | telefisk.org | fidonet 2:230/38.8