[gopher] gopher proxies

[Jacob Dahl Pind]

I would like to suggest a means grande gopher site, a way of controling 
access through the proxies.

with http we have the robots.txt file, and the gopher proxies have one 
such file, but as the individual gopher sites are presenteted as pages 
beloning to proxy site, we have to kindly ask the manageres of those pages 
to include out pages there to limited access from roaming web crawler bots.

We need a standalized way of placing a .robots.txt file in the root of out 
site, and say once a week, month such a file would be automatical 
incorperated into the robots.txt on the proxy site, so we need a simple 
parse, and a set of rules as to just how many linies we can have in that 
file.

at the same time, we could need some way of getting a bit header 
infomation also, useragent string is the one we need for this, if just the 
proxy site would transmite the useragent of every 50th access by the means 
of a sperat access to a comon select with the useragent appeded by the 
means of the web get hack.

Say access to proxy?useragent=<bingbot,googlebot, fake german search 
enginen etc etc>

Could all be done over the caps.txt file, by means or robot= lines and the 
webget argument could be pass to that file also.

--
Jacob Dahl Pind | telefisk.org | fidonet 2:230/38.8