[gopher] gopher proxies

[Wolfgang Faust]

Generally, I think, proxies ought to simply deny access from all
user-agents for the proxied pages. If the owner of the gopher site wishes
to make their site HTTP-indexable, they can set up their own web server.
This also prevents search engines from thinking that proxies contain
duplicate content.
On Jul 23, 2013 5:00 AM, "Jacob Dahl Pind" <rachael at telefisk.org> wrote:

>
> I would like to suggest a means grande gopher site, a way of controling
> access through the proxies.
>
> with http we have the robots.txt file, and the gopher proxies have one
> such file, but as the individual gopher sites are presenteted as pages
> beloning to proxy site, we have to kindly ask the manageres of those pages
> to include out pages there to limited access from roaming web crawler bots.
>
> We need a standalized way of placing a .robots.txt file in the root of out
> site, and say once a week, month such a file would be automatical
> incorperated into the robots.txt on the proxy site, so we need a simple
> parse, and a set of rules as to just how many linies we can have in that
> file.
>
> at the same time, we could need some way of getting a bit header
> infomation also, useragent string is the one we need for this, if just the
> proxy site would transmite the useragent of every 50th access by the means
> of a sperat access to a comon select with the useragent appeded by the
> means of the web get hack.
>
> Say access to proxy?useragent=<bingbot,**googlebot, fake german search
> enginen etc etc>
>
> Could all be done over the caps.txt file, by means or robot= lines and the
> webget argument could be pass to that file also.
>
> --
> Jacob Dahl Pind | telefisk.org | fidonet 2:230/38.8
>
>
> ______________________________**_________________
> Gopher-Project mailing list
> Gopher-Project at lists.alioth.**debian.org<Gopher-Project at lists.alioth.debian.org>
> http://lists.alioth.debian.**org/cgi-bin/mailman/listinfo/**gopher-project<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/gopher-project/attachments/20130723/7283605f/attachment.html>