[gopher] gopher proxies

[Cameron Kaiser]

> > I would like to suggest a means grande gopher site, a way of controling
> > access through the proxies.
> >
> > with http we have the robots.txt file, and the gopher proxies have one
> > such file, but as the individual gopher sites are presenteted as pages
> > beloning to proxy site, we have to kindly ask the manageres of those pages
> > to include out pages there to limited access from roaming web crawler bots.
> >
> > We need a standalized way of placing a .robots.txt file in the root of out
> > site, and say once a week, month such a file would be automatical
> > incorperated into the robots.txt on the proxy site, so we need a simple
> > parse, and a set of rules as to just how many linies we can have in that
> > file.
>
> Generally, I think, proxies ought to simply deny access from all
> user-agents for the proxied pages. If the owner of the gopher site wishes
> to make their site HTTP-indexable, they can set up their own web server.
> This also prevents search engines from thinking that proxies contain
> duplicate content.

I think these are good thoughts to consider, but the problem is that most of
the "overly permissive" proxies are on servers run by folks who either aren't
on these lists, or disagree with the premise that the content should not be
indexed. Realistically, asking them to do more work won't fly, unless there
were a concerted cooperative effort to simply drop requests from them on the
floor and refuse to serve their proxies data until changes were made.

Personally, I have no problem implementing a reasonable approach along
these lines, though I suspect that most of the issues aren't coming from my
proxy. If you know differently, please send me an E-mail offlist. Probably
the same for Brian and pongonova.

-- 
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser at floodgap.com
-- armadillo, n. the act of providing weapons to a Spanish pickle. ------------