[gopher] Robots.txt
Nuno J. Silva
nunojsilva at ist.utl.pt
Wed May 23 20:25:15 UTC 2012
On 2012-05-23, Kim Holviala wrote:
> On May 23, 2012, at 23:17 , Nick Matavka wrote:
>
>> Could I have the spec to, or perhaps somebody's explanation of, the
>> robots.txt file? This will help me to explain it in my RFC.
>
> http://www.robotstxt.org/orig.html
(AFAIK, the gopher crawlers out there which respect robots.txt, and the
server administrators who have such files, are following the same format
used in the web.)
--
Nuno J. Silva (aka njsg)
gopher://sdf-eu.org/1/users/njsg
http://njsg.sdf-eu.org/
More information about the Gopher-Project
mailing list