<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#ffffff" text="#000000">
    Very nice!<br>
    <br>
    On 12/08/2011 08:38 AM, Wolfgang F. wrote:
    <blockquote
cite="mid:CAJvjBNbY-Dkuks735xWVZRTJ4h8y3zQeE4R35MkVBRpAJtXPmg@mail.gmail.com"
      type="cite">The gopher proxy & archive ( <a
        moz-do-not-send="true"
        href="http://gopher-project:gpa-test-675@gpa.internetearth.com/">http://gopher-project:gpa-test-675@gpa.internetearth.com/</a>
      ) should now pay attention to robots.txt exclusion rules. I
      haven't tested this extensively, but it seems to work. Does
      anybody know of any gopher pages that have been blocked by
      robots.txt so I can test?<br>
      <br>
      Also, the spider will begin crawling gopherspace shortly.<br>
      <pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Gopher-Project mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gopher-Project@lists.alioth.debian.org">Gopher-Project@lists.alioth.debian.org</a>
<a class="moz-txt-link-freetext" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project</a></pre>
    </blockquote>
    <br>
  </body>
</html>