[gopher] Robots.txt

[Nick Matavka]

Could I have the spec to, or perhaps somebody's explanation of, the
robots.txt file?  This will help me to explain it in my RFC.

-- 
       /^\/^\
       \----|
   _---'---~~~~-_
    ~~~|~~L~|~~~~
       (/_  /~~--
     \~ \  /  /~
   __~\  ~ /   ~~----,
   \    | |       /  \
   /|   |/       |    |
   | | | o  o     /~   |
 _-~_  |        ||  \  /
(// )) | o  o    \\---'
//_- |  |          \
//   |____|\______\__\
~      |   / |    |
       |_ /   \ _|
     /~___|  /____\