On May 23, 2012, at 23:17 , Nick Matavka wrote: > Could I have the spec to, or perhaps somebody's explanation of, the > robots.txt file? This will help me to explain it in my RFC. http://www.robotstxt.org/orig.html - Kim