On May 16, 2012, at 17:50 , Damien Carol wrote: > Thanks for the source code, I will try to use something similar. > If your server is a finger service, I will remove this address of the crawler. The right way is to just ignore robots.txt for ports 79. The gopher request (and crawling) should be carried on normally. - Kim