[gopher] Strange implementations of "Missing file"

[Cameron Kaiser]

> > Thanks for the source code, I will try to use something similar.
> > If your server is a finger service, I will remove this address of the
> > crawler.
> 
> The right way is to just ignore robots.txt for ports 79. The gopher request
> (and crawling) should be carried on normally.

I agree. This is exactly how V-2 does it.

-- 
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser at floodgap.com
-- I know we can make it together! -- "Shogo" ---------------------------------