[gopher] Strange implementations of "Missing file"

[Damien Carol]

Hello everybody,

I'm facing a big pb with my search engine/crawler.

The crawler sends selectors that don't exists sometimes. (Updated / removed
/ don't exists / whatever)

I use to handle error like that :
1) parse response
2) Try to parse as Menu
3) If the first item of the menu is '3' error item type then ERROR

But this method don't works. Mainly because some servers reply very odd
responses

Some old servers sends that :

*BAD:*
==> 0Sorry, but the requested token could not be
found<TAB>Err<TAB>localhost<TAB>70
gopher://wss-ds.no-ip.info:70/0/robots.txt

==> 0'/robots.txt' does not exist<TAB><TAB>error.host<TAB>1
gopher://gdead.berkeley.edu:70/0/robots.txt
gopher://net.bio.net:70/0/robots.txt
gopher://newkraitch.cs.berkeley.edu:70/0/robots.txt
gopher://nemesis.cs.berkeley.edu:70/0/robots.txt
gopher://quix.us:70/0/robots.txt

=> <Empty String>
gopher://sdf.org:79/0/robots.txt

=> finger: /robots.txt: no such user
gopher://holviala.com:79/0/robots.txt


*GOOD:
gopher://gopher.r-36.net:70/0/robots.txt
gopher://jgw.mdns.org:70/0/robots.txt
gopher://grids.be:70/0/robots.txt
gopher://schot.a-eskwadraat.nl:70/0/robots.txt
gopher://www.quux.org:70/0/robots.txt
gopher://dams.zapto.org:70/0/robots.txt
gopher://go.nickshanks.com:70/0/robots.txt*


It's very strange. I can understand the "Empty String" could be an easy
implementation. But others servers sends Menu with 0 items as info...

Does it old implementations ?

-- 
Damien CAROL
gopher://dams.zapto.org/1/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/gopher-project/attachments/20120516/a7908ccd/attachment.html>