[gopher] GopherMole - a gopher media crawler

[Mateusz Viste]

On 01/13/2015 12:10 PM, James Mills wrote:
> Q: What languages and libraries did you use to implement Gophermole?

ANSI C89. No libraries, besides libc and a SQL connector.

> I assume the "sites to crawl" is some kind of FIFO Queue?

It's a queue, yes, but not FIFO. It's randomized, but with some weights 
for servers that weren't ever polled (so discovery of new servers is 
quick), and then priorities on sites that haven't been revalidated for 
the longest time.

> Do you run the spider(s) on some cloud?

Nope, all the magic happens on my server.

Mateusz