[Shootout-list] Directions of various benchmarks

[John Skaller]

On Thu, 2005-05-19 at 13:22 +0200, Bengt Kleberg wrote:
> Einar Karttunen wrote:
> ...deleted
> > You seem to advocate tests as black boxes "do whatever you want inside
> > as long as they satisfy X, Y and Z". I would like an approach where the
> > what and how is in the specification and can be discussed on the list
> > if one is not sure whether some solution fulfills it.
> 
> this approach (what and how is in the specification) makes it simpler to 
> design tests, too. this is worth remembering.

Yes but this is the status quo, and resulted in a test Felix was
doing well in being disallowed arbitrarily and unfairly,  and half
a dozen other tests being allowed to stand even though they're
demonstrably cheating: regexp is one, another (possibly deprecated now)
required maxiumum 4K I/O buffer but the Ocaml code explicitly set
the buffer to 8K .. in one case (Python was it?) performance
was improved by simply not running destructors/garbage collection
at the end of the test .. but that code wasn't accepted ..

.. Ocaml does this by default, Felix 1.0.23 does too.  

-- 
John Skaller, skaller at users.sf.net
PO Box 401 Glebe, NSW 2037, Australia Ph:61-2-96600850 
Download Felix here: http://felix.sf.net