[Shootout-list] Directions of various benchmarks

[Brent Fulgham]

--- skaller <skaller@users.sourceforge.net> wrote:
> Yes and so a test requiring PCRE is worthless:
> languages using it are using the same code and
> those not doing so are all failing.

This should not be the case.  Regular expression
tests should allow any implementation that
solves the problem.

> Same way tests are semantic nonsense. The only way
> to do something 'the same way as this Java program' 
> -- which is the specification of some tests right
> at the moment -- is obviously to actually use THAT 
> Java program.

I guess, but let's not act like Lawyers and fight
over the meaning of specific words.  The phrases
you are alluding to were an attempt to describe what
a particular test is supposed to do in some concise
way.  I'm not trying to infer from a statement that
"the test should work like this Java program" means
that any successful implementation must be run on
a virtual machine.  Rather, the point is that the
algorithm should work like so, and it should make
use of certain constructs, and that it should not
take certain obvious shortcuts that negate the point
of the test.

> How the heck can you judge if a solution in C or
> Ocaml or Haskell is 'the same way' as a Java program
> when one language is imperative with variables and 
> the other is functional?

I agree that it is not always easy.  But the point
is not to exclude one langauge or another, but to
force the implementations to do the same kind of work
that we are measuring.

I'm sure you can agree that we can't draw a conclusion
from a C program that iterates through the same
structure N times when compared with a Haskell program
(where the compiler wisely skips the useless
iterations).

> Similarly 'concurrency' tests are in question,
> because a requirement to use Posix thread is absurd:
> should we really measure OS thread handling 
> limitations? 

No.  And this limitation has been gone for quite
some time.  Erlang, MzScheme, and others all use
their own internal threading constructs.

> A similar problem with 'repeat the same code' and
> Haskell cleverly 'lazifying' the repitition away. 
> Again a test designed in with a dumb imperative 
> programming model in mind can't be expected
> to mean much for an advanced system like Haskell.

Agreed.  That's precisely why Isaac, Einar, and I
have been replacing these kinds of tests.

-Brent