[Shootout-list] Directions of various benchmarks

[Jon Harrop]

On Thursday 19 May 2005 17:43, Brent Fulgham wrote:
> On the other hand, we have encouraged Jon and others
> to reduce the scope of tests to stay within the 100
> LOC for proposed new test.
>
> I think this has worked fairly well.

I agree entirely. You can actually get a lot done in 100 LOC, even in C++. 
Indeed, most programs written by scientists are not much longer.

> But seriously -- should we require all benchmark
> proposals state the algorithm using some kind of
> formal language?

I think we should not require that all benchmark proposals state the 
algorithm. Instead, we should require that all benchmarks state the problem 
to be solved and, for non-trivial benchmarks, perhaps an algorithm that can 
be used to solve it or an example program. In particular, the benchmark 
should never be able to state a "quick route" to the answer (e.g. for 
checking), or the benchmark is flawed.

In terms of a formal language, a mathematical expression will probably suffice 
in many cases (e.g. implicitode, spectralnorm, nbody) but may be subject to 
problems from finite-precision arithmetic.

For example, the ray tracer should require that the <100 LOC self-contained 
program reproduces the same output image as the original OCaml, C and C++ 
programs. Any data structures and algorithms can be used. For each language, 
the shootout should contain the fastest program <100 LOC or, if there are 
none, the shortest program >100 LOC.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists