[Shootout-list] Simple tests

[Jon Harrop]

On Saturday 18 June 2005 10:19, Einar Karttunen wrote:
> Hello
>
> A simple test has typically a LOC of <20 in most languages and takes one
> simple parameter for input and prints out simple output. What does this
> mean?
>
> The test is easy to implement and understand.
> It is easy to analyze what different implementations do and detect if
> someone is not following the spec. With the complex benchmarks the
> solutions may diverge (e.g. raytracer with odd N) for values we don't
> test.
>
> Most of the time the best performing solutions share the same
> bottleneck.
>
> It is fairly easy to understand what is tested.

I agree with everything you've said up to here.

> The test can be precomputed. One can create an array which contains the
> answers for various N.

No, if the input to the test is a single 32-bit int then your array would take 
up 72Gb. Although that is theoretically ok, it isn't going to happen in the 
real world.

> But on the other hand in the complex tests one can 
> provide precomputed answers for 'interesting cases' (namely the ones we
> test). I don't think this is a problem in practice if we use common
> sense and refrain from the "do the same thing N times".

This clearly already is a problem in practice and we already refrain from the 
"do the same thing N times".

> For these reasons I think we should keep the simple tests in the
> Shootout. Complex tests have their place too, but I think that
> discarding the simple ones would lose much of the original spirit.

Absolutely. I agree entirely. But Skaller's suggestion that I am advocating 
doesn't mean we can't have simple tests, it just means we need to have other 
simple tests.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists