[Shootout-list] Stuff (benchmark design)

[skaller]

On Wed, 2005-04-27 at 16:09, Bengt Kleberg wrote:

> if the design of the benchmark allows something, then this solution 
> should be allowed to be the ''real'' solution. it should not be judged 
> unfit, becuse it is different to how other languages have solved the 
> design requierments.

Which is of course impossible in principle for 
any 'same way' tests because the whole purpose
of the Shootout is to measure the differences
between languages that necessarily do NOT do
things the 'same way'.

So benchmarks should not be allowed to 'allow' or 'disallow'
anything. They should simply specify the inputs and outputs
required and perhaps constraints on resource usage:
what happens in the 'black box' in between is up to the
test implementor and should be judged entirely on the
basis of whether it produces the correct result (whilst
limiting its resource usage as required).

This kind of condition is easy to measure mechanically
without too much argument.


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net