[Shootout-list] Stuff (benchmark design)

[skaller]

On Wed, 2005-04-27 at 20:42, Bengt Kleberg wrote:
> skaller wrote:
> > On Wed, 2005-04-27 at 16:09, Bengt Kleberg wrote:
> > 
> >>if the design of the benchmark allows something, then this solution 
> >>should be allowed to be the ''real'' solution. it should not be judged 
> >>unfit, becuse it is different to how other languages have solved the 
> >>design requierments.
> > 
> > 
> > Which is of course impossible in principle for 
> > any 'same way' tests because the whole purpose
> > of the Shootout is to measure the differences
> > between languages that necessarily do NOT do
> > things the 'same way'.
> 
> 
> i apologise, but i do not follow your reasoning here.
> do you mean that it is impossible (in principle) to allow a solution to 
> be the ''real'' solution even if it obeys the constraints of the 
> benchmark, but does something different than the other solutions?
> why?

No, the complete opposite :)


> > This kind of condition is easy to measure mechanically
> > without too much argument.
> 
> i see a danger here. the shootout exels in very few inputs. it is even a 
> ''pride'' thing. (narrow focus, etc). take  fannkuch. input is 7, 8 and 9.
> Correct output N = 7 is:
> Pfannkuchen(7) = 16
> 
> i could make a very simple black box that handles input/output correctly 
> amd that is about as fast as ''hello, world''.

Yes. A simple lookup table.

> so only input/output and black box is not sufficient test design. imho.

No. You have to fix the problem, and here the problem is with
the test NOT the solution. By my rules, a lookup table
is an *allowed* solution. If you don't like it,
eliminate the problem -- the test specification is 
at fault NOT the solution.

Tests like Ackemann are very useful to a compiler writer
trying to improve the compiler on a fixed piece of code.

This is more or less the same observation Isaac made
about comparing object and non-object solutions within
the same language. That's a sensible test, and so is
ackemann, using Version 1 and Version 2 of the compiler,
to see if Version 2 does any better.

However these tests are totally useless for comparing 
different encodings of the same requirements.

** So don't put these kinds of tests in the Shootout **

Clearly, we want a function which MUST do a lot of work,
and which will accept a lot of distinct inputs.
Ackermann, Tak, etc, don't meet these requirements
so they should be thrown out.

A test which is O(n) in the sum of 10 arguments
would be suitable. So it is quite possible to
devise such tests.

Conway's Life game is an idea. There are LOTS of inputs.
The game was devised precisely because it is almost
impossible to theorise about the result -- you have
to actually calculate it.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net