[Shootout-list] Stuff (benchmark design)

[Bengt Kleberg]

skaller wrote:
> On Wed, 2005-04-27 at 16:09, Bengt Kleberg wrote:
> 
> 
>>if the design of the benchmark allows something, then this solution 
>>should be allowed to be the ''real'' solution. it should not be judged 
>>unfit, becuse it is different to how other languages have solved the 
>>design requierments.
> 
> 
> Which is of course impossible in principle for 
> any 'same way' tests because the whole purpose
> of the Shootout is to measure the differences
> between languages that necessarily do NOT do
> things the 'same way'.


i apologise, but i do not follow your reasoning here.
do you mean that it is impossible (in principle) to allow a solution to 
be the ''real'' solution even if it obeys the constraints of the 
benchmark, but does something different than the other solutions?
why?


> So benchmarks should not be allowed to 'allow' or 'disallow'
> anything. They should simply specify the inputs and outputs
> required and perhaps constraints on resource usage:
> what happens in the 'black box' in between is up to the
> test implementor and should be judged entirely on the
> basis of whether it produces the correct result (whilst
> limiting its resource usage as required).
> 
> This kind of condition is easy to measure mechanically
> without too much argument.

i see a danger here. the shootout exels in very few inputs. it is even a 
''pride'' thing. (narrow focus, etc). take  fannkuch. input is 7, 8 and 9.
Correct output N = 7 is:
Pfannkuchen(7) = 16

i could make a very simple black box that handles input/output correctly 
amd that is about as fast as ''hello, world''.

so only input/output and black box is not sufficient test design. imho.


bengt