[Shootout-list] Stuff (benchmark design)

[Bengt Kleberg]

skaller wrote:
> On Wed, 2005-04-27 at 20:42, Bengt Kleberg wrote:
> 

...deleted
>>i apologise, but i do not follow your reasoning here.
>>do you mean that it is impossible (in principle) to allow a solution to 
>>be the ''real'' solution even if it obeys the constraints of the 
>>benchmark, but does something different than the other solutions?
>>why?
> 
> 
> No, the complete opposite :)

i apologise. i am so used to beeing told i am wrong that i did not 
understand that you where agreeing with me.
it feels strange. strange, but nice :-)


...deleted
>>so only input/output and black box is not sufficient test design. imho.
> 
> 
> No. You have to fix the problem, and here the problem is with
> the test NOT the solution. By my rules, a lookup table
> is an *allowed* solution. If you don't like it,
> eliminate the problem -- the test specification is 
> at fault NOT the solution.

so far we agree.
however, i think it should be possible to specify more that input/output 
in the test design. if i want to test the limits on the number of 
parallell process/threads/continuations/etc i would like to manadate 
concurrency. even if there is a simpler single thread of control 
solution to the test input/output.

i tried to design a concurrency test without having white-box rules. i 
failed. it would be correct to view my thoughts on benchmark design as a 
croutch for insufficint intelligence. :-)


...deleted
> Clearly, we want a function which MUST do a lot of work,
> and which will accept a lot of distinct inputs.
> Ackermann, Tak, etc, don't meet these requirements
> so they should be thrown out.

i am all for ''a lot of distinct inputs'', provided i finally get to see 
a lot of measurments.
if the only way to achive this is an abolishment of the simple 
micro-benchmarks i have grown up with, i am willing to throw them out.
out of sentimentality i would enjoy if they could stay.


bengt