[Shootout-list] X per second scoring system

[Bengt Kleberg]

Brandon J. Van Every wrote:
...delered
> Probably 'X per second' is the easisest way to accumulate a composite
> sum of scores, where X is some desired computation to get through.  For


''x per second'' is much more difficult to measure than ''time to do x''

one way would be to increase n for, say, the ack test until it takes one 
second. this would be a very high penalty for the languages that take 
one second to start. that is solved by using more than one second.

it would also take a long time to run all tests on all languages since 
we are using trial-and-error to find the right n for each language. the 
nice thing is that we would not have to rerun any test unless the 
language/operating system/hardware changed.

the alternative would be to run te test with a very high n, and 
interrupt the test after a certain time. this seems error prone to me, 
as the language would have to be interrupted in a polite manner. 
otherwise we would loose the value we want from the test (the x times it 
managed to do the test). moreover, some test are not possible to do this 
way (reverse)


> instance, in Viewperf it's all measured in 'frames per second'.  I think
> all the 3D benchmarks use a 'higher is better' paradigm.

''higher is better'' is one way of doing it. the other one is ''lower is 
better''. since everybody i know has show their ability to deal with 
both systems during the recent olympics, i think it does not mattter 
that much which one we choose.


>  PC Magazine
> tends to mix and match higher and lower scoring systems, but they're

this is not good. one should avoid mixing tests where ''higher is 
better'' with tests where ''lower is better''. it is only useful when 
one wants to lie with statistics.


bengt