[Shootout-list] logarithmic scoring system

[Bengt Kleberg]

Bengt Kleberg wrote:
...deleted
> ''x per second'' is much more difficult to measure than ''time to do x''
> 
> one way would be to increase n for, say, the ack test until it takes one 
> second. this would be a very high penalty for the languages that take 
> one second to start. that is solved by using more than one second.
> 
> it would also take a long time to run all tests on all languages since 
> we are using trial-and-error to find the right n for each language. the 
> nice thing is that we would not have to rerun any test unless the 
> language/operating system/hardware changed.
> 

(it is bad netiquet to answer ones own emails, but i suddenly realised 
how stupid i have been.)


''x per second'' is not a big problem.

we already do lots of runs for each test on each language. i have been 
stearing myself blind on how to get exactly the right time in ''x per 
second'', not realising that we can do a series of runs of each test for 
each language (like we do now) and then just see the value of n at the 
place where the graph intersects the time we want (be it 1, 2, 10 or 60 
seconds).

we might save time since we could stop running tests on the slow 
languages once they have had a run with a value of n that took more than 
the time limit.


bengt