[[Shootout-list] important facts i can not find

[skaller]

On Tue, 2005-05-17 at 16:57, Bengt Kleberg wrote:

> on my computer the shortest time is very unrealiable (it is the 
> granularity of the time). i hope that the shootout computer can reliably 
> measure sub milliseconds, otherwise 0.01 seconds is too short.

FYI on my little benchmarking script I am doing this:

run each test for increasing N 
until one test exceeds T seconds or max N for
some limit, then stop. 

This has the pleasant effect of clipping the graph
of results for all languages, so both slow and fast languages
'run off' the graph, either upwards (slow languages) 
or rightwards (fast languages). 

The effect is that slower languages are categorically 
comparable (by the value of N achieved), whereas faster
languages all achieve the maximum N and their times
for the maximum N can be compared for higher granularity 
comparison. 

The result is still a proper ranking of all languages,
however you can only say 'X is 10% faster than Y' for
the fast languages.

It isn't quite clear how this measurement model might
fit in with the Shootout, it does, however, ensure
'automatically' that all the measurements are over
a significant period of time AND that the time is
bounded for the benchmarking process (modulo the 
assumption that the first test failing the limit
is not over the limit by an excessive amount)


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net