[Shootout-list] Timing variations

[John Skaller]

On Mon, 2005-05-23 at 09:56 +0200, Bengt Kleberg wrote:
> On 2005-05-20 19:23, Brent Fulgham wrote:
> ...deleted
> > But you are right -- our runtimes are probably
> > low enough that we are encountering measurement
> > noise.
> 
> that could be quantified if we find out the granularity of the timer. 
> and decide upon a accuracy level. ie, we would still be encountering 
> measurement noise but we would know how much it is.

No. I think you are 'barking up the wrong tree'.

I have just done some preliminary experiments with some test,
and I have found on my slow x86 box variations of +/- 20%.
This is pretty serious and suggests averaging just 3 runs is
inadequate -- and we're assuming here tests of the order
of 5 seconds run time where the timer resoltion is definitely
adequate.

My AMD64 appears to produce more stable results.

In the tests below, ALL the programs are run for increasing n,
until the result is over 5 seconds. gccopt is using -O3
and -fomit-frame-pointer. Felix is using gcc to compile
its output and still trashes it convincingly. On the AMD64
Ocamlopt, gccopt, and Felix all perform at the same speed.

For example, ackermanns function, n=12

Felix: 11.06 seconds
gccopt: disqualified for being too slow
ocamlopt: 7.6 seconds

Felix: 8.32 seconds
gccopt: disqualified for being too slow
ocamlopt: 7.45 seconds

Felix: 8.53 seconds
gccopt: disqualified for being too slow
ocamlopt: 9.07

Felix: 9.92 seconds
gccopt: disqualified for being too slow
ocamlopt: 8.88 seconds

The variations in times are alarming. 

-- 
John Skaller, skaller at users.sf.net
PO Box 401 Glebe, NSW 2037, Australia Ph:61-2-96600850 
Download Felix here: http://felix.sf.net