[Shootout-list] fannkuch

[Brent Fulgham]

--- Einar Karttunen <ekarttun@cs.helsinki.fi> wrote:
> Christophe TROESTLER <del-con@tiscali.be> writes:
> > The cosmetic change I made was only to add a
> > newline at the end (NO code change)!  I replaced 
> > the old code with the new one (so, yes,
> > there is only OCaml #3) and I wrote down the
> > running time (which was computed from a run some 
> > time ago).  As you remarked, the new time
> > vastly differs...
> 
> Runtimes of programs differ naturally. If you
> benchmark the same program ten times you will likely

> get ten different results.

True, but I would not expect such large differences.
The shootout runs each test three times and IIRC
takes the lowest run time.  They usually do not
differ significantly.

Examining a diff of the ndata.csv file is interesting
(see
http://cvs.alioth.debian.org/cgi-bin/cvsweb.cgi/shootout/website/data/ndata.csv.diff?cvsroot=shootout&r1=text&tr1=1.226&r2=text&tr2=1.234&f=h)

This diff shows (including the bogus 09/30/2004
dates!)
two time periods -- one before the change, and one
after:

Apparently changes were made to Ocaml-2 and Ocaml-3.
Ocaml-2 did not get the strange time changes you
observed with Ocaml-3.

version 1.226, 2005/05/17 04:29:25
 fannkuch,ocaml,3,7,-0.002,0.00,5160,36, 09/30/2004	
 fannkuch,ocaml,3,8,0.008,0.01,5720,36, 09/30/2004	
fannkuch,ocaml,3,9,0.088,0.09,5680,36, 09/30/2004
version 1.234, 2005/05/20 04:20:23
 fannkuch,ocaml,3,7,0.028,0.03,0,36, 09/30/2004
 fannkuch,ocaml,3,8,0.038,0.04,0,36, 09/30/2004
 fannkuch,ocaml,3,9,0.118,0.12,520,36, 09/30/2004

I would believe the second set of numbers over the
first -- they are more consistent.

But you are right -- our runtimes are probably
low enough that we are encountering measurement
noise.

-Brent