[Shootout-list] ackermann, values for ''n''

[John Skaller]

On Tue, 2005-06-07 at 14:19 -0700, Brent Fulgham wrote:

> This isn't really a hard-and-fast rule; it's just
> an observation that it takes a long time to run
> the benchmarks en masse.  Therefore, I tend to avoid
> large structural changes since this requires many
> hours of time to get back up again.
> 
> Currently it takes about 36 hours to run all the
> tests.

Then IMHO what you need IS a large structural change
to solve this problem. What happens if you find a bug
in the Shootout 30 hours into the tests?

The Python code I recently posted avoids
this problem in two distinct ways:

(a) it automatically seeks an appropriate time,
allowing you to test run the benchmarking in 
a smaller amount of time

(b) if the test harness is only 90% right,
there is a good chance that the tests that
run will produce useful data, and that data
is retained (all individual benchmarks individually
contribute to the results)

(c) A test that fails 90% of the way into
the process doesn't require rerunning that
90% of the process again. (the ordering
is random)

I also note that the code is very short,
although it represents a 'major structural change'
in the design of the benchmarking process, it
has no impact on the web site at all. The output
data can simply be analysed by another script and
a standard data.csv, ndata.csv pair created.

Another effect of this would be that the data in
these .csv files would be a lot more reliable.

And another is that an additional web site could
be created showing the results for a *different*
test machine, eg an Mac running OSX on a 64 bit G5.

There is no particular reason to use the actual code
I wrote, you could just write a new benchmarking
tool in Perl.

To summarise: there is a time when it is easier and
cheaper to solve a problem with a complete rewrite
than bother patching old code. If you have a 36 hour
test run to check for problems in in either the Shootout
benchmarking process, or perhaps an option on the
compilation command line of one language is wrong,
or there is a problem with a new version of some
language .. if it takes 36 hours to find this, repair it,
and rerun the process to check your repairs are right,
then you have a MAJOR problem!


-- 
John Skaller, skaller at users.sf.net
PO Box 401 Glebe, NSW 2037, Australia Ph:61-2-96600850 
Download Felix here: http://felix.sf.net