[Shootout-list] Lies, Damn Lies, and Statistics ...

[Brent Fulgham]

Those of use who have been working on the shootout for
some time have learned the truth behind Benjamin 
Disraeli's famous quote.  While I hope that the 
shootout can provide useful data for comparing the 
strengths of various languages, the metrics we use are

far from perfect and should never be the sole
criterion
for deciding on a language for a given problem.

Lately I have seen the shootout invoked in a number of

language debates [1, 2] to prove the superiority of
one
language over another.  These various flame wars have 
had positive effects for the shootout (in that new 
people have subscribed to the list [welcome, by the 
way!]), and some negative (in that some [deserved] 
criticism [and flames] have been sent to me via
private
e-mail.)

While I do think the shootout provides valid data for 
many comparisons (for example, the GNU C compiler is 
very close in efficiency to the propietary Intel 
compiler, at least on a uniprocessor machine), it is 
dangerous to draw too much inference from the existing

data.

We are a fairly diverse group of people working on 
these benchmarks.  We have people working in embedded 
software, scientific software (such as myself), 
university professors, language implementors, and 
people who fall into multiple groups.  Although we
each
have expertise in our local areas of work, none of us 
are experts in all the various languages and systems 
available today.

Consequently, it should not be surprising that we find

cases where languages are not optimized uniformly, or 
that certain opportunities for optimization are not 
taken.  That's what I rely on the readers of this list

(and visitors to the shootout site) for -- to be the 
'auditors' of this database.  I figure that with
enough
language zealots watching over our shoulders, we 
should have a fairly solid and 'fair' set of 
benchmarks.

I welcome all the new subscribers, and hope that all
of
you will contribute solutions to the various languages

that are in need of implementations.

As for the language zealots, I challenge them to help 
keep us honest by identifying compiler options and 
similar tweaks that show their favorite languages in 
the best light.

Thanks,

-Brent

[1] A quite interesting thread on comp.lang.ada
[http://groups-beta.google.com/group/comp.lang.ada/browse_frm/thread/703c4f68db81387d/3c5e9fc4fde1989b#3c5e9fc4fde1989b]
[2] A Slashdot story
[http://developers.slashdot.org/article.pl?sid=05/03/14/2258219]
[3] Not directly related to the shootout, but a
benchmark nonetheless [http://groups-beta.google.com/group/comp.lang.ada/browse_frm/thread/243dc2fb696a49cd/b4c123d3d22f7022?tvc=1&q=Ada+C%2B%2B+comparison#b4c123d3d22f7022]