[Shootout-list] Stuff

[skaller]

On Wed, 2005-04-27 at 01:13, Isaac Gouy wrote:

> Ralph's posting to comp.lang.functional summarized the inherent
> stupidity of large-scale cross-language benchmarking:
> 
> - compare language performance using the same algorithms and data
> structures; and the comparison will be unfair because for some language
> the 'standard' approach would use a different algorithm or data
> structure.

Worse than that, 'same algorithms and data structures' 
can't be well defined.

> - compare language performance using different algorithms and data
> structures; and then it's a comparison of algorithms and data
> structures and programming skill, not a comparison of language
> performance.

The Shootout solves that problem by allowing
a community to contribute, refine, and improve solutions.

However, in both cases, the question arises,
whether a solution is 'in the language' or not.

And that one has no really good answer.

The most obvious example concerns 'external' libraries,
where both terms 'external' and 'library' cannot
be well defined.

A less obvious example is whether you're allowed
to fiddle compiler switches -- in principle, you
should not be: gcc -O3 is NOT the same translator
as gcc -O2, however you can use #pragma O3 because
pragmas are allowed in ISO C .. hmm ...

Another example is a language like Felix, which 
is designed to embed and be embedded in C/C++:
in one sense all Felix code is C++, and all C++
code is Felix.. the languages have utterly
distinct syntax, but they bind together at 
the source code level. Can an embedded C function
provided for performance be deemed improper when
the central design principle of the language was
precisely to allow this?

Even more extreme examples are advanced partial
evaluators. We can happily ban a C solution that
precomputes values of ackermanns function .. because
such code is manifest, but can we ban a partial evaluator 
that does the same thing behind the scenes? An example
is Meta-Ocaml, FISh can probably do this too.

I believe the answer is: the purpose of the Shootout
should NOT be to measure performance. That is indeed
inherently stupid, whether large or small scale.

My view is the shootout should *showcase* languages.
The tests should be designed to exhibit strengths
and weaknesses, perhaps to allow people to choose
the best tools for the job, perhaps to allow people
to learn new techniques and new languages.

In my view, the program times should not be
the results of the test, but part of the specification:
the idea should be to write the cleanest possible
code to solve the problem with reasonable performance.
The result of the test is the *code*, not the time.

With this kind of purpose, many of the arguments
about what is allowed and what is not are moot.
If an external library is used, that should be
considered perfectly acceptable, the main point
would be to be sure that the need is documented.
Hopefully, the extra messiness on the comandline
needed to link the library will create a suitable
negative impression for a C solution, for example,
compared to a Python one (where an import statement
in the program is all that is needed).


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net