[Shootout-list] Directions of various benchmarks

[skaller]

On Mon, 2005-05-16 at 23:28, Jon Harrop wrote:

> > just that 
> > adding yet another number crunching test does not seem very good.
> 
> I think we should add more number crunching tests before throwing some out.

Again some reason to think of categories. Low level integer calculations
are optimised differently to floats, so there are two kinds of 
'number crunching to test. Modern applications do a lot of string
handling that could be checked. At least FPLs have polymorphism
and first class functions so that can be checked. Some algorithms
run on the stack and some need heap, so heap performance has to
figure somehow.

And there are integration issues with languages, and various features
which are often the key distinguising features of the language:
C++ --> upgrade of C, Felix --> upgrade of C++, Python etc have
C bindings, plugins, microthreading, etc.

My impression is that the original Shootout had more emphasis
on measuing things which were meaningful because most of the
languages were interpreters. If we move to just pure performance
testing, there seems little point bothering to include Python,
Perl, or other interpreters because they can't compete: the competition
will be between C, C++ and modern FPLs.

For example, Python and Perl have hash tables built in. There is a 
big difference between implementing an algorithm using a builtin
hash table, and implementing one that needs a binary tree.

For compiled languages, the performance tests are fairly meaningless:
the actual translator and compiler switches and precise test code
are more significant than any language differences. For ackermann,
try running gcc without -funroll-loops: the performance is woeful.
It's a hack. gcc is too stupid to see the tail calls without that
switch. If you rewrite the function longhand that switch isn't
needed.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net