[Shootout-list] Directions of various benchmarks

[Jon Harrop]

On Monday 16 May 2005 22:18, skaller wrote:
> Again some reason to think of categories. Low level integer calculations
> are optimised differently to floats, so there are two kinds of
> 'number crunching to test. Modern applications do a lot of string
> handling that could be checked. At least FPLs have polymorphism
> and first class functions so that can be checked. Some algorithms
> run on the stack and some need heap, so heap performance has to
> figure somehow.

Good points. Although I'd give up on trying to design tests which stress, for 
example, HOFs. Instead, I'd like to design tests which can be implemented in 
a wide variety of different ways. Then users will be able to see which ways 
are easiest and most efficient to implement in each language.

In the ray tracer, for example, to get the best performance you can sacrifice 
the nice inheritance hierarchy for a specialised short-hand hack which is 
slower, but which gives you the LOC you need to implement some more juicy 
optimisations.

> My impression is that the original Shootout had more emphasis
> on measuing things which were meaningful because most of the
> languages were interpreters. If we move to just pure performance
> testing, there seems little point bothering to include Python,
> Perl, or other interpreters because they can't compete: the competition
> will be between C, C++ and modern FPLs.

I disagree. Anything which fits well into an interpreted language (e.g. 
regexps in Perl) will be both fast and succinct. General code will, of 
course, be much slower. But the challenge for the interpreted-language 
authors is then to shoe-horn all problems into the limited capabilities of 
the language. This is an accurate reflection of (some) real-world 
programming.

> For example, Python and Perl have hash tables built in. There is a
> big difference between implementing an algorithm using a builtin
> hash table, and implementing one that needs a binary tree.

Yes, and this is something the shootout should illustrate, not hide.

> For compiled languages, the performance tests are fairly meaningless:
> the actual translator and compiler switches and precise test code
> are more significant than any language differences. For ackermann,
> try running gcc without -funroll-loops: the performance is woeful.
> It's a hack. gcc is too stupid to see the tail calls without that
> switch. If you rewrite the function longhand that switch isn't
> needed.

For equivalent arithmetic-intensive code, yes.

For different implementations of arithmetic-intensive code, no. The algorithms 
used will have a bigger effect than compiler switches. This comes into play 
when verbose languages start to exceed 100 LOC and have to be cut down, 
rather than optimised.

For data structure intensive code, no. The "nth" program from my book, for 
example, sees comparatively tiny performance differences when compiled with 
different optimisation flags because the programs are memory limited. The 
performance is then a function of the algorithmic efficiency of the standard 
library implementation. This is also something which can be productively 
measured by the shootout.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists