[Shootout-list] Directions of various benchmarks

[skaller]

On Tue, 2005-05-17 at 10:53, Jon Harrop wrote:

> > My impression is that the original Shootout had more emphasis
> > on measuing things which were meaningful because most of the
> > languages were interpreters. If we move to just pure performance
> > testing, there seems little point bothering to include Python,
> > Perl, or other interpreters because they can't compete: the competition
> > will be between C, C++ and modern FPLs.
> 
> I disagree. Anything which fits well into an interpreted language (e.g. 
> regexps in Perl) will be both fast and succinct. 

How interesting is it to compare Perl/PCRE with C/PCRE
with Python/PCRE with Ocaml/PCRE .. is it interesting to
compare PCRE with PCRE??

IMHO it is more interesting to compare Felix regexps with PCRE:
Felix has *real* builtin regexps which Perl does NOT, and they
have 'in principle' O(1) execution speed compared to Perl's
exponential worst case, at the cost of exponential memory..
and I have no idea if they're actually faster on significant
or nasty problems.

> General code will, of 
> course, be much slower. But the challenge for the interpreted-language 
> authors is then to shoe-horn all problems into the limited capabilities of 
> the language. This is an accurate reflection of (some) real-world 
> programming.

Isn't it the same for compiled languages:
trying to shoehorn the problem into a way the compiler handles
efficiently: eg tail recursive vs. nontail recursive list
handling in an FPL, writing C code in a way that
C compilers will optimise, etc.

> > For example, Python and Perl have hash tables built in. There is a
> > big difference between implementing an algorithm using a builtin
> > hash table, and implementing one that needs a binary tree.
> 
> Yes, and this is something the shootout should illustrate, not hide.

Yes. However it won't do that by emphasising performance over
everything else. Somehow I would like to see much more emphasis
on clarity of code and developer time. perhaps by CRAPS weighting
LOC equal to speed by default (last time I tried that GHC
won the Shootout hands down).

> For different implementations of arithmetic-intensive code, no. The algorithms 
> used will have a bigger effect than compiler switches. This comes into play 
> when verbose languages start to exceed 100 LOC and have to be cut down, 
> rather than optimised.

Yes, but you're agreeing with me, only you're relying on a 
100 LOC limit, whereas I'd prefer to see the LOC reflected
directly in the defaut craps weighting.

I wouldn't exclude a 200 line C solution that outperformed
Ocaml's 50 line solution -- but I'd make sure that the overall
rankings reflected the tradeoff.

Hmm .. I wonder about a performance vs. loc picture?
This would be a two dimensional grid with an X for each
language..

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net