[Shootout-list] Many tests cheat

[skaller]

On Tue, 2004-11-02 at 18:48, Einar Karttunen wrote:
> On 02.11 15:37, skaller wrote:
> > For example, there is a limit on how much buffering
> > is allowed in some tests that do I/O (4K).
> 
> This would shift many benchmarks to "how optimized is the default IO
> solution".

The benchmark obviously must be changed so the IO speed isn't
relevant -- right now several benchmarks measure nothing
else.

> > In particular, Haskell doesn't have arrays in the C sense:
> > you can't 'write' to a purely functional data structure.
> > Functional arrays are actually an active research topic.
> 
> Actually Haskell has many kinds of mutable arrays:
> boxed and unboxed arrays in both the IO and the ST monad and
> even the parallel array extension. The last one is the fastests 
> in many cases, but the compiler support is quite buggy.

Sure, because it's a hard problem. So it is likely
Haskell will pay some price one way or another.

Ocaml also pays a price. Every write to
general storage is done via a subroutine call
which is necessary to manage the minor heap,
it's called a write barrier I believe.

However access to specialised arrays -- string, unboxed
floats arrays and int arrays and some of the new fatter
integer arrays -- avoids this overhead I think.

So an array access test that only examines 'int' is giving
Ocaml an unfair advantage -- it will be almost as fast as C.
For general values, however, it will be slower because of the
write barrier .. and if the array is big enough,
the collector will also have to scan it.

So measuring 'array access' in a naive way is failing,
because of the assumption that arrays of everything are
the same as they are in C. In Ocaml this isn't the case.
There is a small price to pay for running the Ocaml 
collector in the presence of mutable data structures.

It is similar in Felix -- there's no write barrier,
but an array of pointers is going to cost a heap (pun intended)
when it comes to garbage collection. But an array of integers
is free from overhead.

> As for hashes there were some bugs that will be fixed in 6.4 
> which should be released shortly. I think I will be submitting
> better hash entries then...

I'd be very interested to see how Haskell copes, mainly
because, from working on Felix, I'm beginning to believe
that in the end purely functional language will outperform
others in many circumstances, mainly because of the 
optimisation opportunities. I originally specified eager
evaluation .. but I'm moving now to 'sloppy' evaluation,
meaning the compiler can evaluate anytime it wants.
(eagerly if usage is high, lazily if low).

> Changing scoring to make some languages fare better 
> is imho not a good thing.


That isn't point of the suggestion. The idea is that
if you can solve a problem efficiently with complex
code, you should also pay a price for the complexity
of the code: by default we should (IMHO) indicate overall
performance which includes speed, memory use,
*and* if both those things can be obtained simply.

The scripting languages are unfairly penalised,
but just measuring speed. We end up with C at the top
of the CRAP score, way ahead of the competition,
when "we all know it's a pretty bad language".
Otherwise we wouldn't have all those other languages
to have fun with, and there'd be no shootout . :)

Maybe 1:1:1 isn't a good ratio, but 1:0:0, the current
default, seems a little biased towards C. Perhaps
4:1:1 is better, I don't know. Anyhow this is a minor
issue and I'll shutup now because Felix does best with
1:0:0 .. :)

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net