[Shootout-list] Many tests cheat

[Einar Karttunen]

On 02.11 15:37, skaller wrote:
> For example, there is a limit on how much buffering
> is allowed in some tests that do I/O (4K).

This would shift many benchmarks to "how optimized is the default IO
solution".

> In particular, Haskell doesn't have arrays in the C sense:
> you can't 'write' to a purely functional data structure.
> Functional arrays are actually an active research topic.

Actually Haskell has many kinds of mutable arrays:
boxed and unboxed arrays in both the IO and the ST monad and
even the parallel array extension. The last one is the fastests 
in many cases, but the compiler support is quite buggy.

> So *reading* arrays in Haskell can be just as fast as 
> in C, so the speed of read-only access should be measured
> separately from mixed read./write. Haskell will do fine
> on the read only test. It will have trouble with read/write,
> but the answer may not be to use a functional array at all.
> Depending on the problem, a Haskell programmer might have
> to use lists, some kind of balanced tree, etc.

Writing arrays is simply not as elegant in Haskell, but there 
is no inherent reason why it should be slower than C. The 
current performance is much more due to GHC, in which arrays 
have never been very important.

As for hashes there were some bugs that will be fixed in 6.4 
which should be released shortly. I think I will be submitting
better hash entries then...

> You can say this doesn't measure array access in the C
> sense and the answer is: you're right, you have to
> disqualify Haskell completely from this test,
> because even a functional array is NOT actually an array
> like a C array.

Haskell has very C like arrays in addition to the more 
functional alternatives. Blame ghc not the language.

> I think any way to produce the correct answer is
> acceptable if it doesn't exceed time/memory limits.
> I also think the default 'CRAPS' rating should
> include memory and LOC = 1, not 0. This means a
> high performance array access test solution in Haskell
> might exist, which is very fast .. but it may require
> a hand coded data structure and algorithm, and the
> test should be required to include all that code.

Changing scoring to make some languages fare better 
is imho not a good thing.

- Einar Karttunen