[Shootout-list] Many tests cheat

[Einar Karttunen]

On 01.11 22:28, skaller wrote:
> So .. perhaps you might CC to me as well as the list
> if replying? 

ok, will keep the CC.

> Many of the tests I've seen cheat, one way of the other.
> One Ocaml test uses double the allowed buffer size
> reading a file. Python cheats blatantly on the
> same test, using commands that read the whole file
> at once.

Eliminating such things is good and fixes are welcome. 

> I'm generally in favour of functional testing,
> and doubt that 'do it the same way' tests prove
> anything at all. They tend to bias the tests in favour
> of language with 'same way' facilities and against
> others that don't -- for example Haskell can't do arrays
> so well.

I agree to that, the only problem is finding suitable problems. 
Also they should be looked into so a language cannot win by a better
algorithm, forcing the existing solutions to be rewritten.

> I would prefer to see tests where the requirement was
> for a significant outcome which suggests some technique
> which is efficient in one language, but doesn't require it.

This might create a situation where we compare apples to 
oranges... But is not necessarily bad.

What do you have in mind? e.g. LZ77 and a small interpreter 
could be nice.

> For example, to test array access, create a test to do
> a significant array operation, such as multiply two
> large matrices. This may well penalise Haskell too,
> but there is opportunity to solve the problem using
> functional techniques.

Actually matrix multiplication is not too slow in Haskell 
if one uses parallel arrays ;)