[Shootout-list] Many tests cheat

[skaller]

Hi -- I'm having trouble with Alioth, it just won't
send me mail. I have two distinct email addresses:
one is registered (but I get no mail and can't remove
the registration). I applied again using an alias
(skaller@users.sf.net) but didn't even get a confirmation
mail. I have no problem with Mailman on other lists.
So .. perhaps you might CC to me as well as the list
if replying? 

Many of the tests I've seen cheat, one way of the other.
One Ocaml test uses double the allowed buffer size
reading a file. Python cheats blatantly on the
same test, using commands that read the whole file
at once.

I'm generally in favour of functional testing,
and doubt that 'do it the same way' tests prove
anything at all. They tend to bias the tests in favour
of language with 'same way' facilities and against
others that don't -- for example Haskell can't do arrays
so well.

I would prefer to see tests where the requirement was
for a significant outcome which suggests some technique
which is efficient in one language, but doesn't require it.

For example, to test array access, create a test to do
a significant array operation, such as multiply two
large matrices. This may well penalise Haskell too,
but there is opportunity to solve the problem using
functional techniques.

Comments?

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net