[Shootout-list] Many tests cheat

[Bengt Kleberg]

skaller wrote:
> Hi -- I'm having trouble with Alioth, it just won't
> send me mail. I have two distinct email addresses:
> one is registered (but I get no mail and can't remove
> the registration). I applied again using an alias
> (skaller@users.sf.net) but didn't even get a confirmation
> mail. I have no problem with Mailman on other lists.
> So .. perhaps you might CC to me as well as the list
> if replying? 

as per request i am sending my answer to the private address, too.


> Many of the tests I've seen cheat, one way of the other.
> One Ocaml test uses double the allowed buffer size
> reading a file. Python cheats blatantly on the
> same test, using commands that read the whole file
> at once.

as long as the about-this-test claims that some things should hold, then 
the test programs should follow this if they are to be measured. imho.


> I'm generally in favour of functional testing,
> and doubt that 'do it the same way' tests prove
> anything at all. They tend to bias the tests in favour
> of language with 'same way' facilities and against
> others that don't -- for example Haskell can't do arrays
> so well.
> 
> I would prefer to see tests where the requirement was
> for a significant outcome which suggests some technique
> which is efficient in one language, but doesn't require it.

this is what i too would prefer. however, i have found it to be beyond 
my ability to create such a test in some cases. therefore i think it is 
ok to have 'same way' tests.


> For example, to test array access, create a test to do
> a significant array operation, such as multiply two
> large matrices. This may well penalise Haskell too,
> but there is opportunity to solve the problem using
> functional techniques.

i think what you have described is a multiply-matrices test. it will 
tell us something about accessing matrices, but we would be in the dark 
about array performance. this does not have to be ''bad'', but it is not 
the same thing.


bengt