[Shootout-list] What people say about shootout

[Jon Harrop]

On Saturday 30 April 2005 14:46, Isaac Gouy wrote:
> Did you intend to mail me rather than the list?

Oops. :-)

> --- Jon Harrop <jon@ffconsultancy.com> wrote:
> > > > If nbody is to be kept then I really think we should factor out
> > > > all of those magic numbers - it's really quite bad!
> > >
> > > Why? Concern with LOCs?
> >
> > In this case the downsides are:
> >
> > 1. lots of unnecessary duplication between the programs, which will
> > bloat the programs and put people off porting them
>
> That doesn't seem to have stopped anyone so far!

True. It certainly put me off that test.

HEY! I just had a really obvious thought. Do you gather statistics on the 
CRAPs weights that people choose? That could be contrued as a kind of voting 
system.

Also, are the scores normalised, so weights of 1,1,1 really count time, memory 
and LOC equally?

> > 2. real code would load its data and we're losing out on seeing how
> > to load such data in a variety of languages.
>
> We're gaining a test untainted by the cost of file IO, or random number
> generation.

I'd say you're gaining a test tainted by poor programming style.

> We have plenty of tests that do file IO and random number 
> generation.

You have plenty of well written tests so why not have a poorly written one as 
well. I think there may be some merit in this, but I'd like to see the task 
description state that the program must be written in poor style with hard 
coded data appearing all over the place as it won't be given any data.

Also, hard coding the data makes the performance results dubious as languages 
are much more likely to do partial specialisation which, on real code, they 
wouldn't be able to do.

> > 3. encourages poor programming style if newbies read these programs
> > (shouldn't embed magic numbers).
> > 4. Skews the LOC count a lot.
>
> 5. Doesn't make OCaml look good :-)

OCaml still does comparatively better on LOC, of course, but the score doesn't 
reflect real programs as the majority of the code is actually data. If you 
take out the data then the OCaml program is likely to be 1/2 the length 
rather than 10% shorter. I think that would be much more representative of 
real code.

I'd be more than happy to see numerical tests which show OCaml in a poor light 
provided they actually represent something that people would do. For example, 
I don't expect OCaml to perform well on my "relax" benchmark.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists