[Shootout-list] Stuff

[Jon Harrop]

On Tuesday 26 April 2005 17:48, skaller wrote:
> On Wed, 2005-04-27 at 01:13, Isaac Gouy wrote:
> > - compare language performance using the same algorithms and data
> > structures; and the comparison will be unfair because for some language
> > the 'standard' approach would use a different algorithm or data
> > structure.
>
> Worse than that, 'same algorithms and data structures'
> can't be well defined.

Very true.

> > - compare language performance using different algorithms and data
> > structures; and then it's a comparison of algorithms and data
> > structures and programming skill, not a comparison of language
> > performance.
>
> The Shootout solves that problem by allowing
> a community to contribute, refine, and improve solutions.

Also very true.

> Another example is a language like Felix

I had a hunch that would be appearing here somewhere. :-)

> Even more extreme examples are advanced partial
> evaluators. We can happily ban a C solution that
> precomputes values of ackermanns function .. because
> such code is manifest, but can we ban a partial evaluator
> that does the same thing behind the scenes? An example
> is Meta-Ocaml, FISh can probably do this too.

That is an excellent point, and all the more reason to include compile time as 
well as run time. I'd like to see some of the shootout benchmarks implemented 
in MetaOCaml.

> I believe the answer is: the purpose of the Shootout
> should NOT be to measure performance. That is indeed
> inherently stupid, whether large or small scale.
>
> My view is the shootout should *showcase* languages.
> The tests should be designed to exhibit strengths
> and weaknesses, perhaps to allow people to choose
> the best tools for the job, perhaps to allow people
> to learn new techniques and new languages.

I half agree. I do think it is good that the shootout tries to measure 
performance, even though any such measure is subjective due to the arbitrary 
requirements placed upon the implementations.

I do think that the shootout does and should "showcase" languages by providing 
lots of little, well-coded examples in many different languages. This is 
hugely useful for anyone who knows one language and wants to see the 
equivalent in other languages. As Brent has said, the LOC measure tells 
people how expressive each language is at each task.

> In my view, the program times should not be
> the results of the test, but part of the specification:
> the idea should be to write the cleanest possible
> code to solve the problem with reasonable performance.
> The result of the test is the *code*, not the time.

Yes, I think that is basically what the shootout does ATM. No doubt some of 
the programs could be sped up by quadrupling their size, but an intermediate 
run-time vs writing-time performance is useful in practice.

I do like the idea of allowing several versions of each program, each 
minimising one of CPU time, memory use, compile time and LOC, and then having 
CRAPS pick the best code for each category and weight the goodness of the 
language of the performance vs expressiveness variety it can give. There's a 
problem with partial specialisation here though...

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists