[Shootout-list] Directions of various benchmarks

[Jon Harrop]

On Tuesday 17 May 2005 07:23, skaller wrote:
> On Tue, 2005-05-17 at 14:02, Jon Harrop wrote:
> > Why does the OCaml implementation of regex current use Markus' PCRE
> > rather than OCaml's built-in Str module?
>
> Perhaps it doesn't, but the point is the same: we're not
> testing the language involved in ANY of these cases,
> in all cases we're comparing C libraries.

I disagree: such benchmarks would test languages. They test the language's 
interfaces to C libraries or the parts of the language which happen to be 
implemented in C.

Using your argument, you could also rule out data structure intensive 
benchmarks because they test the GC which is written in C, or all programs 
compiled via C/C++ using GHC or Mlton (or, dare I say it, Felix ;-).

> > I expect everyone would agree. The problem is that clarity of code and
> > developer effort are inherently qualitative. LOC is much more objective
> > and quantitative but still suffers from "do we or don't we count lines
> > with a single brace on in C/C++/Java/C#" etc.?
>
> Yes, but as I have pointed out a few times here the actual
> performance measurements are ALSO just as flaky!

Yes, of course. Measuring more "n" would be better but it will never be ideal. 
We just have to do the best we can.

> For example I just checked ackermann and takfp on my AMD64,
> and guess what? Felix TRASHES gcc on both by a significant
> margin!
>
> Hey hey, isn't Felix compiled by gcc? Yeah, well Felix
> uses -O3 -fomit-frame-pointer .. and gcc with those options
> trashes Felix.
>
> So we could just use LOC until we have a better measure.

Yes.

> My opinion is: we should count lines, and ALL lines including
> comments and blanks, and make sure the tests are all equally
> nicely spaced and commented.

I'm for counting all lines with any code on. My justification for counting 
lines with a single brace in C-like languages is that this part of the 
grammar leads to verbosity which carries no useful information for someone 
hunting through the source for a familiar-looking piece of code (which is 
something I do a lot and which, I think, is why LOC is more important than 
bytes/tokens or anything else I can think of). In my experience, I'd say that 
hunting through code is one of the biggest contributors to development time.

> > I like the idea of asking authors to emphasise performance when <100 LOC
> > but emphasise brevity when >100 LOC.
>
> Interesting. However that would encourage things like manually
> unfolding ackermann's function in languages with a compact
> representation but which were hopeless at optimisation...

Yes, I don't think that would be a bad thing. The OCaml standard library 
contains several manually unrolled loops which is (correctly) a good 
indicator that ocamlopt does a poor job of unrolling loops.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists