[Shootout-list] Rule 30

[Jon Harrop]

On Friday 20 May 2005 07:12, Isaac Gouy wrote:
> --- John Skaller <skaller@users.sourceforge.net> wrote:
> > ALL compilers do constant folding which is a trivial
> > example of this...
>
> -snip-
>
> The question had a specific context - pidigits.
>
> Jon's answer was "We don't know which do. That's the point."
>
> Somewhere between none (an imaginary problem) and all of them.
> Perhaps the point is to find someone who knows more.

I can phrase that a little stronger: "Nobody can predict which languages will 
precompute the answer" or "It is not even theoretically possible to determine 
what this benchmark will be measuring".

In theory, you can determine exactly how the current shootout implementations 
are compiled by current compilers and, therefore, show what precomputation 
(if any) is not being measured at the moment. In practice, this is 
intractably difficult. In the future, the compilers will change and, 
therefore, so will the amount of precomputation. So even if you could find 
and motivate this "clever person", they would have to constantly regenerate 
their results.

However, all of these problems can be neatly side-stepped just by designing 
the benchmarks properly. Specifically, by making each benchmark a function 
which takes one of many possible inputs (say, an int) and returns one of many 
possible outputs. All benchmark programs which return the same result as the 
original program (the specification) are assumed to be correct. The input can 
even be generated randomly every time the shootout is run, provided this 
doesn't affect the time taken too much.

As a simple example, perhaps a better input for the ray tracer would be to 
accept the angle the scene is drawn at?

On the basis of this, most of the current benchmarks are flawed (to varying 
degrees). For example, an nbody implementation might contain a result for 
N=10^6 and start from there, instead of from the beginning. The pidigits 
program might contain the digits of pi. The random benchmark might contain 
the 900,000th random number. Note these extras may only appear in the 
assembler, and not in the source.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists