[Shootout-list] benchmark design, hard limit for how many 'n'?

[Brent Fulgham]

--- Bengt Kleberg <bengt.kleberg@ericsson.com> wrote:
> may i take this opportunity to ask what kind of
> output from minibench that is used by the rest of
> the system?

Each iteration of the test produces one output file
for each test program.  The output file is compared
against an official "known good" output file.  If
they match, the program is accepted as valid and
it timings are added to an output table (otherwise
the program is marked as 'failed'.)

Each iteration, the output file (for the specific
program) is overwritten.  So if I run the 
'hello.erlang' program for N=3,4,5 it will run
the program once for N=3 (output to
tmp/hello.erlang_out) which will be compared to
tmp/Output.3.  Then it runs for N=4, (output to
tmp/hello.erlang_out again), which is now compared
to tmp/Output.4, and so on.

Eventually, the output data table is compiled into
the 'data.csv' and 'ndata.csv' that are used to
run the website.
 
-Brent