[Shootout-list] benchmark design, hard limit for how many 'n'?

[Bengt Kleberg]

Brent Fulgham wrote:
>>could someone tell me if there is a hard limit for
>>how many 'n', and if so, what it is?
> 
> 
> There is no limit (save memory limitations, I
> suppose).
> The minibench script takes an input argument, which
> is a comma-delimited list of input values to run for
> each benchmark.
> 
> This could be effectively any number, but I have
> tried to keep it just a handful because it can add
> significantly to the time required to run the
> benchmark.
> I apologize if anything I said led you to believe that
> we can only use 3 input values.

no need to apologise. it was not you.

may i take this opportunity to ask what kind of output from minibench 
that is used by the rest of the system?

there is presumably one output-of-test for each argument. one command 
line for starting the test, one result of test (ie time, memory used).
any others?
are these all kept in files? are the files created and destroyed, or do 
all files exists for all tests and arguments?


> I also recently removed the limitation that it
> could only compare against one specific output file.
> It can now compare against a different output file
> for each value of N. 

that is a good thing. it must have been interesting to use the shootout 
before this limitation was removed. :-)