[Shootout-list] Directions of various benchmarks

[Brent Fulgham]

--- Isaac Gouy <igouy2@yahoo.com> wrote:
> --- Brent Fulgham <bfulg@pacbell.net> wrote:
> > But there are practical limitations.  If we want
> > to have 1,000 different values of N for each test,
> > then we move from a shootout run that takes
> roughly
> > 24 hours to something that could take weeks to run
> > (since run times increase as N increases.
> 
> If there is large variability in program run-time
> then should we re-run all the programs for a
benchmark,
> rather than single programs?

Probably.  But we should have a dedicated build
machine live on the network if we want to get to that
point.

It would be better if we were above the measurement
noise threshhold so we would only have to rerun tests
when the sources changed.

-Brent