[Shootout-list] X per second scoring system, resume

[Brandon J. Van Every]

Bengt Kleberg wrote:
> Brandon J. Van Every wrote:
> ...deleted
> > Part of my point is *all* tests should be run for 60 seconds, to
> > eliminate inaccuracies of OS process startup / shutdown / slicing.
>
> you are correct. it would help if we made sure that the time
> for various
> n included a value below 60 seconds and one above. however,
> 60 seconds
> is long, and might not be needed. perhaps 30 seconds (or 20)
> is sufficient?

Really one would need to do some testing of the target to see what the
rate of variability actually is.  For a particularly slow-to-start
environment, 20 would not be enough.  Like what if some pig language
takes 1 second to crank up, that's 5% of your score.  I think GC
languages are often piggy like this.  At least, that's my impression
using Visual Studio 2003... granted it's loading tons of crap and is a
fine Microsoft product.

More seconds is safer.  The real question is whether you can start it
when you go to bed and have the results in the morning.  Well, the other
real question is whether you can run a given test quickly when you're
trying to optimize it.  I'd use 'quick and dirty shorter' benches for
that, mentally calculating the number of burps I was seeing.  Things are
better when you have more accurate timers, you don't need as many
seconds to pad stuff.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur