[Shootout-list] X per second scoring system

[Brandon J. Van Every]

Bengt Kleberg wrote:
> Brandon Van Every wrote:
>
> ...deleted very well thought out arguments that mostly comes back to
> this here below:
> > Just loop it again.  I think the problem is you're thinking
> that "loop
> > timers" around the source are a bad idea or something.  I
> will need to
> > look at the Shootout source code to understand exactly what your
> > objection is here.  What you're saying is making no sense
> compared to
> > other benchmarking suites I'm familiar with.  I would
> simply have start,
> > stop, loop, and counter functions implemented in a trivial
> C library.  A
> > language test would either have to call the C library or implement
> > equivalent timers.
>
> the current shootout, and all the test, are timed from
> without the test code.

*Timed*, yes.  But it seems you *count* from within the code.  It's
invasive.  Now I see what you meant by 'guessing N'.  You've got half of
a more traditional counter+timer system.

> i did not realise that we should be prepared to rewrite all the
> tests in all the languages. again i have been caught beeing lazy.

I suppose it's way easier to implement accurate counters than accurate
timers in various languages.  The startup/shutdown skewing problem is
mitigated at the cost of multiple runs to determine what N needs to be.

> i would much prefer not having to add timers to all test
> cases. it makes code harder to read.

You already have obfuscating loop code, and obfuscating command line
argument processing code.  Adding a timer would be a trivial amount of
additional text.  The real difficulty is creating and adding the timers
in different languages.  It's gruntwork.  It would make it possible to
run the benchmark twice as fast, however, assuming you're using a
doubling calibration for N.  How are you actually calibrating N?


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur