[Shootout-list] X per second scoring system

[Brandon J. Van Every]

Bengt Kleberg wrote:
> Brandon J. Van Every wrote:
> ...deleted
> > *Timed*, yes.  But it seems you *count* from within the code.  It's
> > invasive.
>
> often the count is the number of times something should be
> done (create
> n threads).
> other times n is used to create input data that decides the amount of
> work done by the test.
> both ways are easier to keep accurate than having external
> timers stop
> the test and still manage to extract correct values from it.
>
>
> ...deleted
>
> > You already have obfuscating loop code, and obfuscating command line
> > argument processing code.  Adding a timer would be a
> trivial amount of
> > additional text.  The real difficulty is creating and
> adding the timers
> > in different languages.  It's gruntwork.  It would make it
> possible to
> > run the benchmark twice as fast, however, assuming you're using a
> > doubling calibration for N.  How are you actually calibrating N?
>
> you are correct about there already beeing obfuscating code
> present. i
> think it should be kept at a minimum. adding timers is
> adding, even if it takes a trivial amount of additional code.

The phrase "penny wise and pound foolish" comes to mind.  Adding a
trivial amount of timer code is not visually a big deal.  It doesn't
affect comprehension of the test suite at all.  If you REALLY REALLY
REALLY CARED about visual inspectability, you'd have some easily
recognized comment delimiters in all of the tests, to separate the 'real
work' from the counting boilerplate.

I can respect the argument of 'it being work'.  But 'it's tough to look
at' is just silly.

> not to mention that some
> languages might need more than trivial. perhaps.

I can't think of any reason why that would be true.  Languages either
have a way to call timers or C functions, or they don't.  There's
nothing complicated about it.

> the calibration of n is somebody-elses-problem.

It's a problem for anyone who wants to run the tests.  Who trusts who's
N?  What if your machine was 2GHz and mine is 4GHz?  Again I ask, how is
N currently calibrated?  Just a Wild Assed Guess?

> as mentioned in one of
> my previous emails to the list it does not matter so much
> since we have
> a table for each test with lots of n. this is a good thing (tm), see
> Timing Trials, or, the Trials of Timing
> (http://cm.bell-labs.com/cm/cs/who/bwk/interps/pap.html) as to why.

Are you saying you do N calibration to detect loop optimizers?  Is this
automated in the Shootout, or at least readily graphed and displayed?
Otherwise the advantage is theoretical.  Also, if you're doing N
calibrations to find linear scaling sequences, you're talking about
doing an awful lot more benchmark runs than if you just had timers in
the tests and didn't have to guess N.

> as an aside: why is there no link from the shootout to Timing Trials?

Is there some reason there should be?  It's a good paper, but it's not
the Shootout.  Where do you  think such a link should appear?


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur