[Shootout-list] X per second scoring system, resume

[Brandon J. Van Every]

Bengt Kleberg wrote:
>
> the shootout have the possiblity to remove the startup/stop
> times from
> the timings in the scores. great thing, imho. if the interested party
> wants startup included (or not) they can choose.
>
> for the simple bench mark i suggest removing the startup time
> by default.

Startup time varies by language.  Within a language, startup time varies
by data structures and housekeeping initialized before the language's
equivalent of main() begins.

Shutdown time is highly dependent on the data structures and control
flow of the test.

Slicing cannot be eliminated.  You can only run a test long enough that
it's averaged out.  Too short a test, and you get an OS process switch
spike.

Let us also not forget OS startup/shutdown timing hiccups.

So, given these realities, how does the Shootout allow for startup/stop
times to be removed from the scores?  I don't see anything in the tests
themselves to indicate per-test control.  I'm supposing you mean you can
subtract a number for a completely different test, presumed to represent
the startup/stop times for a given language.  I say this is a false
assumption.

But if empirical evidence says it's a good assumption, I'll stand
corrected.  I'd ask how the empirical evidence is derived and verified.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

20% of the world is real.
80% is gobbledygook we make up inside our own heads.