[Shootout-list] X per second scoring system, resume

[Brandon J. Van Every]

Bengt Kleberg wrote:
> Brandon J. Van Every wrote:
> > Bengt Kleberg wrote:
> >>
> >>for the simple bench mark i suggest removing the startup time
> >>by default.
> >
> > Startup time varies by language.  Within a language,
> startup time varies
> > by data structures and housekeeping initialized before the
> language's
> > equivalent of main() begins.
>
> this is correct. we would need the startup time of that particular
> language when we remove the startup time from the results of that
> particular language in a test.
> this was my intention all the time. if it came out as if i
> wanted to use
> one and the same startup time for all languages it was a mistake.

Even for 1 language it's a mistake.  What you're starting up varies *by
test*, although I don't know how much it actually is in various cases.

> > Shutdown time is highly dependent on the data structures and control
> > flow of the test.
>
> this is correct. and, hopefully a good thing (tm). ie, we
> have a fixed
> shutdown time that is measured for a as-short-as-possible program.

It varies *per test*.  What if the language has all sorts of
housekeeping that needs to be performed between the end of the program
and the end of the process, for normal termination?  Especially for
things like system memory, threads, and IO.

> it is not totally correct. but i would say that it is much more
> truth-like that having the results of many years ago archived

Um, no, the *archiving* part is uncontroversial...

> and used for comparing with todays results :-)

And this depends entirely on how you choose to compare the results.  I
gave a possible comparison, 'sufficiently linear'.

> and that is ok, is it not :-)

Sure, if you can prove startup / shutdown is 'sufficiently bounded'.
Just measuring a different test isn't proof though.  Proof is actually
putting timers into the code and seeing what you get.  You might opt to
take out the timers afterwards, satisfied that they're overkill.

> what if we run the smallest program several times and see how much
> variation there is in the results?

You need to allocate system memory, start threads, perform IO...


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

20% of the world is real.
80% is gobbledygook we make up inside our own heads.