[Shootout-list] X per second scoring system, resume

[Einar Karttunen]

On 30.09 12:28, Brandon J. Van Every wrote:
> - yes it would be nice to have some larger, roughly constant amount of
> time for all tests

Does this mean to fix input sizes? Many test won't scale in a linear
fashion... So this will penalize in most case faster languages.

> - 'guessing N' never goes away, it's just per test or per invasive timer
> iteration.

So we don't reduce the amount of work.

> - Ideally we guess N automatically.  'Somebody-else's-problem' works but
> creates labor for any new user of the Shootout test suite.

Choosing a different N can make the order vary.

> - invasive timers are usually more accurate, but are gruntwork to
> implement

Invasive timers are more accurate depending on the language,
making timer scales vary with language. 

> - we haven't checked if some language poses special implementation
> difficulties

Yes, they do.

The problem will be the end of timeslice, I don't see how 
that can be implemented very well in many languages.

> - it doesn't matter much if a slow language has a slow timer.
>   Only whether a fast language has a slow timer.

Yes it does. It makes the slow language appear even slower..
And if the timer is slow to react to the end of the timeslice
it could even make a slow language seem fast.

> - startup/shutdown times vary both by language and by test.
>   How much is unknown.

The variable portions of the startup/shutdown time should be 
calculated into the test score in many tests.

Consider two languages A and B. A initializes the thread subsystem
before main and B when it is first used. You would like to penalize B.

> - test startup/shutdown times can be guessed by noninvasive means,
>   but only 2 invasive timers can prove them

But the information can be worthless (see previous).

- Einar Karttunen