[Shootout-list] X per second scoring system, resume

[Einar Karttunen]

On 30.09 13:55, Brandon J. Van Every wrote:
> > Choosing a different N can make the order vary.
> 
> You mean like, some language dies once N hits a certain threshold?  Or
> do you mean small noisy variances?  Noise isn't important.

No I mean that cache effects (and GC) can play a quite large role. 
Cache performance will degrade as N becomes larger but it is very
implementation defined how this will happen...

> > > - invasive timers are usually more accurate, but are gruntwork to
> > > implement
> >
> > Invasive timers are more accurate depending on the language,
> > making timer scales vary with language.
> 
> Languages already vary with languages.  :-)  Any timer on the order of
> microseconds will do fine.  We're talking about START() STOP() gaps of 1
> second.  Even milliseconds is tolerable if the language is slow.

The current timer inaccurancies affect various languages in the same way,
but with timers inside the language they would affect different
languages in different ways.

I have already argued that discarding startup and stop times is not
wanted on many occasions. 

You can already measure that in a certain way. Look at the analysis
with different N. *If* you presume that N does affect the running time 
with the function f, then you will have:

f(x) = myfunc + (start + stop)
f(1000) = value_1
f(2000) = value_2
f(3000) = value_3

Now it should be trivial (knowing myfunc) to solve what 
start + stop is. I don't see a need to revamp the system...


> > > - we haven't checked if some language poses special implementation
> > > difficulties
> >
> > Yes, they do.
> 
> Example then please?

Lets look at haskell for example (I don't think it is one of the worst
ones). 

If the timer is implemented with a signal then you run into the
following problems:

* latency to handle signals
* latency of signal handler
* latency of delivering asynchronous exception to the benchmark
* noticing that there is no easy way to see where the aborted
  computation was

If you periodically call the signal handler function you have to
make it be in the IO monad to tell haskell that it cannot move it
outside the loop. 

Now most of the haskell benchmarks don't use IO monad in the inner
loop and would need to be rewritten. Did I mentioned that this will
probably affect what will get inlined?

> The crux of your argument is, you believe some languages will have
> timers so bad, that invasive timing overhead will be unacceptable in
> some cases.  I'm not ready to buy your premise yet.  What's your
> specific example?
> 
> I will also wager that all the (relatively) high-performance languages
> have a high-performance timer available.  Either natively or through a C
> call.


I think that the it is quite interesting *how* slow a language is, not
just that it is slower than some other language.

- Einar Karttunen