[Shootout-list] X per second scoring system

[Bengt Kleberg]

theBrandon J. Van Every wrote:
...deleted
> 
> The phrase "penny wise and pound foolish" comes to mind.  Adding a
> trivial amount of timer code is not visually a big deal.  It doesn't
> affect comprehension of the test suite at all.  If you REALLY REALLY
> REALLY CARED about visual inspectability, you'd have some easily
> recognized comment delimiters in all of the tests, to separate the 'real
> work' from the counting boilerplate.
> 
> I can respect the argument of 'it being work'.  But 'it's tough to look
> at' is just silly.

how about
not only is it work (major problem), but it also tough to look at (minor
problem). if it would give us a minor benefit i can take the
obfuscating. if it would give us a major benefit i can take the work (i
am here assuming that it would be my resonsibility to add internat
timing to the erlang tests).


>>not to mention that some
>>languages might need more than trivial. perhaps.
> 
> 
> I can't think of any reason why that would be true.  Languages either
> have a way to call timers or C functions, or they don't.  There's
> nothing complicated about it.
> 

neither can i think of any reason why it would be complicated. i do not
assume that my inablity to come up with such a reason makes it
impossible. therefore the ''perhaps''. is your comment ''or they don't''
supposed to mean that such a language is disqualified? ie, all languges
that does not have sufficiently fine grained and exact timers, nor the
ability to call c functions, are not of interest?



> 
>>the calibration of n is somebody-elses-problem.
> 
> 
> It's a problem for anyone who wants to run the tests.  Who trusts who's
> N?  What if your machine was 2GHz and mine is 4GHz?  Again I ask, how is
> N currently calibrated?  Just a Wild Assed Guess?

imho the idea of n is to make the fastest language do the test in x (1,
2, or somehting elkse) seconds. then the other languages takes longer
for the same n, thereby proving that they are weaker in performance.
different hardware/os combinations needes different n. the shootout is
only corrrect on the hardware/os combination where it was run.


>>as mentioned in one of
>>my previous emails to the list it does not matter so much
>>since we have
>>a table for each test with lots of n. this is a good thing (tm), see
>>Timing Trials, or, the Trials of Timing
>>(http://cm.bell-labs.com/cm/cs/who/bwk/interps/pap.html) as to why.
> 
> 
> Are you saying you do N calibration to detect loop optimizers?  Is this
> automated in the Shootout, or at least readily graphed and displayed?
> Otherwise the advantage is theoretical.  Also, if you're doing N
> calibrations to find linear scaling sequences, you're talking about
> doing an awful lot more benchmark runs than if you just had timers in
> the tests and didn't have to guess N.
> 

one quoute (out of many) is:

Varying the problem size helped us to detect unusual runtime effects,
while a graphical presentation highlights patterns and trends in runtime
instead of individual performance scores.

i find your question strange. you have read the paper, yes?
moreover, i also find it hard to understand the question about ''graphed 
and displayed''.
there is a graph for every test. or are you refering to some other kind
of graph?



>>as an aside: why is there no link from the shootout to Timing Trials?
> 
> 
> Is there some reason there should be?  It's a good paper, but it's not
> the Shootout.  Where do you  think such a link should appear?

i think the link is a good thing (tm) since the original shootout was
inspired by it. see http://www.bagley.org/~doug/shootout/ack.shtml. it
will also explain why some of the tests exists.

when this shootout becomes sufficiently different from the original 
shootout (by introducing timers in all the test :-), i think it will be 
ok to (exageration) ''hide'' the link to Timing Trials inside a single 
test case.

right now this shootout and the original shootout are close enough 
(imho) for this shootout to link in the same way as the original.
imho.


bengt