[Shootout-list] bench mark test run time, max bad score

[Bengt Kleberg]

Brent Fulgham wrote:
> I'm not really interested in changing the current logarithmic scoring 
> system.

if we can get a (much) better system it would be worth switching. but 
the burden is on the new system to prove that it is better. imho.


...deleted

> 
> The advantage of this approach (conceived by Stephen Weeks) is that 
> high-performing tests get
> spread out a bit (amplifying the relative advantages) to help determine 
> "best".

any alternative would have to do this in a better way to be acceptable.


> If this approach is used on all machines running the shootout, the 
> scoring should be very consistent
> because the scores are based on relative ranking, which is somewhat less 
> likely to change based on
> hardware than the particular value of a given run.

note that on my machine (sparcstation running solaris) i consistenly get 
  erlang scores that are higher than mzschemes scores. this is not true 
for the computer used in the shootout. :-(


bengt