[Shootout-list] Directions of various benchmarks

[Bengt Kleberg]

On 2005-05-27 15:24, John Skaller wrote:
...deleted
> Um .. I need to be able to run the tests and have them
> finish in an 'estimable' and 'controllable' time, so my
> computer is useful for things other than running tests.
> 
> Exactly how to control that -- for either my tests or the
> Shootout -- is a difficult issue I think. 

if something is difficult to get exactly right it is sometimes ok to 
start with almost right, and see if that is sufficient. in this case it 
would be possible to assume that we have x seconds (3600?) and divide 
with tests*languages => maxtime seconds (or maybe maxtime*2). not very 
exact, but perhaps ok?


...deleted
>>do you mean result as in the current 3 metrics (time, memory-usage and 
>>loc) or result as in the result of a test?
> 
> 
> Good question.

i propose that the result is not the metrics (just to get names sorted 
out). the metrics are always of interest for a successfull test. the 
result is only of interest once per test (ie, the result is the same for 
all languages).
if the test fails, then the result is of interest (to help with finding 
the error). no metric is of interest.


...deleted
> 
> The initial design wouldn't permit that, because it doesn't make
> sense to compare 'gcc' on one test with 'gcc -O3' on another.

i agree, but the current shootout does not. so if this design is 
supposed to lead us towards a better shootout this proposal should not 
make too much sens (supposed to be slightly funny).


...deleted
> 
> Good point: you're saying the existing minibench already produces
> enough data, it just needs to be collated.

i was not trying to say that. it could be true, none the less.
i weas trying to mention that the shootout has many uses, but the main 
use is to compare all languages at their best. so that should be the 
simplest data set to get hold of.


>>one thing i consider important is to have a per language based 
>>configuration. i would appreciate if i could add flags for a new 
>>language without editing a file with lots of other language flags in it.
> 
> 
> Do you mean 'translator' rather than 'language'?

there are cases where the run time is influenced by flags passed to the 
byte code virtual machine. when you say translator i think of the 
compiler that produced the byte code.
this could be a problem on my part of understanding what names mean.


bengt