[Shootout-list] Directions of various benchmarks

[José Bollo]

Le mercredi 18 Mai 2005 06:37, skaller a écrit :
> On Wed, 2005-05-18 at 02:26, José Bollo wrote:
> > > Yes I know, but if you apply this 'external library rule' it all
> > > boils down to some arbitrary distinctions, and you end up,
> > > instead of comparing PCRE with PCRE with PCRE with PCRE,
> > > comparing PCRE with PCRE with FAIL and FAIL. Again not
> > > really interesting is it?
> >
> > I vote for the 'dont use external library rule' because think about all
> > the problems that come when you need to include external libraries.
>
> Perhaps the rule is a good rule, however that is not
> the real issue here (I'll answer it separately). The argument is:
>
> Comparing Perl/PCRE Ocaml/PCRE C/PCRE XYZ/PCRE is not very interesting
> because all you're doing is comparing PCRE with PCRE. If you then apply
> the external library rule so you're comparing Perl/PCRE
> with Python/PCRE with  Ocaml/No regexp library with
> C/No regexp library .. then the comparison
> is STILL not very interesting.
>
> It isn't about the external library rule, since as indicated
> above that doesn't make any real difference.

maybe i have readen too badly and do fastly
sorry for that. accept apologies.

> The issue is about Shootout test direction: requiring
> things be done "THE SAME WAY" is *ENTIRELY BAD"
> because it is self defeating. Things that work the
> same way will give the same results and are not
> interesting to compare, those systems than don't
> support "THE SAME WAY" will simply fail, which contributes
> very little other than boolean data.
>
> Actually I think boolean conditions  'supports this feature yes/no'
> are worth including in language comparison AND in the shootout,
> but each one isn't a benchmark -- the whole list of yes/no
> answers is a single 'feature comparison' and there's not
> really any issue of measuring performance.

the have feature yes/no is very accurate for the shootout.
if many languages use the same library, then they have the same results in 
time. but some languages may use a different library.

> Bottom line of argument: "same way" tests make no sense
> because by their very nature they defeat the whole purpose
> of comparisons, to compare how similar problems are
> solved in DIFFERENT ways .. and what the consequences
> for performance, code clarity, ease of use, etc, are.

that would be a long debate, here is my humble opinion: relaxing the 'same 
way' will improve the competition and then every implementer will copy the 
algorithm of the fastest realisation. So the 'no same way' will tend to 'same 
way'. BUT what will happen for languages where there is no competitor? Their 
performances will reduce in the time. Not so good then because one has to 
watch each an other to be in.

josé