[Shootout-list] demoting marginal languages

[Bengt Kleberg]

Brandon J. Van Every wrote:
...deleted
> If the penalties are really really big, such that a language with an
> 'incompleteness penalty' always loses to a language with complete tests,
> then you might as well just call it 0 anyways.  The only difference
> would be alphabetical ranking for the losers (in the case of 0) vs. some
> fudge meaningless half-baked pseudo-performance ranking (for exceedingly
> low but nonzero scores).  I suppose in the latter case, you'd be
> measuring the completeness of the testing suite rather than the
> performance proper.  It strikes me as a big discontinuity in the
> testing.

1 we have yet to decide if a good performance means a low score or a 
high score. or has that already been done?
2 yes, i did intend for the missing test to be awarded maximum bad 
score, giving this language a loss to a language with complete tests. 
this would save us the trouble of having to invent a separate 
'Provisonal' category.


> Why not just keep such languages in a separate 'Provisonal' category,
> decline to award scores, and note the number of tests they fail?  A
> 'Provisional' category might motivate some people to work on moving the
> language out of 'Provisional', by completing the tests.  Or maybe call
> it 'Purgatory', for kicks.

with the new name i think it starts to seem fun to to have a separate 
'Purgatory' category. :-)


...deleted
> I admit I have a RISC-like design mentality for these sorts of things.
> I'd want to see orthogonal coverage, not like 5 different string
> handling tests.  Unless such tests are regarded as 1 aggregate unit, not
> 'separate tests' as far as scoring and weighting are concerned.  I think
> orthogonal areas of testing should be given equal weight.
> 
> I haven't decided whether this orthogonal design sensibility is
> achievable in the real world.

in a way it is. we could have one test for all string handling. that 
test would then be subject to debate about how much of it that should be 
  set aside for the different string handling primitives that exist.
do we want
10% execution time for string add
20% execution time for string search
20% execution time for string regexp
perhaps string reverse search should be counted in string search? etc? 
hours of endless fun awaits :-)
the good thing with the very more difficult to administrate approach of 
having several string tests is that we could then weight the various 
results much more easily.


bengt