[Shootout-list] demoting marginal languages

[Brandon J. Van Every]

Bengt Kleberg wrote:
>
> 2 yes, i did intend for the missing test to be awarded maximum bad
> score, giving this language a loss to a language with complete tests.
> this would save us the trouble of having to invent a separate
> 'Provisonal' category.

In general, I don't think we should be looking to avoid 'trouble' if
we're talking about small amounts of work.  I think laziness should be
applied to big things, not little things.

That said, I'm not sure if a 'Provisional' category achieves what I have
in mind.  I'm thinking, it would be good if there's some way to
encourage people to complete the tests for a given language that lacks
them.  If a 'Provisional' category is very large, if there are a lot of
entries, they all get lost in a sea of noise.  I don't know if this
encourages anyone to work on anything.  They might get lost in a sea of
hopelessness.

Another scheme I thought of, is to list languages as follows:

- rank the languages that complete all the tests
- rank the languages that complete all but 1 test
- rank the languages that complete all but 2 tests
- rank the languages that complete all but 3 tests...

This is fair if the tests are orthogonal and equally weighted.  If they
aren't, then I suppose one could do a similar thing based on "% weight
incomplete."  For instance, if a language can't run various tests that
are responsible for 40% of the score, then it is demoted accordingly.
Thus a language that is 5% incomplete would be ranked higher than
languages missing more important tests.

This way, languages that are just shy of completion might bubble up to
the top.  Someone might see them and do something about them.  On the
other hand, if there are many languages with complete tests, this
bubbling doesn't occur.  So perhaps a 'Provisional' category *and* this
completion ranking system is desireable.

I have also realised, however, that this is based on a RISC-like
orthogonal philosophy of few tests.  Another way to go is to have many
tests, so missing a test here and there isn't considered to be a big
deal.  One could pick up the most points by completing all the tests,
but one needn't do so.  In some ways this might be easier on people in
practice and would provide sort of a "free market mechanism" for what
tests are worth it in the real world.  That said, many factors can skew
what tests get written.  I don't think we can escape our role as judges
and arbiters of what tests should be selected.  I'm just noting a
different philosophy of "lots of little tests."

> with the new name i think it starts to seem fun to to have a separate
> 'Purgatory' category. :-)

I do like 'Purgatory'.

> > I haven't decided whether this orthogonal design sensibility is
> > achievable in the real world.
>
> in a way it is. we could have one test for all string handling. that
> test would then be subject to debate about how much of it
> that should be
>   set aside for the different string handling primitives that exist.
> do we want
> 10% execution time for string add
> 20% execution time for string search
> 20% execution time for string regexp
> perhaps string reverse search should be counted in string
> search? etc?
> hours of endless fun awaits :-)
> the good thing with the very more difficult to administrate
> approach of
> having several string tests is that we could then weight the various
> results much more easily.

Such breakdowns would indeed be similar to Viewperf weights for a given
app.  As you said, endless hours of fun about who gets to decide.

I'm not a string guy, so I can't comment on this one.  I've judiciously
avoided string handling my entire programming career.  Except back with
the Atari 800, where I used them to manipulate 2D graphics!  That wasn't
my brilliance or anything, it was a common technique at the time to
retarget strings at various places in memory in order to move things
around on the screen.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur