[Shootout-list] major comparison categories

[Brandon J. Van Every]

Brent Fulgham wrote:
> Brandon J. Van Every <vanevery@indiegamedesign.com> wrote:

Incidentally, why is my e-mail address appearing in cleartext?  I do not
particularly want to make it easier for all the spam harvesters out
there, and I'm sure others feel similarly.

> > Here we have a fundamental difference of philosophy.  You think the
> > shootout is about each 'hacker' having a 'fun time' evaluating
> > languages
> > according to personal preference.  I think the Shootout is about
> > 'PHBs'
> > looking up 'winning scores' to decide whether to approve the use of
> > some
> > newfangled language.  The latter goal requires consistency.
> > Managers
> > don't sit around customizing stuff, they look at the
> > marketing numbers and say, "Ok, that's a winner."
>
> Do your PHB's care about concurrency or system safety?  I find that
> most complain
> about bugs and system stability , while simultaneously
> forcing the use of languages
> like Java and C# on their development teams.

Yes, but as I said before, the *MAIN BENCHMARK* should obey a least
common denominator that all languages can actually do.  I have no
problem defining *AUXILIARY* benchmarks for things like garbage
collection, concurrency, numerical accuracy, etc.  I think a garbage
collection benchmark is the most important auxiliary scoring set we
could define.  But I do want to see *STANDARD* benchmark definitions.
Not just everyone showing up here and rolling their own.  There's
nothing wrong with providing that capability on the website somewhere,
but 'roll your own' has zero value for swaying all the PHB
Decisionmakers out there.  Decisionmakers need to see standard
competitions and hard evidence that Language A is superior to Languages
B, C, D, and E.

> > We should be conditioning the major categories of comparison that
> > people use.
>
> Okay -- I can buy that.  What categories would you suggest?

- main benchmark.  least common denominator comparison
- garbage collection
- concurrency
- numerical accuracy

I'm not convinced that numerical accuracy is 'major', though.  Feel free
to add more categories, and we can debate whether they're 'major' or
not.

'Safety' is a nebulous category that undercuts all of the above.  I'm
realizing that in the OpenGL world, performance and safety aren't
handled by the same test suites.  Performance is a Viewperf thing,
safety is an OpenGL Conformance thing.  Conformance also can't be
expressed for certain problems in an automated way, like antialiased
lines.  There are simply too many legitimate ways to draw them on a
screen.  You could have a human being manually eyeball them and pass
them YES / NO, but my point is, designing Safety tests is probably about
as difficult as designing OpenGL Conformance tests.

Designing benchmarks that measure both performance and safety
simultaneously, such as for GC, is probably a R&D area.  Discussion on
comp.benchmarks would be adviseable.  In fact, discussion of any / all
issues of what the Shootout should be is adviseable.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur