[Shootout-list] main benchmark

[Brandon J. Van Every]

Isaac Gouy wrote:
> Brandon Van Every wrote:
> >
> > The *MAIN BENCHMARK* should be language neutral.
>
> No such thing.

Fine.  The *MAIN BENCHMARK* should only test the least common
denominator of all languages.  It cannot, for instance, mandate garbage
collection.  3rd generation capabilities, i.e. like C, are the only
things all of the languages have in common.

> > Industry benchmarks aren't driven by "roll your own."
>
> Aren't they just "roll your own" benchmarks that have been
> around for a while?

I hope this debate is not about to take a completely exasperating turn.
I will say, "No," and ask you to explain why you think otherwise.

I am most familiar with Viewperf... it has never been left up to the
individual benchmark runner to decide how things are scored.  Companies
sat down on Day 1 and defined what the standard was going to be.  Such
standards usually produce a 'fair' contest, although the rules can be
abused.  When they are, the OpenGL ARB votes on how to correct the
problem.

Over the years I've seen every industry benchmark change its scoring
method.  That's the natural evolution of a benchmark.  But, they've
never done anything as loosey-goosey as leave the scoring system up to
the person running it.  I'm trying to imagine the PC Magazine article
that would recommend, "Eh, well, you're a smart hacker... look at
whatever you feel like.  We're sure you'll pick a good PC...."

I hope, rather than make 'subtle' arguments that bore us all to death,
that if you're really opposed to standardized scoring systems, you just
outright say so.  And maybe some reasons why.  I think it would save us
a lot of hassle to agree to disagree, since what I'm advancing here is
hardly rocket science.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

"We live in a world of very bright people building
crappy software with total shit for tools and process."
                                - Ed McKenzie