[Shootout-list] main benchmark

[Brandon J. Van Every]

Brent Fulgham
>
> Insofar as the goal of understanding features and
> performance match your so-far-unspecified commercial
> goals, great.  But if your idea is that we discard
> various data points because they don't fall into the
> matrix of ideas you want to get across to some
> audience I think we may have incompatible goals.

I have proposed, several times now, a framework.  There should be:

- a main, least common denominator benchmark for all languages
- a garbage collection benchmark
- a concurrency / threading benchmark
- feel free to propose other MAJOR categories of benchmark

This is not 'discarding' information.  This is classifying information
more appropriately.  The current CRAPS system is not appropriate.  It is
an unfair competition for languages that don't have OO features, and it
overweights string handling, for instance.

> I guess I view the Shootout as an impartial experiment
> that attempts to advance the state of the art, and to
> help provide a framework for language comparison.  I
> have every reason to believe that this kind of work
> has value to the world at large, and have not seen any
> convincing argument to the contrary.

I think you're ignoring the arguments presented to you because you don't
value them.

> I took a quick look at the Viewperf stuff at this site.
>
> What are they providing that we are not?

Consistency of scoring system.

> Is it just
> that you want there to be a single score that can be
> compared for all languages?

Yes, the *MAIN BENCHMARK* should be a fair contest for *ALL* languages.

Secondary benchmarks would test garbage collection, concurrency,
numerical accuracy, etc.  They would have secondary scores.  The
secondary scores wouldn't be the 1st numbers waved in front of some
PHB's nose.

> What if we provided a BARC (Brandon's Arbitrarily
> Reduced Criteria) screen that showed each languages'
> score would that be sufficient?

Because the point is to reach consensus on what actually *would* be a
fair contest.

Let me put it in the simplest terms I can.  Let's say you were devising
the rules of a contest.  You'd have to agree about exactly what the
rules are, right?  Let's say BYTE Magazine was interested in using the
Shootout as a basis for comparing languages.  Is the Shootout ready for
this?  No, it isn't.  It isn't a basically fair, consistent benchmark.

In fact, philosophically speaking, it completely ducks responsibility
for the issue.  It only offers a Completely Random and Arbitrary Point
System (CRAPS).  Which actually is the de facto MAIN BENCHMARK, as it's
the default.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

"We live in a world of very bright people building
crappy software with total shit for tools and process."
                                - Ed McKenzie