Brent Fulgham wrote:
> Brandon J. Van Every wrote:
>
> > I have proposed, several times now, a framework.
> > There should be:
> >
> > - a main, least common denominator benchmark for all
> > languages
>
> This is where our communication is breaking down. I
> want to know what you view as a
> least-common-denominator benchmark. You have told us
> what is *not* in the LCD -- bignums and apparently now
> string functions. Why don't you say what *is* in this
> set. Simple math? file I/O? What goes in and stays
> out?
C is the model. If you can do it in C, it's LCD. I also think the
Standard Library of any given language should be included as testable
functionality, so anything the C Standard Library can do is fair game.
By that standard, I don't see any problem with having 'some string
functions' in the LCD benchmark. The problem with the current tests is
that they're heavily lopsided towards string functions, like that's the
main job anyone's gonna have to do.
> Previously I tried to address your issue by listing
> each test and what its aim was. Bengt did a much
> better job of this with his breakdown.
>
> Assuming we agree that concurrency is not a LCD,
I agree that concurrency is not LCD.
> and I
> think we DO agree that object oriented tests should
> not be included,
I agree that OO is not LCD. For instance, various scripting languages
aren't OO, and various functional languages aren't OO.
> which remaining tests should be
> retained for BARC, and which should be left for BITE?
I don't want to discuss prejudicial labels BARC and BITE. I'm mainly
interested in discussing the LCD. I will look at Bengt's list again.
> > - a garbage collection benchmark
>
> We don't currently test this explicitly. Anyone have
> ideas on how best to measure this one issue?
I posted yesterday on comp.benchmarks asking this very question. I have
yet to receive an answer. I have not felt like digging through the
comp.benchmarks archives, although I'm sure that would yield some
benchmarking models.
> > I think you're ignoring the arguments presented to
> > you because you don't value them.
>
> Perhaps. But so far you are the only voice from this
> perspective.
Yes. I have a seemingly unique talent for showing up in open source
communities that are dominated by techies and raising business /
marketing concerns. I think this is because I have a 'Windows glossy'
developer culture, not a 'UNIX hacker' one. The vast majority of open
source communities are populated by UNIXen.
I started worrying about what open source could do for me,
business-wise, about a year ago. It has taken me this long to realize
that I usually don't share goals with other people in open source
communities, and that our cultural conflicts are inevitable. Really, I
belong in companies to do these things. Where people have well-defined
financial stakes in the outcomes. The problem is, most companies don't
do these open source things. Open source is not so new... but open
souce as a *business model* is very much pioneering work.
For instance, the idea that "The Shootout is to have fun" just makes my
skin crawl. There's no business model in that.
> I want the shootout to be as fair and
> unbiased as possible, but I don't want to change
> things just for change's sake,
> or just to satisfy one person's requests.
The problem is, who will you ask to get the different opinion on what
should be done? Just your UNIX hacker buddies? Will you actually go
and ask businesses, managers, and Decisionmakers what they really want
out of benchmark information? Will you go ask the SPEC guys what's
worked for them over the years? PC Magazine? If you don't, then your
audience is pretty self-selecting. Shootout remains a hobby toy for
UNIX hackers. Your group will be unified in its opinions of what should
be done, what's important, and what's worth bothering with. But it
won't be scientific, objective, or far-reaching. Far-reaching is, like,
when Microsoft actually cares how C# performs in the Shootout.
> I don't think we take issue with most of your
> suggestions regarding the idea of "core" tests versus
> "language-specific" or ancillary tests, but I don't
> know that we all agree on what belongs in each area.
That's nice to know. I'll stick around for a "how to classify things"
discussion.
Cheers, www.indiegamedesign.com
Brandon Van Every Seattle, WA
"The pioneer is the one with the arrows in his back."
- anonymous entrepreneur