Fwd: Re: [Shootout-list] Science-related benchmarks (speedoptimisation)

[Brent Fulgham]

--- Dave <davejf@frontiernet.net> wrote:
> I completely disagree.. It is common and widespread
> to suit compiler switches to the job at hand, or
> even to the stage of development (debug vs.
> 'gold' code for example) of the software. That's
> what the switches are intended for.

I would like to keep the compiler switches, because
it provides interesting information about what
compiler switches do, and how they affect the
output.

When C/C++ hackers tell me to enable compiler option
X or Y for a specific test, it provides useful
data for other site visitors that they can then use
on their own programs later.

I worry that limiting the flags is pushing us into
a useless rigidity that makes the shootout interesting
for a narrow, theoretical view but effectively
worthless for 'real world' developers.

> It's the job of the compiler implementers to make 
> sure it compiles the software to function in the
> same way given the same input with different 
> switches.

Indeed, a few compiler errors have been caught
(thanks to the shootout, no less!) because of this.

Plus, if I have the huberis to believe the shootout
is on the radar of compiler writers, it is reasonable
to expect that they might modify the compiler to
act as though -O3 (or similar) is enabled by
default, so that they will continue to see good
shootout performance.

Let's not go down this road.

John Said: 
> > If you just want to measure 'C', with the best
> > tool, then only one of Intel C and gcc compilers
> > should be allowed for any particular test. After
> > all, the main difference is in the optimising 
> > behaviour of the back ends, which varies far more
> > by switches on one tool than between the tools .. 

John is right.  And the big flaw in the shootout
is that we are not measuring langauges, per se,
but implementations.  But I say, "so what".  It's
like Fabian Pascal's work in Databases.  Sure it
all makes sense in theory, but in practice people
can't design their databases in his style without
suffering performance problems.  The RDBMS's just
aren't "correct" enough with respect to the
relational calculus.

-Brent