[Shootout-list] testbed normalization

[Brandon J. Van Every]

Isaac Gouy wrote:
>
> > On Linux that's GCC, for instance.
> > On Windows it's MSVC.
>
>    Intel C++ for Linux, Windows
> http://www.open-mag.com/754088105111.shtml

It may perform better, but it is not the dominant compiler on either
Windows or Linux.  The point was 'which compiler to normalize against',
not 'which compiler is fastest'.  Also GCC and MSVC have versions freely
available.  I presume Intel C++ offers no more than a trial period.

Maybe you are implying 'it supports 2 platforms', and that this an
advantage.  I don't think so.  Different OSes are still different OSes,
so it's not like all things are held constant across platforms.  What if
this compiler doesn't support the same features on both platforms, or
they are released / updated / bugfixed at different frequencies?  What
if platform binary formats affect performance, either for legitimate
reasons or boneheaded reasons?  I can see plenty of reasons why one
might get different results using Intel C++ under Linux or Windows.  We
can't just state that Linux Intel C++ and Windows Intel C++ have the
same score, we have to measure those scores.  Thus they might as well be
completely different compilers from completely different vendors, even
if in practice they turn out to yield similar scores.

Also, this is only applicable to Intel HW.  I know that Intel machines
are what we personally have available now, but in the future (presuming
success at popularization), there will be AMD machines, PowerPC
machines, SPARC machines, MIPS machines, etc.  Thus again, no inherent
advantage in 1 compiler covering 2 Intel platforms.

I think we're leading to a general problem that results on a given HW +
OS testbed are not, strictly speaking, comparable to results on another
HW + OS testbed.  If we normalize across testbeds, we are lying to a
greater or lesser degree.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur