[Shootout-list] testbed normalization

[Brandon J. Van Every]

Brandon J. Van Every wrote:
>
> So, what to normalize against?  How about the performance of the C
> benchmark on any given machine?

Which again is a slightly political question.  However, I think it has a
reasonable answer: the dominant C compiler on the platform.  On Linux
that's GCC, for instance.  On Windows it's MSVC.  There's a free MSVC
compiler available sans Visual Studio GUI,
http://msdn.microsoft.com/visualc/vctoolkit2003/, so that's not an
issue.

Uuuh, but another issue is, what about when new compilers are
released???  In Windows-land, the 'mainstream' MSVC version number
changes every 3 years.  How frequently does the GCC version number
change?

And yet another difficulty, is if 1 test machine is a P-III, and another
is a P4, and you've got different compiler flags with different
optimizations, it's not really comparable to normalize the results
between the 2 machines.  You're running completely different code.  So
it would seem that test binaries should be standardized and controlled.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur