[Shootout-list] HW + OS database

[Brandon J. Van Every]

Brandon J. Van Every wrote:
>
> I think we're leading to a general problem that results on a
> given HW +
> OS testbed are not, strictly speaking, comparable to results
> on another
> HW + OS testbed.  If we normalize across testbeds, we are lying to a
> greater or lesser degree.

The only objective way I'm seeing out of this quagmire, is to keep a
database of benchmark submissions for various HW + OS testbeds.  In such
a database, you'd look for machines that are "sufficiently close" to
your own target configuration.

The benchmarks would have to be user-submitted as we don't have every HW
+ OS combo under the sun.  This in turn opens an issue of trust and good
faith.  It might be hard to track down cheaters if the database becomes
large.  I think Futuremark has probably had these issues, as they take
tons of user submissions.  Also the storage, network bandwidth, and
support code for tons of submissions is a big project.  This is a better
agenda for a corporate backer at some future date.  Maybe one of you
guys decides you like this benchmarking stuff so much you go into
business, using all that MIT starter code.  :-)

I guess the "poor man's solution" is to build the abstract capability,
but keep it to the test machines actually under our control.  Maybe 1
particular machine is "the standard profile," but the profiles of other
machines are also available.  The user would eyeball whatever he thinks
he's most interested in.  To please the PHBs, "the standard profile"
should be a current PC of some kind, if someone has one.  I don't,
myself.  I do think old machines are worth testing, however.  Installed
base is very important to a game developer, and slower machines do tend
to make performance differences more noticeable.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur