[Shootout-list] preserving historical data

[Brandon J. Van Every]

Bengt Kleberg wrote:
> Brandon J. Van Every wrote:
> ...deleted
> > Some HW+OS platforms are sufficiently similar to make
> comparisons.  For
> > instance, I feel comfortable comparing the performance of
> any 2 Intel
> > boxes on Windows 2000 and Windows XP.  For a lot of kinds
> of tests, it's
> > not going to be that different.
>
> let me check that i have misunderstood by asking you to
> confirm what i
> mistakenly belive you to have written:
> would you feel comfortable comparing the performance of any 2 Intel
> boxes, no matter their hardware (apart from the intel bit),
> as long as they where running Windows 2000 and Windows XP?

Ok, 'any' is too strong a word.  I would care that the Intel boxes are
at least 800 MHz and have at least 256MB of RAM.  But otherwise, between
Windows 2000 and Windows XP I'd usually expect their performance to
probably scale.  Maybe not for threading, since that is highly OS
dependent, but definitely for most things.  It wouldn't be difficult to
write a postprocessor for 2 datasets to see if the results do in fact
scale.  We can't assume it, but we can detect it.

Of course my expectations could be violated.  But this is how I'd define
'sufficiently similar' machines.  If the results of the tests scale
linearly between them, they are sufficiently similar.

> ...deleted
> > just don't see why you think old data should be thrown
> away.  It's a lot
> > of work to acquire such data, and it isn't expensive to
> archive.  If the
>
> since it is a computer that is generating the data, i fail to
> see how it is hard work to generate it. and i am lazy.

The languages, compiler versions, and tests have to be set up and run by
a human.  The results have to make it to the website, again a human.
The website has to be protected against catastrophic loss, again a
human.  The test code has to be maintained by a human.  People have to
buy new 80GB hard drives and install them and pay for web services so
that tests happen at all.  Tests don't just magically float onto the
internet.  We don't have that kind of strong AI yet.  ;-)

> > I think you underestimate the degree to which people form
> impressions
> > about things based on their experiences of several years ago, rather
> > than what's true today.  It's better to be able to say,
> "No, look, that
> > was true 4 years ago when you last looked at it.  Here's the data."
>
> in this case i think that an archive would make it more likely for
> everybody concerned to ignore what is true today.

That strikes me as a Wild Assed Guess on your part.  It's just more
data, to be regarded by however people's individual brains actually
work.  I say, Before and After is a way to show that technological
change has occurred.

I've been avoiding the word, but I will invoke the principle of
Curiosity.  You're not even vaguely curious how language technology is
changing over time?


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur