[Shootout-list] preserving historical data

[Bengt Kleberg]

Brandon J. Van Every wrote:
...deleted


> Windows 2000 and Windows XP I'd usually expect their performance to
> probably scale.  Maybe not for threading, since that is highly OS
> dependent, but definitely for most things.  It wouldn't be difficult to
> write a postprocessor for 2 datasets to see if the results do in fact
> scale.  We can't assume it, but we can detect it.
> 
> Of course my expectations could be violated.  But this is how I'd define
> 'sufficiently similar' machines.  If the results of the tests scale
> linearly between them, they are sufficiently similar.

with that definition of sufficiently similar i would also expect that 
the tests scale linearly between them.
actually i would be very surprised if this was not the case :-)


> 
> The languages, compiler versions, and tests have to be set up and run by
> a human.  The results have to make it to the website, again a human.
> The website has to be protected against catastrophic loss, again a
> human.  The test code has to be maintained by a human.  People have to
> buy new 80GB hard drives and install them and pay for web services so
> that tests happen at all.  Tests don't just magically float onto the
> internet.  We don't have that kind of strong AI yet.  ;-)
> 


first, yes i do agree that static data is easier to handle than a 
runnable test.
secondly i think that many of these actions will have to be done even 
for an archive. thirdly i hope that most of them are automated in the 
shootout. atleast i automated them in my own scheme test bench (no web 
interface, i have to admit). the maintain bit i find interesting. 
provided that the tests actually need maintaining there might have been 
something slightly wrong with them when they generated the archive. that 
might mean we have to maintain the archive. perhaps.

...deleted

> I've been avoiding the word, but I will invoke the principle of
> Curiosity.  You're not even vaguely curious how language technology is
> changing over time?


curiosity would be best served by a live test with several versions of 
the same languages available in the shootout.

i think that if we want to have comparable data in the archive that data 
needs close to the same amount of maintainance as a live test.
this is just an opinion.


bengt