[Shootout-list] CVS updates not visible on the website

[Isaac Gouy]

This follows on from Greg's post and to some extent Simon's post

> It seems that CVS updates are not automatically transfered to the
> website with the new layout. 

(afaict They weren't automatically transfered with the old layout.)

1. The source code has been updated in CVS on the Alioth server.
2. The program has not been built & measured on Brent's machine.
3. After that, the build logs and measurements would still need to be
transfered from Brent's machine to the Alioth server.

The immediate issue is that Brent is unavailable. 
Let's hope he's enjoying some family time.

Now what are the process issues?

1. I don't know how we can avoid someone looking at the code to check
that it does indeed implement the benchmark, instead of returning the
correct value from a print statement. There will always be a code
inspection. Let's assume that whoever does CVS checkin takes that
responsibility.

2. We need computer time to run the benchmarks (Brent doesn't have a
dedicated machine to run these benchmarks, does anyone else?)

3. We need to compile and run programs - would you want some unknown
programs to be automatically compiled and run on your computer? Maybe
whoever does CVS checkin takes that responsibility, and maybe the owner
of the test machine will always want to double-check ;-) 

Maybe we could figure out a heavy duty chroot jail.

4. We need to install new language implementations and update old
implementations. That's always going to be time consuming. That's one
of the issues with C# Mono and Fortran. 
Figuring out how to build and run new language implmentations is always
going to be time consuming.


So what can we automate?

1. Timestamp check for CVS content without measurement

2. Build process for new benchmarks - can we have a single makefile
that  checks if tools/programs are available for
language-implementation/benchmark-program-X and if so queues build and
measurement. (afaict each benchmark has it's own makefile which is
manually updated.) 

3. automate nightly build/measurement for x hours

4. automate sweep of build logs, measurements, html content, to website


> What procedures should be performed whenever I update one benchmark? 
> e.g. the new fibonacci for ghc.

Should we add alternatives rather than overwrite a program? Maybe the
new version is slower on Brent's test machine!

(Of course, as we're using CVS we always have the old source code - but
we don't have the old measurements, so we aren't able to compare the
time of the old and new version.)

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com