[Shootout-list] Missing programs and score calculation (Was: Re: TinyCC and Io)

[grfgguvf at gmail.com]

On 9/14/05, Isaac Gouy <igouy2 at yahoo.com> wrote:
> --- grfgguvf at gmail.com wrote:
> > hello what about adding tinycc to the shootout as well?
> > http://tinycc.org/
> > also there is an interesting new unused language called io that id
> > like to see in the shootout: http://www.iolanguage.com/
> Why don't you include language X?
> http://shootout.alioth.debian.org/faq.php?sort=fullcpu#acceptable
> 
> FAQ Suggest new features, new languages...
> http://shootout.alioth.debian.org/faq.php?sort=fullcpu#help
> 
> Please will you include my favourite language?
> http://shootout.alioth.debian.org/faq.php?sort=fullcpu#please

ok i see

a) an other thing i wanted to ask is why have this many benchmarks?
most of them measure the same thing and with many benchmarks the less
popular languages will always be behind while one of the goals of the
shootout is to promote those languages right?

b) it seems to me that missing programs dont contribute to a languages
score. that may even be fair (it shows there are few programmers who
know/want to write in the language) it may be better to only include
those benchmarks in score calculation which have programs in all
languages or
use the average score of all the written programs in a language for
the missing program(s).

c) why is c fast? it's often said that c is fast becuase it doesnt
offer some fancy features other languages do so programmers are forced
to use simpler data structures and algorithms. see rob pikes five
rules of programming http://www.ece.utexas.edu/~adnan/pike.html
while the shootout is an amazing work and i respect you for doing it
in many programs its apparent that the person who wrote them didnt
know the programming language very well. he/she just translated an
other program "word-by-word". it would give more realistic results if
the programs were written in the languages usual "style".
for example in lisp one has the luxury of the built-in list data type
so one would naturally use it while in c the same program would
usually be implemented using a buffer and pointer arithmetics. its the
compilers task to generate the best possible machine code in the end.
while its fair to require all programs to do the same thing, thats not
what would happen in real life.

d) some tests dont benchmark the languages implementation. for example
the "threads-flow" description says:
  Each program should create, keep alive, and send integer messages
  between N explicitly-linked threads. Programs may use kernel threads,
  lightweight threads, cooperative threads…
so what do you want to benchmark? i thought of contributing a c
version. but how to implement it? use posix threads? or fork 500
processes? linux would just emulate the threads with processes. use
rfork? what are we benchmarking? it may be the kernels fork overhead
or the pthreads library, but doesnt have anything to do with the
compiler or the language. (for example try creating 500 posix threads
on freebsd5 vs fork 500 processes on linux. the language is c and the
compiler is gcc in both cases but you will get wildly different
results.)

e) hello world or ackermann sure highlight differences but those were
already evident. what about some more realistic benchmarks? a simple
equation solver and a simple http server in all the languages may take
a little more effort to implement but will give meaningful results in
the end. a very important property of a language is ease of
programming in it and maintainability of the result. hello world
doesnt measure that. in fact its the second most important property of
a language, in some cases the first. otherwise everyone would still
hand-code assembly. thats the whole point of the shootout, to prove
that all these languages have a reason to exist.