[Shootout-list] Missing programs and score calculation (Was: Re:TinyCC and Io)

[Dave]

>
> ok i see
>
> a) an other thing i wanted to ask is why have this many benchmarks?
> most of them measure the same thing and with many benchmarks the less
> popular languages will always be behind while one of the goals of the
> shootout is to promote those languages right?
>

I don't get you here... The tests have been pared down - there used to be
two recursive function int tests and now there is one, for example.

There are four fp tests left (off hand) - one measures a simple and tight
loop, another is OOP, another is recursion and the last uses arrays, as
another example.

Decent benchmarks suites (like this one, IMO) measure how a language
implementation does across several tests using the same sub-component of a
computer in different ways, like CPU perf. on integer operations, or memory
access performance. Some often used library functionality is also
represented in the Shootout, which is usually missing from others.

> b) it seems to me that missing programs dont contribute to a languages
> score. that may even be fair (it shows there are few programmers who
> know/want to write in the language) it may be better to only include
> those benchmarks in score calculation which have programs in all
> languages or
> use the average score of all the written programs in a language for
> the missing program(s).
>

This policy is utterly pragmatic - 1) encourages people to contribute, learn
and teach (which are the real goals of the Shootout, I think) 2) otherwise
the only submission(s) you'd get would be for what an implementation happens
to be real good at.

> c) why is c fast? it's often said that c is fast becuase it doesnt
> offer some fancy features other languages do so programmers are forced

C is fast because a) the more abstraction, the more overhead (and little of
C can be said to be abstract unless you compare it to assembly) b) the
amount of work going into the compilers and libs. due to market demand c) it
was originally constructed to be 'easily' translated to assembly and allow
coding 'close to the metal' (really the abstraction thing again).

> to use simpler data structures and algorithms. see rob pikes five
> rules of programming http://www.ece.utexas.edu/~adnan/pike.html
> while the shootout is an amazing work and i respect you for doing it
> in many programs its apparent that the person who wrote them didnt
> know the programming language very well. he/she just translated an
> other program "word-by-word". it would give more realistic results if
> the programs were written in the languages usual "style".

I think there are a few examples of this too, but I think you're insinuating
that it is worse than it really is. There is absolutely nothing stopping you
or anyone else from rectifying this by submitting 'better' code and I
suspect that some people spend a lot of time 'tuning' some benchmarks for
their favorite language.

> for example in lisp one has the luxury of the built-in list data type
> so one would naturally use it while in c the same program would
> usually be implemented using a buffer and pointer arithmetics. its the
> compilers task to generate the best possible machine code in the end.

It isn't all that simple to take a series of abstract statements and make
good machine code for all of the combinations of and situations for which
those statements can be used. For example, there are many ways to implement
lists, but the implementors of the Lisp built-in list had to choose an
algorithm that would perform well in many different situations for many
different algorithms, so it would be very hard or impossible to implement it
in such a way as to offer the best performance in every case. It's really
good to know that if you need a list algorithm that pre-allocates alot of
memory (in order to run fast for a certain task), you can write a custom
lib. in C if you really need to - otherwise Lisp is good enough.

> while its fair to require all programs to do the same thing, thats not
> what would happen in real life.
>

See the response to a) above - that's why there is a broad range of tests
and algorithms represented.

> d) some tests dont benchmark the languages implementation. for example
> the "threads-flow" description says:
>   Each program should create, keep alive, and send integer messages
>   between N explicitly-linked threads. Programs may use kernel threads,
>   lightweight threads, cooperative threads…
> so what do you want to benchmark? i thought of contributing a c
> version. but how to implement it? use posix threads? or fork 500
> processes? linux would just emulate the threads with processes. use
> rfork? what are we benchmarking? it may be the kernels fork overhead
> or the pthreads library, but doesnt have anything to do with the
> compiler or the language. (for example try creating 500 posix threads

The results, on the same machine, are still wildy different for different
languages, which is the point. I think it's pretty obvious what it is trying
to measure: How efficiently an implementation can create, use and dispose of
many threads, which of course can be important for a lot of server software.
It would be nice to maybe see more rational for why a test was included in
the About section though.

> on freebsd5 vs fork 500 processes on linux. the language is c and the
> compiler is gcc in both cases but you will get wildly different
> results.)
>
> e) hello world or ackermann sure highlight differences but those were
> already evident. what about some more realistic benchmarks? a simple
> equation solver and a simple http server in all the languages may take
> a little more effort to implement but will give meaningful results in

Please submit something to the sandbox (see the FAQ).

> the end. a very important property of a language is ease of
> programming in it and maintainability of the result. hello world
> doesnt measure that. in fact its the second most important property of
> a language, in some cases the first. otherwise everyone would still
> hand-code assembly. thats the whole point of the shootout, to prove
> that all these languages have a reason to exist.