[Shootout-list] Missing programs and score calculation (Was: Re:TinyCC and Io)

[Jon Harrop]

On Thursday 15 September 2005 07:43, Dave wrote:
> I don't get you here... The tests have been pared down - there used to be
> two recursive function int tests and now there is one, for example.
>
> There are four fp tests left (off hand) - one measures a simple and tight
> loop, another is OOP, another is recursion and the last uses arrays, as
> another example.
>
> Decent benchmarks suites (like this one, IMO) measure how a language
> implementation does across several tests using the same sub-component of a
> computer in different ways, like CPU perf. on integer operations, or memory
> access performance. Some often used library functionality is also 
> represented in the Shootout, which is usually missing from others.

The shootout tries to test many aspects of performance limitations. If you 
"par down" the tests too much then you only have one data point in each 
dimension and you lose the ability to draw any useful information from the 
results.

> > c) why is c fast? it's often said that c is fast becuase it doesnt
> > offer some fancy features other languages do so programmers are forced
>
> C is fast

No, it isn't. Try porting my ray tracer to C and then look back and think how 
many high-level optimisations could have been implemented in other languages 
with the same effort. C is one of the slowest compiled languages...

> because a) the more abstraction, the more overhead (and little of 
> C can be said to be abstract unless you compare it to assembly)

You've incorrectly assumed that abstraction implies overhead. In fact, 
abstraction implies more high-level information available to the optimiser. 
Hence, many abstractions improve performance in ways inaccessible to the C 
programmer.

> b) the amount of work going into the compilers and libs.

The C stdlib doesn't even expose a balanced binary tree that you can use.

> > to use simpler data structures and algorithms. see rob pikes five
> > rules of programming http://www.ece.utexas.edu/~adnan/pike.html
> > while the shootout is an amazing work and i respect you for doing it
> > in many programs its apparent that the person who wrote them didnt
> > know the programming language very well. he/she just translated an
> > other program "word-by-word". it would give more realistic results if
> > the programs were written in the languages usual "style".
>
> I think there are a few examples of this too, but I think you're
> insinuating that it is worse than it really is. There is absolutely nothing
> stopping you or anyone else from rectifying this by submitting 'better'
> code and I suspect that some people spend a lot of time 'tuning' some
> benchmarks for their favorite language.

No, there _are_ things "stopping you or anyone else from rectifying this by 
submitting 'better' code". Skaller and I have both submitted code (Skaller 
even submitted a language!) only to have it ignored, rejected for no reason 
or accepted and then deleted a little while later. This can be partially 
attributed to the fact that the benchmarks are ill defined (e.g. 
binary-tree). Skaller and I have even proposed a solution to that problem. 
I've had my access to the shootout revoked so I can't upload my submissions 
myself.

> > for example in lisp one has the luxury of the built-in list data type
> > so one would naturally use it while in c the same program would
> > usually be implemented using a buffer and pointer arithmetics. its the
> > compilers task to generate the best possible machine code in the end.
>
> It isn't all that simple to take a series of abstract statements and make
> good machine code for all of the combinations of and situations for which
> those statements can be used. For example, there are many ways to implement
> lists, but the implementors of the Lisp built-in list had to choose an
> algorithm that would perform well in many different situations for many
> different algorithms, so it would be very hard or impossible to implement
> it in such a way as to offer the best performance in every case.

You've incorrectly assumed that the Lisp compiler can't choose the most 
appropriate list representation for a given piece of code. This situation is 
no different in Lisp than in C, except the process is automated in Lisp.

> > e) hello world or ackermann sure highlight differences but those were
> > already evident. what about some more realistic benchmarks? a simple
> > equation solver and a simple http server in all the languages may take
> > a little more effort to implement but will give meaningful results in
>
> Please submit something to the sandbox (see the FAQ).

How about a ray tracer?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists