[Shootout-list] Missing programs and score calculation (Was: Re: TinyCC and Io)

[skaller]

On Wed, 2005-09-14 at 15:53 -0400, grfgguvf at gmail.com wrote:

> c) why is c fast? 

Because they won't include Felix .. which is faster:

http://felix.sourceforge.net/current/speed/en_flx_perf_0005.html

:)

> it's often said that c is fast becuase it doesnt
> offer some fancy features other languages do so programmers are forced
> to use simpler data structures and algorithms. 

Crap. C isn't faster. It only seems like it is, when implementing
trivia which is easy to implement in C. Higher level languages
are much faster than hand written C for doing more sophisticated jobs.

For example Felix, and to some extent Haskell, defeat the 
intent of some of the Shootout tests because those tests
are specified badly, and the language translators optimise
away the very code intended to be tested.

C is a very poorly structured language, and very hard to
optimise. Higher level, more advanced languages, in particular
Haskell (and perhaps Clean), but also ML style languages like
Ocaml and ML (MLton implementation) and Felix, 
are amenable to high level optimisations which C compilers 
would have no hope of performing.

Felix outperforms C/C++ even though it actually generates C++!
Why? Because it can be much smarter with optimisations, because
it has a lot more information available than a raw C program,
and because it is, in itself, written in a high level programming
language (Ocaml) which makes implementing those optimisations relatively
easy. It also knows a few tricks of particular compilers (such as gcc)
and can apply them more consistently than a human coder.

Compare with g++ which hasn't even been able to *parse* C++
correctly for over a decade .. let alone generate efficient code.
[Version 4.x finally seems to parse most of it correctly]
Why? Because it is a difficult task, implemented in a brain
dead language.

g++ uses massive amounts of memory, which simply isn't required.
Why? Because it is written in a brain dead language. It is 
very slow as well.

Compare that with the Ocaml native code compiler .. which is 
lightning fast -- both because it is written in a high
level language, and also because it is implementing one.

So when it comes to *serious* programming tasks, C very
rapidly loses its performance advantage because the ability
of programmers to write fast code is lost in managing both
the abstract complexity of the problem and solution, along
with all the implementation details.

So basically, no, C is NOT faster.

> while the shootout is an amazing work and i respect you for doing it
> in many programs its apparent that the person who wrote them didnt
> know the programming language very well. he/she just translated an
> other program "word-by-word". it would give more realistic results if
> the programs were written in the languages usual "style".

No, this is the case because the administrators chose to
provide 'do it the same way' tests despite advice that
this is nonsense.

As a result, many languages have tests which are not natural
style for that language, just so as to convince the administration
to accept the tests.

> for example in lisp one has the luxury of the built-in list data type
> so one would naturally use it while in c the same program would
> usually be implemented using a buffer and pointer arithmetics. 

Yes, but to do that the test requirements have to functional,
rather than structural, that is, they have to be specified
in a black-box format: the test should be judged entirely on
the correctness of its output (and provided it is a reasonably
short)

The ray tracing test is of this kind, AFAIK it is the only one.

> its the
> compilers task to generate the best possible machine code in the end.
> while its fair to require all programs to do the same thing, 

No it isn't, its nonsense. Trivially several of the languages
are purely functional and thus declarative and manifestly cannot
meet any requirement to 'do the same thing' as some java program
because they can't be told to 'do' anything :)

> d) some tests dont benchmark the languages implementation. for example
> the "threads-flow" description says:
>   Each program should create, keep alive, and send integer messages
>   between N explicitly-linked threads. Programs may use kernel threads,
>   lightweight threads, cooperative threads…
> so what do you want to benchmark? i thought of contributing a c
> version. but how to implement it? use posix threads? or fork 500
> processes? linux would just emulate the threads with processes. use
> rfork? what are we benchmarking? it may be the kernels fork overhead
> or the pthreads library, but doesnt have anything to do with the
> compiler or the language. (for example try creating 500 posix threads
> on freebsd5 vs fork 500 processes on linux. the language is c and the
> compiler is gcc in both cases but you will get wildly different
> results.)

Indeed. Try creating some threads on Felix. No, forget about
ridiculous numbers like 500!!

500 threads?? Are you kidding??

On my AMD64, 1Gig of Ram, running Linux: I can create
500 THOUSAND threads in 1 second.

Is that measuring context switching performance fairly?
Heck no! Almost all that time is spent in 'malloc'.
The real transaction rate is MUCH higher.

MLton can do this too, probably even faster than Felix.
Ocaml bytecode -- and even Python -- can also implement
extremely fast user space threads (although neither seems
to be as blindingly fast as Felix).

> e) hello world or ackermann sure highlight differences but those were
> already evident. what about some more realistic benchmarks?

The ray tracer is a more realistic benchmark.

>  a simple equation solver 

There is a 'standard' test, used to measure MFLOPS.
The problem is, this test is too simple, most languages
such as Python have built in matrix handling for simple
operations such as inversion etc.

A realistic test should be something *not* amenable
to using a canned solutions package, at least not
a high level one, to force the programmer to hand
code enough stuff rather than doing a couple of library
calls, so that there is actually some code in that
language to translate, so the language/translator performance
can be measured.

The problem with chosing a 'real' problem is that canned
libraries do exist for simple ones, and more complex
ones are too hard to code for a site like the Shootout.

The ray tracer is a counter-example, and there are surely
other good tests, however some thought is required to
find them, and also a consensus that it is worth doing so.

> and a simple http server 

Not a good example for two reasons: many languages come
with canned http servers, and, it is hard to measure
the performance on a single machine in a short time.

> in all the languages may take
> a little more effort to implement but will give meaningful results in
> the end. a very important property of a language is ease of
> programming in it and maintainability of the result. 

Yes.

> hello world
> doesnt measure that. 

Actually it does. Just compare Python:

print "Hello world"

with C:

#include <stdio.h>
int main(int argc, char **argv) 
{
  printf("Hello world\n");
  return 0;
}

1 line vs. 6. Felix is 2:

include "std";
print "Hello world"; endl;

and I'm sorely tempted to remove the need to explicitly
include the standard library just to get the line count
down by 1 :)

-- 
John Skaller <skaller at users dot sourceforge dot net>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.alioth.debian.org/pipermail/shootout-list/attachments/20050916/d1fecfc7/attachment.pgp