[Shootout-list] Re: Shootout-list digest, Vol 1 #208 - 14 msgs

[Dave]

> Date: Thu, 17 Mar 2005 14:23:46 -0800 (PST)
> From: Brent Fulgham <bfulg@pacbell.net>
> To: shootout-list@lists.alioth.debian.org
> Subject: [Shootout-list] External Libraries
>
> Until recently, I was safely able to ignore the whole
> question of external libraries.  However, it appears
> that it is time for us to decide how we want to handle
> this.
>
> On the one hand, reusable libraries are the
> cornerstone
> of Software Engineering.  It makes no sense to
> reinvent
> the wheel when excellent tools exist to solve our
> problems.
>
> However, the language shootout is designed to show
> the performance of different languages for a variety
> of problem types.  We don't really learn much we end
> up with a bunch of entries calling the same external
> library to present the results.
>
> So, I propose the following:
>
> 1.  All shootout solutions must be written using the
> native provisions of the language (no external
> libraries).
>
> 2.  Solutions using external libraries are encouraged,
> and will be shown on the site as "Interesting
> Alternative" solutions (see
> http://shootout.alioth.debian.org/benchmark.php?test=nbody&lang=all&sort=fullcpu
> for an example of how this will look).
>
> Given those two rules, we have a few complications:
>
> a)  Many language use an external library (such
> as libgmp) internally.  For example, I know this is
> the case with MzScheme (it is built using libgmp).
>
> b)  What about external libraries written in the
> host language (e.g., Lua's BigInt library)?
>
> c)  What about external libraries that are simple
> wrappers around a non-host language core (e.g.,
> Pythons NumPy extension?)
>
> My judgement on these three cases are:
>
> *  Languages that fall into category (a) luck out --
> if the source for the solution does not load an
> external library (of any kind) the solution is
> perfectly fine.
>
> *  Solutions that fall into category (b) are also
> fine.  The only 'cheat' is in lines of code, and
> that is pretty meaningless anyway.
>
> *  I'm afraid category (c) should be placed in the
> "Interesting Alternative" category, because they
> provide their performance by exiting the host
> language and running assembly or some other language.
>
> ===============================================
>
> So, what does this mean?
>
> Well, at least a few existing solutions need to
> move into the "Interesting Alternative" category.
> I know that Clean's regexp solution calls out to
> 'libpcre', and D's pidigits using 'libgmp'.  There
> are probably others.
>
> I know that these (arbitrary) rules are not indicative
> of how a language would generally be used in the
> real world.  However, I'm trying to balance two
> differing uses for the shootout:  (1) as an objective
> measure of the performance of languages for different
> categories of problems, and (2) as examples of how
> a given language can be used to solve a particular
> problem.
>
> I think (1) is best served by restricting solutions
> to those in the host language.  I think (2) is
> satisfied by showing "Interesting Alternatives" to
> a problem.
>
> I invite your comments (and criticism) on this
> approach.
>
> Thanks,
>
> -Brent

Just to clarify what my understanding of the proposal is:
a) a 'ban' of external libraries only (as opposed to runtime libs. that are 
included with a base implementation using default link switchology), and
b) you're also speaking of including only the primary tests (not 
'interesting alternatives') in the "default CRAPS rating", so the results at 
a glance reflect the more 'pure' implementations.

If that's correct, then I think you've come up with a great compromise. I 
was wondering about this very thing recently..

If I may suggest a couple of other things (if these have already been argued 
over previously than I apologize and withdraw.)..

'Ban' inline assembler in any of the actual implementation code (not, of 
course, in the runtime libs.), in case anyone gets any ideas <g>

And - much less importantly - include in the LOC count things like 
non-library headers that are really part of the implementation code in 
disguise :) For example, C/simple_hash.h.

This could be interesting even wrt. to some of the g++ vs. Intel C++ 
implementations (i.e.: by these rules, someone would have to roll their own 
hash class, or use std::map or whatever).

- Dave