[Shootout-list] Directions of various benchmarks

[Bengt Kleberg]

skaller wrote:
...deleted
> Same way tests are semantic nonsense. The only way to
> do something 'the same way as this Java program' -- which is
> the specification of some tests right at the moment --
> is obviously to actually use THAT Java program.

i agree that same way tests are not as good as same thing. i especially 
agree that ''the same way as this Java program'' is much worse than same 
thing.


> How the heck can you judge if a solution in C or Ocaml
> or Haskell is 'the same way' as a Java program when one
> language is imperative with variables and the other
> is functional?

that is currently left to the judgment of the people with access to the 
shootout repository. from the faq:
''We will accept and reject benchmarks in a capricious, unfair, biased 
fashion.''


> This isn't a rhetorical question or philosophising: already
> Isaac had the audacity to ban the Felix solution to the
> EH test because it was orders of magnitude faster than every
> other solution, with eth excuse that "by my(skaller) own 
> admission it wasn't using the same mechanism as C++/Java" 
> -- but failed to similarly ban the C solution and 
> the Scheme solution because they
> both also use variants of the same technique -- all three

the mail list has many examples of the presence of isaac. often 
uninformative, always impolite, he is a burden everybody posting here 
has to carry.
my experince is that he is worse in private communication.


> Similarly 'concurrency' tests are in question, because
> a requirement to use Posix thread is absurd: should we
> really measure OS thread handling limitations? 

i agree that this is absurd, and the posix thread bit should be removed.


...deleted
> OTOH if you allow user space context switching, as supported by 
> several systems (including Felix and I think Haskell?), and also
> supported by Ocaml bytecode interpreter and Python .. then there's
> no way to know if you will really get concurrency .. and in the
> test cases there was none .. the Felix optimiser just eliminated

concurrency is often a result of designing with async interaction in 
mind. this is rarely possible to have in a small benchmark test of the 
kind used in the shootout. since i still want to see how other languages 
handle concurrency i think it is a worthwhile addition to the shootout.


...deleted interesting info about felix

> Example. Suppose you want to say 'sort this file using
> a bubble sort'. Do it 'the same way' at the algorithmic level.
> 
> Well you CANNOT sanely require that. So, what you require instead
> is 'sort this array with less than 5 lines of code without allocating
> more than a constant amount of memory'.

i am not saying that you are wrong here. but i am not able to understand 
why bubble sort is not a sane requirement.


> And you can add that 'the intent is to use a bubble sort'. 
> (Maybe its 6, but you can't implement a quicksort in such a small loc :)
> 
> So anywhere you really want to specify a particular algorithm,
> it can almost always be done by specifying limits on the space
> and time complexity order plus perhaps a constraint on the LOC of code.
> There is a way to get what we want *without* gratuitous and arbitrary
> limititations.

i agree. but i still think that this way is very difficult. it has 
proved to be too difficult to me. therefore i can not support that we 
require it. sorry.


bengt