[Shootout-list] Directions of various benchmarks

[Bengt Kleberg]

skaller wrote:
> On Wed, 2005-05-18 at 16:25, Jos=C3=A9 Bollo wrote:
>=20
>>Le mercredi 18 Mai 2005 08:06, Bengt Kleberg a =C3=A9crit :

...deleted
>>>imho it is a good idea to care about loc.
>>>the amount of bugs is proportional to the amount of code.
>>>less code =3D> less bugs.
>>
>>yes at a constant programming language.
>>
>>but they are not comparable.
>=20
>=20
> Actually they ARE comparable, research has actually been done on this.
> It shows (from memory of a second hand account :) that the maintenance
> cost and bug rates for ALL code depended on the number of lines of code=
,
> independently of language. Which means that more compact languages
> have less bugs 'per unit of functionality'.

this is mentioned (as beeing the result of studies done in the us) in=20
the book Safeware, by  Nancy G. Leveson.


...deleted
> Come on! I did some tests and find Felix comfortable trashes
> gcc on some tests, but gcc with a couple of optimisation switches
> trashes Felix .. what are we *really* measuring?

this is (imho) the most basic question in the shootout. i think it=20
should be addressed (as one of the) first.

we need to know how to make the best possible (given the set of=20
constraints we have) tests. to add new tests without having clearified=20
this, seems to me to be a waste of time. to change/bugfix the current=20
system without having clearified this, seems to me to be a waste of time.

the result of this should go into the faq, to give us the possiblity to=20
improve what the shootout does without having to start from scratch. it=20
will be very helpful for newcomers, too.

i find it strange that i am having such difficulties to get hold of the=20
the set of constraints we have. perhaps this will change if more people=20
become interested in this question?


> These are rhetorical questions .. the point is that I don't see why
> measuring complexity of the code is any harder to get totally wrong
> than measuring speed 1/2 ;(

i think that the problem with code complexity is not one of ''harder to=20
get totally wrong''. i think code complexity is more labour intensive=20
''to get totally wrong''.
the shootout is understaffed and need to concentrate the available=20
resources. therefore i think it is wise to deal with performance first.


bengt