Testing the compiler and code coverage

[Nicolas Boulay]

2010/3/6 Mildred Ki'Lya <ml.mildred593 at gmail.com>:
> On 03/06/2010 09:45 AM, Nicolas Boulay wrote:
>> The test have a problem not often understood : it must be simpler than
>> the code it self. If the test is as big as the code size, you will
>> take time to debug also the test part.
>>
>> Specification have the drawback to be unprecise. Then you define a way
>> to define precise specification (it's a harder work) and you find that
>> you can compile this description, now you have problem with the upper
>> level spec. It will always be like that.
>>
>
> Sure, tests are not that perfect, they don't test really everything,
> bugs can still be there when we have tests. I understand that. But I
> think that no method is perfect, a combinaison must be used.
>> Lisaac have the most advance feature : contract. We must use it. This
>> is the check part of a test (low level test). Then you need "only"
>> higher level test.
>>
>
> Contracts are at the level of the code, not really the specification level.
> We should use them, but the problem is that to write contracts, we must
> know the code we write contracts for.
>
> And for the moment, the compiler source is except for Benoît still very
> obscure. Well, it is for me anyway. I can't write contracts on the
> compiler. But I can write higher level tests.
>
> Moreover, tests in English can become a specification, and the scenarios
> in tests can becomme acceptance tests. It sets in stone the behaviour of
> the compiler, not having to rely on the even more unprecuse reference
> manual.
>
> Don't be mistaken, the reference manual is great to learn the language,
> but it doesn't makes a formal specification. (Firtunately, or beginners
> would run away)
>

So you could just write bugs for trickiest part, only. You could
consider that the rest will be tested by "normal" code.

>> Coverage is nice, but there is 3 kinds of it. What you describe is
>> statement coverage. In DO178b software design, they use decision
>> coverage (DC) and modified condition/decision coverage (MC/DC). DC
>> need to have every if-clause, to be true and false. In statement
>> coverage, an if-clause without "else" part, is covered with "true"
>> only.
>>
> I became awayre of the DO178B, especially since I work in a tool similar
> to cucumber but for the Ada language, and AdaCore (the creators of the
> GNAT compiler, Ada for GCC). And they work with companies that are bound
> to the DO178B.

You work on "Couverture" ? Is it fonctionnal ?

I'm looking for a tool coverage at object code level.

>
> By the way, if you are in Paris the 11th of March, there is the Open-DO
> Conference. Unfortunately I won't be there (I go to my IUT instead)
> http://www.open-do.org/conference-2010/
> Open-Do is a project started by AdaCore to gather open-source tools for
> the DO178B.
>> MC/DC is even worse. It asks that in the condition of the IF, we must
>> find 2 vectors for each boolean variable to make the condition true
>> and false. This reflect the effect of each boolean.
>>
>
> Worse or better ?


Worse to use, better to find bugs.

>
> Unfortunately, I gave some thought, and I don't think this is applicable
> to Lisaac for there is absolutely no test constructions in Lisaac. If
> are only a special case of polymorphic call. And the constraint that all
> polymorphic calls must call all the types of their hierarchy is far too
> much to be useful.

MC/DC is still not completly defined. The idea is to cover all the
decisions that make the control path change.

>> Coverage is used to find dead code or unexpected result. But it deal
>> only with input vector of the test. So, if there is no checker (or
>> contract), it did not check anything.
>>
>
> Of course, tests must check the result is correct
> But coverage is very good to notice special cases that were not included
> in the test suite. I do that every day and I find it very useful.
>

Sur it is.

My point is only to say that it is very complexe to finish the job
that way. In a DO178b project, only 10% of the time is dedicated to
code. Lisaac it's almost 95% :)

>
> Mildred
>
> --
> Mildred Ki'Lya
> ╭───────── ｍｉｌｄｒｅｄ５９３＠ｏｎｌｉｎｅ．ｆｒ ──────────
> │ Jabber, GoogleTalk: <mildred at jabber.fr>
> │ Website: <http://ki.lya.online.fr>           GPG ID: 9A7D 2E2B
> │ Fingerprint: 197C A7E6 645B 4299 6D37 684B 6F9D A8D6 9A7D 2E2B
>
>
>
> _______________________________________________
> Lisaac-devel mailing list
> Lisaac-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/lisaac-devel
>
>