Testing the compiler and code coverage

Sat Mar 6 08:45:45 UTC 2010

The test have a problem not often understood : it must be simpler than
the code it self. If the test is as big as the code size, you will
take time to debug also the test part.

Specification have the drawback to be unprecise. Then you define a way
to define precise specification (it's a harder work) and you find that
you can compile this description, now you have problem with the upper
level spec. It will always be like that.

Lisaac have the most advance feature : contract. We must use it. This
is the check part of a test (low level test). Then you need "only"
higher level test.

Coverage is nice, but there is 3 kinds of it. What you describe is
statement coverage. In DO178b software design, they use decision
coverage (DC) and modified condition/decision coverage (MC/DC). DC
need to have every if-clause, to be true and false. In statement
coverage, an if-clause without "else" part, is covered with "true"
only.

MC/DC is even worse. It asks that in the condition of the IF, we must
find 2 vectors for each boolean variable to make the condition true
and false. This reflect the effect of each boolean.

Coverage is used to find dead code or unexpected result. But it deal
only with input vector of the test. So, if there is no checker (or
contract), it did not check anything.

For my point of view, we must use contract when we write code.
Corverage could only help to find hole.

But that's not enough without big work.

Coverity is a static analyse tools for C. They deliver a big paper
about there experience. There message was : Take the most stupid
mistake, given enough code, you will find it. This encourage to use
even more contract in our code. Coverity is a kind of bunch of generic
contract that is tested staticaly.

What is even more interresting it is that, the stupid error are common
and complexe error are hard to understand and explain; people often
translate them to false positive error !

Regards,
Nicolas

2010/3/6 Mildred Ki'Lya <ml.mildred593 at gmail.com>:
> Hi,
>
> (writing the same in French, you'll get it in a few minutes)
>
> Where I work now, we focus very much on tests and having the tests covering
> all the source code, only it is in Ada, not Lisaac (though they are
> interested in Lisaac).
>
>
>
> I discovered thanks to them Cucumber, it is a tool to make tests easier.
> http://cukes.info/
>
> My idea would be to create tests that would be a specification of the Lisaac
> language, the behaviour reference if you want.
>
> The specificity of Cucumber is that the tests are written in English and are
> called features. That makes them easy to understand for everyone. That is
> also a great way to write specifications that two people can agree with
> (even over a contract).
> Example of features: (files with .feature suffix)
> http://git.debian.org/?p=lisaac/compiler.git;a=tree;f=features;hb=HEAD
>
> Explanation of features, how to read them
>
> (look at how a feature file is structured before continuing)
>
> The first part of a feature (after "Feature:") is descriptive only
> Then, the scenarios are run one after another. The background is run before
> each feature to set up the environment.
> In scenarios (and in the background), actions to be done are prefixed by one
> of the keywords "Given", "When", "Then". The special keyword "And" is a way
> to repeat the keyword of the previous line.
> "Given" lines set up the environment
> "When" lines run actions
> "Then" lines check the result of those actions
>
> Each of the lines match some ruby code to be found in the
> features/step_definitions directory. A regular expression is matched against
> the lines and matches are extracted. Then ruby code is executed.
>
> Run the features
>
> To install cucumber, you would need ruby gems (I heard that on
> Debian/Ububntu, the ruby gems weren't working properly, you might have to
> install ruby-gems manually as my coworker had to). Then you can simply
> install cucumber using gem:
> gem install cucumber
> gem install rake
>
> And you can start playing with cucumber, for example, in the compiler
> directory, just run:
>
> cucumber features/*.feature
>
> And all the features in the feature directory will start (and you get a
> report).
>
> Code coverage
>
> Now, tests are useful, but they are even more so when the cover the complete
> source code and do not left some part of the software untested. That's why
> there are plenty of tools for plenty of languages to check the code
> coverage.
>
> Unfortunately there is none for Lisaac.
>
> Thinking about it, I don't think that creating a code coverage tool for
> Lisaac would be that hard. First, let's look at how code coverage tool
> works:
>
> You compile the code to add instrumentation
> You run the instrumented executable
> The executable creates files containing the code coverage information
>
> Which lines were executed
> Which lines weren't executed
> Which lines cannot be executed (comment, blank lines ...)
>
> You run a tool that would convert the files generated by the executable into
> useful information, HTML reports...
>
> So, that got me thinking, how can we generate coverage information. Lisaac
> makes it easy because it strictly forbid branching within a {BLOCK} or in a
> (list). So we only have to record once in a block that the execution flow
> got here and we know the whole block or list was executed.
>
> Now, trickier, how to differentiate between blocks and lists that are not
> executed, and parts of the code that are not source code. This involves the
> parser. Each time it finds a block or a list, it has to record it somewhere
> so that we can match blocks lists executed with the complete set of blocks
> and lists.
>
>
>
> The way I see it is like that: When the parser find a block (or list) it
> record it in a list with its beginning position and end position (to match
> with the source code afterwards). It also given them an id. And the compiler
> will end up storing all that in a file for future use.
>
> The parser would also add an instruction at the beginning of a block (or
> list). That instruction would be a call to a function (either using a C
> external, quick and easy, or using Lisaac code, more flexible). The
> parameters to this function is an id of the block (or list).
>
> This function will, when the executable runs, gather the list of all
> executed blocks (or lists) retaining their id, and when the program exit (or
> on the fly) write them on a file.
>
> Then, we run a tool to match the two files, and we are done.
>
> Conclusion
>
> That way, we can have a very robust compiler, and make sure that tests are
> effective.
>
> It can also be very useful to remove dead code that is no longer used, I
> think Benoit will like this side of things.
>
>
> Mildred
>
> --
> Mildred Ki'Lya
> ╭───────── ｍｉｌｄｒｅｄ５９３＠ｏｎｌｉｎｅ．ｆｒ ──────────
> │ Jabber, GoogleTalk: <mildred at jabber.fr>
> │ Website: <http://ki.lya.online.fr>           GPG ID: 9A7D 2E2B
> │ Fingerprint: 197C A7E6 645B 4299 6D37 684B 6F9D A8D6 9A7D 2E2B
>
> _______________________________________________
> Lisaac-devel mailing list
> Lisaac-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/lisaac-devel
>
>