Testing the compiler and code coverage

[Mildred Ki'Lya]

Hi,

(writing the same in French, you'll get it in a few minutes)

Where I work now, we focus very much on tests and having the tests
covering all the source code, only it is in Ada, not Lisaac (though they
are interested in Lisaac).



I discovered thanks to them Cucumber, it is a tool to make tests easier.
http://cukes.info/

My idea would be to create tests that would be a specification of the
Lisaac language, the behaviour reference if you want.

The specificity of Cucumber is that the tests are written in English and
are called features. That makes them easy to understand for everyone.
That is also a great way to write specifications that two people can
agree with (even over a contract).
Example of features: (files with .feature suffix)
http://git.debian.org/?p=lisaac/compiler.git;a=tree;f=features;hb=HEAD

*Explanation of features, how to read them*

(look at how a feature file is structured before continuing)

    * The first part of a feature (after "Feature:") is descriptive only
    * Then, the scenarios are run one after another. The background is
      run before each feature to set up the environment.
    * In scenarios (and in the background), actions to be done are
      prefixed by one of the keywords "Given", "When", "Then". The
      special keyword "And" is a way to repeat the keyword of the
      previous line.
    * "Given" lines set up the environment
    * "When" lines run actions
    * "Then" lines check the result of those actions


Each of the lines match some ruby code to be found in the
features/step_definitions directory. A regular expression is matched
against the lines and matches are extracted. Then ruby code is executed.

*Run the features*

To install cucumber, you would need ruby gems (I heard that on
Debian/Ububntu, the ruby gems weren't working properly, you might have
to install ruby-gems manually as my coworker had to). Then you can
simply install cucumber using gem:
gem install cucumber
gem install rake

And you can start playing with cucumber, for example, in the compiler
directory, just run:

cucumber features/*.feature

And all the features in the feature directory will start (and you get a
report).

*Code coverage*

Now, tests are useful, but they are even more so when the cover the
complete source code and do not left some part of the software untested.
That's why there are plenty of tools for plenty of languages to check
the code coverage.

Unfortunately there is none for Lisaac.

Thinking about it, I don't think that creating a code coverage tool for
Lisaac would be that hard. First, let's look at how code coverage tool
works:

   1. You compile the code to add instrumentation
   2. You run the instrumented executable
   3. The executable creates files containing the code coverage information
          * Which lines were executed
          * Which lines weren't executed
          * Which lines cannot be executed (comment, blank lines ...)
   4. You run a tool that would convert the files generated by the
      executable into useful information, HTML reports...

So, that got me thinking, how can we generate coverage information.
Lisaac makes it easy because it strictly forbid branching within a
{BLOCK} or in a (list). So we only have to record once in a block that
the execution flow got here and we know the whole block or list was
executed.

Now, trickier, how to differentiate between blocks and lists that are
not executed, and parts of the code that are not source code. This
involves the parser. Each time it finds a block or a list, it has to
record it somewhere so that we can match blocks lists executed with the
complete set of blocks and lists.



The way I see it is like that: When the parser find a block (or list) it
record it in a list with its beginning position and end position (to
match with the source code afterwards). It also given them an id. And
the compiler will end up storing all that in a file for future use.

The parser would also add an instruction at the beginning of a block (or
list). That instruction would be a call to a function (either using a C
external, quick and easy, or using Lisaac code, more flexible). The
parameters to this function is an id of the block (or list).

This function will, when the executable runs, gather the list of all
executed blocks (or lists) retaining their id, and when the program exit
(or on the fly) write them on a file.

Then, we run a tool to match the two files, and we are done.

*Conclusion*

That way, we can have a very robust compiler, and make sure that tests
are effective.

It can also be very useful to remove dead code that is no longer used, I
think Benoit will like this side of things.


Mildred

-- 
Mildred Ki'Lya
╭───────── ｍｉｌｄｒｅｄ５９３＠ｏｎｌｉｎｅ．ｆｒ ──────────
│ Jabber, GoogleTalk: <mildred at jabber.fr>
│ Website: <http://ki.lya.online.fr>           GPG ID: 9A7D 2E2B
│ Fingerprint: 197C A7E6 645B 4299 6D37 684B 6F9D A8D6 9A7D 2E2B

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/lisaac-devel/attachments/20100306/1253009b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/lisaac-devel/attachments/20100306/1253009b/attachment-0001.pgp>