RFC: test helper for executables

Christian Kastner debian at kvr.at
Wed Mar 18 20:42:10 UTC 2015


On 2015-03-16 15:38, Antonio Terceiro wrote:
> On Sun, Mar 15, 2015 at 04:06:33PM +0100, Christian Kastner wrote:
>> The tests I want to add are just of a very basic nature; I merely want
>> to increase the _breadth_ of testing. Increasing the _depth_ will be up
>> to the individual maintainers as they have the experience with the
>> package necessary to do this.
> 
> If you end up looking for a place to consolidate shared code that will
> be used in multiple packages, we can think about adding it to autodep8.

I've started to work on some packages, by descending popcon, and already
encountered a few without any sort of tests.

For packages providing executables, I chose the common pattern of
creating a test runner [1] that will compare the output of a test with
an expected output, and fail if they do not match:

debian/tests
 ├ [control]
 │ > Tests: mytests
 │
 ├ [mytests]
 │ > adt-helper-executables mytests.d    [2], [3]
 │
 └ mytests.d/
    ├ [testname.1.in]
    │ > #!/bin/sh
    │ > echo "Foo!"
    │
    ├ [testname.1.expected]
    │ > Foo!
    │
    ├ [testname.2.in]
    | > #!/bin/sh
    │ > factor 1334
    │
    ├ [testname.2.expected]
    │ > 1334: 2 23 29
    │ 
        ....

This makes writing tests for executables quite trivial. The boilerplate
of running an autopkgtest test is consolidated in the helper, and actual test
creation is reduced to a few lines of code (3 in the best case).

I plan to test the efficacy of this approach against src:shadow, which
produces 36 executables -- some of them essential -- yet has no testing
whatsoever. I'll report back with my findings (that will take a while as
I will be busy with something else over the next few days).

You can find a working prototype, with more documentation, in this
version of debianutils I created [5].

This is purely exploratory at this point; I won't submit any tests to the
maintainers until I'm more confident that it's going in the right direction
(or can easily be adapter later on). Feedback would therefore by highly
welcome!

Regards,
Christian

[1] Don't mind the name, it is purely arbitrary at this point.

[2] I'm assuming for now that the helper is somewhere in PATH.

[3] As you can see, the "mytests" autopkgtest is just a trampoline.

[4] An interesting possibility would be to allow regexs in the .expected
files, but at believe it would premature to consider this at this point.

[5] http://www.kvr.at/debian/pool/main/d/debianutils/debianutils_4.4+nmu1.dsc



More information about the autopkgtest-devel mailing list