RFC: test helper for executables

Christian Kastner debian at kvr.at
Wed Mar 18 20:42:10 UTC 2015

On 2015-03-16 15:38, Antonio Terceiro wrote:
> On Sun, Mar 15, 2015 at 04:06:33PM +0100, Christian Kastner wrote:
>> The tests I want to add are just of a very basic nature; I merely want
>> to increase the _breadth_ of testing. Increasing the _depth_ will be up
>> to the individual maintainers as they have the experience with the
>> package necessary to do this.
> If you end up looking for a place to consolidate shared code that will
> be used in multiple packages, we can think about adding it to autodep8.

I've started to work on some packages, by descending popcon, and already
encountered a few without any sort of tests.

For packages providing executables, I chose the common pattern of
creating a test runner [1] that will compare the output of a test with
an expected output, and fail if they do not match:

 ├ [control]
 │ > Tests: mytests
 ├ [mytests]
 │ > adt-helper-executables mytests.d    [2], [3]
 └ mytests.d/
    ├ [testname.1.in]
    │ > #!/bin/sh
    │ > echo "Foo!"
    ├ [testname.1.expected]
    │ > Foo!
    ├ [testname.2.in]
    | > #!/bin/sh
    │ > factor 1334
    ├ [testname.2.expected]
    │ > 1334: 2 23 29

This makes writing tests for executables quite trivial. The boilerplate
of running an autopkgtest test is consolidated in the helper, and actual test
creation is reduced to a few lines of code (3 in the best case).

I plan to test the efficacy of this approach against src:shadow, which
produces 36 executables -- some of them essential -- yet has no testing
whatsoever. I'll report back with my findings (that will take a while as
I will be busy with something else over the next few days).

You can find a working prototype, with more documentation, in this
version of debianutils I created [5].

This is purely exploratory at this point; I won't submit any tests to the
maintainers until I'm more confident that it's going in the right direction
(or can easily be adapter later on). Feedback would therefore by highly


[1] Don't mind the name, it is purely arbitrary at this point.

[2] I'm assuming for now that the helper is somewhere in PATH.

[3] As you can see, the "mytests" autopkgtest is just a trampoline.

[4] An interesting possibility would be to allow regexs in the .expected
files, but at believe it would premature to consider this at this point.

[5] http://www.kvr.at/debian/pool/main/d/debianutils/debianutils_4.4+nmu1.dsc

More information about the autopkgtest-devel mailing list