RFC: test helper for executables

[Antonio Terceiro]

On Wed, Mar 18, 2015 at 09:42:10PM +0100, Christian Kastner wrote:
> On 2015-03-16 15:38, Antonio Terceiro wrote:
> > On Sun, Mar 15, 2015 at 04:06:33PM +0100, Christian Kastner wrote:
> >> The tests I want to add are just of a very basic nature; I merely want
> >> to increase the _breadth_ of testing. Increasing the _depth_ will be up
> >> to the individual maintainers as they have the experience with the
> >> package necessary to do this.
> > 
> > If you end up looking for a place to consolidate shared code that will
> > be used in multiple packages, we can think about adding it to autodep8.
> 
> I've started to work on some packages, by descending popcon, and already
> encountered a few without any sort of tests.
> 
> For packages providing executables, I chose the common pattern of
> creating a test runner [1] that will compare the output of a test with
> an expected output, and fail if they do not match:
> 
> debian/tests
>  ├ [control]
>  │ > Tests: mytests
>  │
>  ├ [mytests]
>  │ > adt-helper-executables mytests.d    [2], [3]

you can use

  Test-Command: adt-helper-executables mytests.d

directly in debian/tests/control and skip the extra script.

>  │
>  └ mytests.d/
>     ├ [testname.1.in]
>     │ > #!/bin/sh
>     │ > echo "Foo!"
>     │
>     ├ [testname.1.expected]
>     │ > Foo!
>     │
>     ├ [testname.2.in]
>     | > #!/bin/sh
>     │ > factor 1334
>     │
>     ├ [testname.2.expected]
>     │ > 1334: 2 23 29
>     │ 
>         ....
> 
> This makes writing tests for executables quite trivial. The boilerplate
> of running an autopkgtest test is consolidated in the helper, and actual test
> creation is reduced to a few lines of code (3 in the best case).
> 
> I plan to test the efficacy of this approach against src:shadow, which
> produces 36 executables -- some of them essential -- yet has no testing
> whatsoever. I'll report back with my findings (that will take a while as
> I will be busy with something else over the next few days).
> 
> You can find a working prototype, with more documentation, in this
> version of debianutils I created [5].
> 
> This is purely exploratory at this point; I won't submit any tests to the
> maintainers until I'm more confident that it's going in the right direction
> (or can easily be adapter later on). Feedback would therefore by highly
> welcome!
> 
> Regards,
> Christian
> 
> [1] Don't mind the name, it is purely arbitrary at this point.
> 
> [2] I'm assuming for now that the helper is somewhere in PATH.
> 
> [3] As you can see, the "mytests" autopkgtest is just a trampoline.
> 
> [4] An interesting possibility would be to allow regexs in the .expected
> files, but at believe it would premature to consider this at this point.
> 
> [5] http://www.kvr.at/debian/pool/main/d/debianutils/debianutils_4.4+nmu1.dsc

This looks nice so far.

-- 
Antonio Terceiro <terceiro at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20150319/cefcbece/attachment.sig>