Bug#741330: autopkgtest: please add ability to wrap a script/runner/adverb around existing tests

[Martin Pitt]

Hey Simon,

Simon McVittie [2014-03-11 10:35 +0000]:
> It would be nice to be able to wrap tests designed for one of these other
> frameworks to make them conform to autopkgtest syntax and requirements.
> [...]
> autopkgtest currently requires that each test is a separate
> executable in debian/tests (or some other single directory in the source
> tarball). In practice, this results in the entire upstream test-suite
> becoming a single autopkgtest

This has come up several times, although in the context of parsing out
individual test results from the output of a test (often, test suites
from a single test program contain many tests).

> making it impossible for a generic autopkgtest-based framework to track
> failures over time: instead of "tests 'foo' and 'bar' have always passed,
> but test 'baz' regressed between versions 1.2 and 1.3" the best available
> report would be "the only test is 'installed-tests' and it regressed between
> 1.2 and 1.3".

Right, you can't immediately tell which individual test failed.
However, we don't currently have an XFAIL concept in autopkgtest, i.
e. we expect all tests to succeed. Having that would be nice though,
for identifying the flaky ones.

> Straw-man syntax proposal:
> 
>     If the Interpreter for a stanza is specified, the autopkgtest
>     implementation will look for it in the Tests-Directory (defaulting to
>     debian/tests as usual). If it is found there, it will be made executable
>     and executed once for each test-case listed in Tests, passing the
>     test-case's name as an argument.

I like that idea. I'm not 100% happy about the term "Interpreter:",
but we can quibble over that :-)

>     If the Interpreter does not exist as a file in the Tests-Directory,
>     it is taken to be a shell command-line to which the
>     test-case's name will be appended; commands which can typically be
>     used in this way include env, dbus-run-session and xvfb-run.

This is essentially the same as above, right? Look for interpreter in
${Tests-Directory:debian/tests}:$PATH.

> It would also be nice if the Tests could be the output of a script, or
> listed in a file that may be created by the package build, or something,
> so that the list of test cases doesn't have to be hard-coded in
> debian/tests/control. That way, GNOME packages could use something like
> 
>     gnome-desktop-testing-runner --list glib/ | sed -e 's/ (.*)$//' | LC_ALL sort -u
> 
> to list all their test-cases automatically.

I'm rather sceptical about using a command like this to generate them
on the fly. First, you'd need some kind of meta-dependency to be able
to generate debian/tests/control, and then that command might fail and
not actually generate any test, etc. But I'd like to support these
two:

 * Read tests from a file:

   Tests: < debian/tests/testlist

 * Globbing:

   Tests: glib*.test

Do you think that would suffice your needs?

Thanks,

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140404/afdc4276/attachment.sig>