Bug#741330: autopkgtest: please add ability to wrap a script/runner/adverb around existing tests

[Simon McVittie]

On 04/04/14 16:44, Martin Pitt wrote:
> Right, you can't immediately tell which individual test failed. 
> However, we don't currently have an XFAIL concept in autopkgtest,
> i. e. we expect all tests to succeed.

Sure, you hope all tests succeed, but when they (inevitably) don't, it
would be nice to be able to tell the difference between "that one
that's always been flaky failed again" and "everything's broken". The
severity of a failure can vary: to put it in Debian bug terms, some
tests are release-critical, some tests are important, and some tests
are merely wishlist.

If I look at Telepathy-Gabble, for instance, failure to connect to the
test server and send messages could reasonably be said to be "grave",
but failure to do file transfer is of "normal" severity, or possibly
"important" if you're really pushing it.

Test failures not automatically being release-critical (like they
usually are in build-time testing) is one of the things I like about
autopkgtest - in practice, tests are often less reliable than the
actual software. I intend to be more trigger-happy about enabling
autopkgtest in Telepathy than I've been about enabling
tests-during-the-build, because I know not all our tests are reliable
enough to be considered release-critical, but they're still valuable
information.

> I like that idea. I'm not 100% happy about the term
> "Interpreter:", but we can quibble over that :-)

I'm not delighted about that name either. Better suggestions welcome!

>> If the Interpreter does not exist as a file in the
>> Tests-Directory, it is taken to be a shell command-line to which
>> the test-case's name will be appended; commands which can
>> typically be used in this way include env, dbus-run-session and
>> xvfb-run.
> 
> This is essentially the same as above, right? Look for interpreter
> in ${Tests-Directory:debian/tests}:$PATH.

Not quite: it can be a shell command line (with multiple words). The
idea was that you can do

    Interpreter: dbus-run-session -- env XDG_RUNTIME_DIR=$ADTTMP --

to wrap the test in both dbus-run-session and env. Perhaps this is
over-engineering and I should be using debian/tests/environment.sh
(containing that command-line + "$@") as my Interpreter, but I think
this would be enough to avoid needing a separate script in many cases.

> I'm rather sceptical about using a command like this to generate
> them on the fly. First, you'd need some kind of meta-dependency to
> be able to generate debian/tests/control, and then that command
> might fail and not actually generate any test, etc.

A reasonable objection... but I'd rather have that than having to
generate debian/tests/control in `debian/rules clean` and ship the
generated file in the debian.tar.gz, which is the next best thing I
can think of :-)

I don't think a meta-dependency is needed: you could mandate that if
you have

    List-tests: list-tests.sh
    Depends: foo, bar, baz

then list-tests.sh must work with foo, bar, baz installed. (Again, I'm
thinking of the GLib tests, which all have the same dependencies.)

Similarly, for error handling, you could say that it's considered to
be an overall test suite failure if list-tests.sh exits nonzero,
prints to stderr, or fails to print any test names to stdout.

> * Read tests from a file:
> 
> Tests: < debian/tests/testlist

If it can be a file in the filesystem, I think that would be fine -
packagers of GNOME stuff could arrange for it to be generated during
the build and installed alongside the *.test files. If not, the only
mechanism I can see for arranging for it to be generated would be to
repeat the entire package build, which seems a shame when the GNOME
installed-tests are specifically designed to not need that.

> * Globbing:
> 
> Tests: glib*.test

In GNOME installed-tests, the actual files of interest are
/usr/share/installed-tests/$PACKAGE/*.test and
/usr/share/installed-tests/$PACKAGE/**/*.test (using the rsync/zsh
notation where ** matches any number of levels of directories), so
there would need to be a way to say "cut off this prefix when naming
the tests".

In the packaging I'm currently doing for Telepathy-1, we do actively
use subdirectories in the installed-tests (e.g. see
<http://people.collabora.com/~smcv/telepathy-1/pool/main/t/telepathy-mission-control-6/>,
which clashes with autopkgtest's restriction to "test names would be
syntactically valid as Debian package names". I could switch to a flat
hierarchy, but not in a particularly straightforward way unless I made
upstream changes for autopkgtest's benefit.

    S