Bug#741330: autopkgtest: please add ability to wrap a script/runner/adverb around existing tests

[Simon McVittie]

Package: autopkgtest
Version: 2.9.1
Severity: wishlist

autopkgtest is not the only implementation of "as-installed" package testing.
Other options include:

* GNOME InstalledTests (a test is a .desktop-style file in a subdirectory
  of /usr/share/installed-tests,
  <https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests>)
* Autotools "make installcheck"
* NetBSD kyua <http://code.google.com/p/kyua/wiki/About>
* Perl-style TAP tests
* whatever other ad-hoc frameworks people have come up with,
  e.g. /usr/lib/telepathy-gabble-tests/twisted/run-test.sh vaguely mimics
  Autotools output and was designed to be screen-scraped by Linaro's LAVA

It would be nice to be able to wrap tests designed for one of these other
frameworks to make them conform to autopkgtest syntax and requirements.
For instance, some tests are executable as-is, but should be wrapped
in "adverb" commands like env, xvfb-run and/or dbus-run-session; and the .test
files representing GNOME installed-tests are not directly executable at all,
but can be run by wrapping them in gnome-desktop-testing-runner.

autopkgtest currently requires that each test is a separate
executable in debian/tests (or some other single directory in the source
tarball). In practice, this results in the entire upstream test-suite
becoming a single autopkgtest, for instance
<http://anonscm.debian.org/viewvc/pkg-gnome/desktop/experimental/glib2.0/debian/tests/installed-tests?revision=40498&view=markup>,
making it impossible for a generic autopkgtest-based framework to track
failures over time: instead of "tests 'foo' and 'bar' have always passed,
but test 'baz' regressed between versions 1.2 and 1.3" the best available
report would be "the only test is 'installed-tests' and it regressed between
1.2 and 1.3".

Straw-man syntax proposal:

    If the Interpreter for a stanza is specified, the autopkgtest
    implementation will look for it in the Tests-Directory (defaulting to
    debian/tests as usual). If it is found there, it will be made executable
    and executed once for each test-case listed in Tests, passing the
    test-case's name as an argument.

    If the Interpreter does not exist as a file in the Tests-Directory,
    it is taken to be a shell command-line to which the
    test-case's name will be appended; commands which can typically be
    used in this way include env, dbus-run-session and xvfb-run.

    If an Interpreter is specified, the Tests are not required to exist
    as files in debian/tests, and may contain "/" in addition to the
    characters allowed in Debian package names.

    Example: wrapping non-executable tests in commands from $PATH

    Tests:
     glib/1bit-mutex.test
     glib/642026-ec.test
     ...
     glib/vfs.test
     glib/volumemonitor.test
    Interpreter: xvfb-run gnome-desktop-testing-runner
    Depends: gnome-desktop-testing, xvfb, ...

    Example: wrapping tests that need a special environment

    # debian/tests/control:
    Tests: fred bill bongo
    Interpreter: wrap-test.sh
    Depends: dbus, dpkg-dev, xvfb, ...

    # debian/tests/wrap-test.sh:
    #!/bin/sh
    exec dbus-run-session -- xvfb-run -- env TEST_OPTIONS=all,verbose -- \
        debian/tests/"$1"

It would also be nice if the Tests could be the output of a script, or
listed in a file that may be created by the package build, or something,
so that the list of test cases doesn't have to be hard-coded in
debian/tests/control. That way, GNOME packages could use something like

    gnome-desktop-testing-runner --list glib/ | sed -e 's/ (.*)$//' | LC_ALL sort -u

to list all their test-cases automatically.

Regards,
    S

-- System Information:
Debian Release: jessie/sid
  APT prefers proposed-updates
  APT policy: (500, 'proposed-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.13-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages autopkgtest depends on:
ii  apt-utils      0.9.15.5+b1
ii  debhelper      9.20140228
ii  pbuilder       0.215
ii  python         2.7.5-5
ii  python-debian  0.1.21+nmu2

autopkgtest recommends no packages.

Versions of packages autopkgtest suggests:
pn  autopkgtest-xenlvm  <none>
ii  lxc                 0.9.0~alpha3-2+deb8u1
ii  qemu-system         1.7.0+dfsg-3
ii  qemu-utils          1.7.0+dfsg-3
ii  schroot             1.6.8-1

-- no debconf information