using autopkgtest for functional upgrade testing

[Antonio Terceiro]

Hello,

I have been experimenting with using autopkgtest to run functional tests on
upgraded systems. This type of automated tests consist of using a testbed on a
previous release (currently wheezy), then upgrading the system to the next
release (currently jessie) after installing the test dependencies, but before
running the tests.

I have been testing this initially with the ruby-defaults source package, using
the following code in the source package + a patched autopkgtest (patches
follow this message):

http://anonscm.debian.org/gitweb/?p=collab-maint/ruby-defaults.git;a=commitdiff;h=61e4d012a851fef740ba9d4af6a40eeee785b1ea

The adt-run invocation looks like this:

$ adt-run \
  --user debci \
  --tests-dir debian/upgrade-tests/wheezy \
  --upgrade-to unstable \
  --built-tree . \
  --- schroot debci-wheezy+backports-amd64

I intend to eventually add support for such upgrade tests to debci.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I would like input on a few points:

0) how to declare the existance of such tests in debian/control.

According to the autopkgtest README.package-tests, the `Testsuite` field is
supposed to be a comma-separated of values, which is fine. We just need to
agree on what value to use there.

I thought about something like

  Testsuite: autopkgtest/upgrade/$codename

In a package that also contains a regular DEP-8 test suite, it could look like

  Testsuite: autopkgtest, autopkgtest/upgrade/$codename

1) a new DEP-N, based on DEP-8

I would like to propose a DEP-N spec for this at some point that would refer to
DEP-8 as a basis, but I would like to drive DEP-8 to an acceptable state before
that. Currently DEP-8 reads as just "go read README.packages-tests at the
autopkgtest source tree".

So: would be there any concerns about taking the contents of
README.packages-tests as a draft test for the DEP-9 spec?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I have also written a patch for sadt(1) from devscripts to implent the
--tests-dir, which is simple enough and can be pushed as soon as we agree to
add --tests-dir to autopkgtest:

http://anonscm.debian.org/gitweb/?p=users/terceiro/devscripts.git;a=commitdiff;h=4876cb72afbda516843e84f9ae01d97310c65bf8

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A summary of the patches that follow this message:

Antonio Terceiro (3):
  adt-run: add --tests-dir option for selecting tests directory
  adt-run: add --upgrade-to option for upgrade tests
  adt-run: set APT_LISTBUGS_FRONTEND=none when installing packages

 debian/changelog |  9 +++++++++
 lib/testdesc.py  |  7 +++----
 runner/adt-run   | 30 +++++++++++++++++++++++++++---
 runner/adt-run.1 | 11 +++++++++++
 4 files changed, 50 insertions(+), 7 deletions(-)

-- 
2.0.0