using autopkgtest for functional upgrade testing

[Antonio Terceiro]

On Sat, Jun 28, 2014 at 04:38:01PM +0200, Martin Pitt wrote:
> Hey Antonio,
> 
> Antonio Terceiro [2014-06-27 21:17 -0300]:
> > $ adt-run \
> >   --user debci \
> >   --tests-dir debian/upgrade-tests/wheezy \
> >   --upgrade-to unstable \
> >   --built-tree . \
> >   --- schroot debci-wheezy+backports-amd64
> 
> I'm still not clear why you need a separate set of tests after
> upgrading and thust the --tests-dir concept, instead of merely running
> the package's already existing autopkgtest in debian/tests/control
> after the upgrade? Sure, a few tests might need to be adjusted to
> check the release they are running on, but that seems easier than
> having to maintain another parallel set of tests?

Tes, in the general case just running the regular set of tests after an
upgrade should probably be enough.

In the specific case I was experimenting with, we had a few upgrade issues with ruby, and I wanted
the corresponding fixes (which are specific to the wheezy→jessie
upgrade) to have automated tests.

Previously I had scripted the tests and run them by hand, but it made
sense to me to use the autopkgtest infrastructure to provide a general
solution to this kind of testing.

I can maintain those tests together with the other ones in debian/tests,
I would just need a mechanism to limit that set of tests to only run
when testing wheezy→jessie upgrades, and not always. That mechanism can
be the ability to override the control file, or it can be something
else.

> > I would like input on a few points:
> 
> Let's discuss them after settling the question above; TBH I'm not
> currently convinced that we even need all this; As far as I can see,
> adding --upgrade-to like functionality (your second patch) and running
> the existing tests ought to suffice?

That and being able to override the control file (or any other way of
restricting a set of tests to a specific scenario) should suffice.

-- 
Antonio Terceiro <terceiro at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140628/c251e334/attachment-0001.sig>