Bug#759753: autopkgtest: support an include directive in test control files

[Martin Pitt]

Hello,

we discussed this a bit in the long pkg-perl@ thread, but for the
records I should also reply here.

Niko Tyni [2014-08-29 16:49 -0700]:
> While there's no avoiding having to do source uploads of all the packages
> for this [1], we'd like to find a way to do it so that future changes
> to these generic tests or adding new tests don't require touching each
> source package.

I think this isn't urgent now as we decided that for Perl etc. we want
no control file at all and rather centralize that with the dynamic
control, "Testsuite: autopkgtest-pkg-perl", and pkg-perl-autopkgtest.

> A few levels of indirection seems to get us pretty close to this [2],
> but having an 'include' directive of some sort in debian/tests/control
> would offer a cleaner and more extensible interface.
> 
> I'm thinking of something like
> 
> Include: <binary-package> /some/file

I'm still trying to find some quiet time to think about this
thoroughly, but my first instinct is that in the currently proposed
form it's not very helpful -- it combines the disadvantage of having
to do boilerplate source changes with the disadvantage of still not
having a self-contained test description (i. e. a fully working
explicit control file).  A file which is essentially just "#include
perl-test.control" doesn't say much beyond the already obvious fact
that this is a Perl package. I think for this purpose the "Testsuite:
autopkgtest-pkg-perl" declaration says essentially the same and is
more flexible.

If you want to run some generic tests like pkg-perl-autopkgtest, and
modify some of their properties (like adding a restriction), I think
it's best to just include the full control file. For an individual
package it won't change all that often, and in those cases it's IMHO
more robust and easier to verify/understand to have the full explicit
control file.

Another use case is if you want to run some generic tests like
pkg-perl-autopkgtest, and additionally some package specific ones. I
would prefer if for that the package specific tests would go into
debian/tests/control as normal, and the "autopkgtest-pkg-perl"
triggers the standard ones. There is no explicit #include directive
necessary, the two concepts would be disjoint from each other. That of
course requires us to actually do add autopkgtest-pkg-perl everywhere
first. Currently, if there is a d/t/control that and only that will
be run, but I'm happy to teach autopkgtest about checking the
Testsuite: header for that, so that the "automatic perl test without
explicit Testsuite: header" semantics will be transitional only.

There might be other use cases for #include files that I haven't
considered. Do you have some? I wouldn't like to change the
specification and implement something potentially disruptive and
brittle without having some actual demand for it.

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140919/c5259651/attachment.sig>