Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests

[Martin Pitt]

Hello Dominic,

Dominic Hargreaves [2015-06-14  0:18 +0100]:
>   To allow test execution environments to discover packages which provide
>   tests, their source packages should have a ``Testsuite:`` header
>   containing ``autopkgtest`` (which is currently the only defined value).
>   Multiple values get comma separated, as usual in control files.
> 
> My expectation having read this was that adt-run would only attempt
> to test packages if the Testsuite header is present; however adt-run
> actually appears to not look for this header at all;

That's correct. If you run adt-run on a source tree/dsc etc., it will
only look for debian/tests/control (or autodep8). The above is just a
recommendation ("should") and states the reason for that.

> it always processes tests for a package, including automagic test
> suites set up by autodep8. This is probably fine in the case of
> invoking adt-run with a single package, but not if you want to look
> for regressions using adt-run; for example I ran it against all
> packages I'd been rebuilding against perl 5.22, and found quite a
> few failures which didn't appear to be related to perl 5.22 but were
> instead tests which just didn't work.

Right, we get the same on Ubuntu CI. Over time the failures pile up, I
figure it's quite difficult for the Perl team to look at/fix all
these.

> Please could adt-run have an option to exit without running any tests
> if the source package in question doesn't define any?

So to clarify, you want an option to disable autodep8, right? If a
source package doesn't have tests and autodep8 does not generate any
either, then adt-run will already report "no tests in this package"
and exit with 8. With a perl package and --no-autodep8 it would then
do the same.

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)