Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests

[Dominic Hargreaves]

Package: autopkgtest
Version: 3.14.3
Severity: wishlist

README.package-tests.rst.gz says:

  To allow test execution environments to discover packages which provide
  tests, their source packages should have a ``Testsuite:`` header
  containing ``autopkgtest`` (which is currently the only defined value).
  Multiple values get comma separated, as usual in control files.

My expectation having read this was that adt-run would only attempt
to test packages if the Testsuite header is present; however adt-run
actually appears to not look for this header at all; it always processes
tests for a package, including automagic test suites set up by
autodep8. This is probably fine in the case of invoking adt-run with
a single package, but not if you want to look for regressions using
adt-run; for example I ran it against all packages I'd been rebuilding
against perl 5.22, and found quite a few failures which didn't appear to
be related to perl 5.22 but were instead tests which just didn't work.

Please could adt-run have an option to exit without running any tests
if the source package in question doesn't define any? I think this
would be useful in other QA contexts too. I would say that this should
be the default, but that is probably too difficult to change now.

Cheers,
Dominic.