Bug#851749: autopkgtest: machine-readable sub-tests within an autopkgtest-level test

[Martin Pitt]

Hello,

Simon McVittie [2017-01-18 12:56 +0000]:
> autopkgtest currently has one level of hierarchy: a test is either an
> executable script in debian/tests/ named in debian/tests/control, or a
> command in debian/tests/control.
> There is often a finer-grained result than that available.

Right, these are the granularity of "testbed refreshes". autopkgtest's purpose
is to create ephemeral testbeds, run an opaque test in it, and copy the results
back and forth, i. e. it is meant to *run* arbitrary (commonly already
existing) test suites.

I do *not* want autopkgtset to get into the business of becoming a framework
for writing test suites in and prescribe how they must look like -- there's a
ton of these available already, and every programming language has their own
standard (like Perl's Test::More, Python's unittest).  More related to your
thought, there are also several existing reporting frameworks, such as subunit,
TAP, or junit.

> I would like to propose TAP as autopkgtest's protocol for finer-grained
> test result reporting, something like this:

TBH, I don't want autopkgtest to make that call. This already came up a few
years ago, but with proposing subunit. Giving any kind of policy or
recommendation towards one would make it much less useful for e. g. running
test suites which already report via a different framework, or writing simple
smoke-tests in 10 lines of shell (which is very common, and arguably it's not
really necessary in many cases to run complex and large unittests in
autopkgtest).

So my thought on this is: If a test wants to provide machine-readable
fine-grained results/artifacts etc., it's welcome to do that and put it into
$AUTOPKGTEST_ARTIFACTS/ where it's easy to grab for a developer. Once a
particular one becomes widely used in Debian, CI systems like debci could even
possibly search for known ones (subunit, TAP, etc.) and present the results.
But this is a disussion to be had between developers (test authors) and e. g.
debci.

Thanks,

Martin