Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

[Simon McVittie]

On Mon, 16 Jan 2017 at 12:34:50 +0100, Martin Pitt wrote:
> Maybe "expected-failure" which is the term used by other test runners? (The
> bikeshedding season is on! ☺ )

The bikeshedding season appears to be 1st January to 31st December,
inclusive. (No bikeshedding allowed during the leap second? :-)

I avoided suggesting "expected-failure" because it is actually not fully
consistent. In some test frameworks (Automake XFAIL_TESTS), success where
a failure was expected makes the overall test *fail*; in others (TAP tests
that report "not ok [n] # TODO"), it leaves the overall test succeeding,
possibly with a warning (because in principle now you can switch it to be
a normal test, if you trust it to be reliably passing, which of course you
don't necessarily for a while).

Because in the real world tests are not always 100% reliable, my priority
is to define something that I can correctly put on a test that
intermittently fails. If you want to have "must fail" *as well*, that
would be fine (but I'm probably not going to use that one).

It would also be nice if we had an exit status that marked a test as
skipped, perhaps 77 like in Automake - at the moment, a test that
probes its environment and determines that it can't run here has to
exit 0 and show up as passed. Would you be interested in a patch
for this?

> >     Features: known-bug=http://bugs.debian.org/1234
> 
> This won't modify the behaviour of the test, so a comment above Restrictions:
> would just do as well -- but as unknown features are ignored, this is fine of
> course.

My thought was that frontends (ci.debian.net) could maybe extract this
information and use it to present autopkgtest's raw results a bit more
helpfully.

> I think autopkgtest would show their result as "EXPECTED-FAIL" or so and exit
> with 0 (at least for that particular test case)?

Something like that, yes; either that, or report EXPECTED-FAIL or similar
in text, but pretend the test case was skipped when deciding what
exit-status autopkgtest should have? (So in practice it would exit 2)

    S