Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

[Martin Pitt]

Hello Simon,

Simon McVittie [2017-01-16 15:29 +0000]:
> I avoided suggesting "expected-failure" because it is actually not fully
> consistent. In some test frameworks (Automake XFAIL_TESTS), success where
> a failure was expected makes the overall test *fail*; in others (TAP tests
> that report "not ok [n] # TODO"), it leaves the overall test succeeding,
> possibly with a warning (because in principle now you can switch it to be
> a normal test, if you trust it to be reliably passing, which of course you
> don't necessarily for a while).

I'm leaning towards the second semantics -- i. e. a passing exfail test is
still a pass. If we need the first semantics we could add a "mustfail" instead
later on -- but this is just syntactic sugar, because if you actually expect
that you can just invert the exit code in your test yourself. (Of course you
can implement an exfail behaviour the same way, but this is the more important
variant IMHO).

> It would also be nice if we had an exit status that marked a test as
> skipped, perhaps 77 like in Automake - at the moment, a test that
> probes its environment and determines that it can't run here has to
> exit 0 and show up as passed. Would you be interested in a patch
> for this?

autopkgtest already defines exit code (bit mask) 2 for that, so it should use
that one. Returning 2 instead of 0 when encountering an exfail test sounds
again like a bikeshedding exercise to me, but I do agree that it's good to
mechanically determine if there were any exfailing tests; and as "0 or 2 is
success" has been a long-standing API now and being relied on in several CI
systems (at least debci, ubuntu's britney, and autopkgtest.ubuntu.com) I
wouldn't like to introduce a new code either. So I suppose "2" it is.

> > >     Features: known-bug=http://bugs.debian.org/1234
> > 
> > This won't modify the behaviour of the test, so a comment above Restrictions:
> > would just do as well -- but as unknown features are ignored, this is fine of
> > course.
> 
> My thought was that frontends (ci.debian.net) could maybe extract this
> information and use it to present autopkgtest's raw results a bit more
> helpfully.

Fine for me in principle. This would be a mere exercise in convention and
documentation. However, I don't like that it's independent from the proposed
restriction. I. e. that feature makes no sense without "exfail". How about
merging it:

  Features: exfail=http://bugs.debian.org/1234

and if a failing test has an "exfail" or "exfail=..." feature it gets treated as
"skipped" instead.

> Something like that, yes; either that, or report EXPECTED-FAIL or similar
> in text, but pretend the test case was skipped when deciding what
> exit-status autopkgtest should have? (So in practice it would exit 2)

Agreed.

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)