Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI
Simon McVittie
smcv at debian.org
Mon Jan 16 15:29:32 UTC 2017
On Mon, 16 Jan 2017 at 12:34:50 +0100, Martin Pitt wrote:
> Maybe "expected-failure" which is the term used by other test runners? (The
> bikeshedding season is on! ☺ )
The bikeshedding season appears to be 1st January to 31st December,
inclusive. (No bikeshedding allowed during the leap second? :-)
I avoided suggesting "expected-failure" because it is actually not fully
consistent. In some test frameworks (Automake XFAIL_TESTS), success where
a failure was expected makes the overall test *fail*; in others (TAP tests
that report "not ok [n] # TODO"), it leaves the overall test succeeding,
possibly with a warning (because in principle now you can switch it to be
a normal test, if you trust it to be reliably passing, which of course you
don't necessarily for a while).
Because in the real world tests are not always 100% reliable, my priority
is to define something that I can correctly put on a test that
intermittently fails. If you want to have "must fail" *as well*, that
would be fine (but I'm probably not going to use that one).
It would also be nice if we had an exit status that marked a test as
skipped, perhaps 77 like in Automake - at the moment, a test that
probes its environment and determines that it can't run here has to
exit 0 and show up as passed. Would you be interested in a patch
for this?
> > Features: known-bug=http://bugs.debian.org/1234
>
> This won't modify the behaviour of the test, so a comment above Restrictions:
> would just do as well -- but as unknown features are ignored, this is fine of
> course.
My thought was that frontends (ci.debian.net) could maybe extract this
information and use it to present autopkgtest's raw results a bit more
helpfully.
> I think autopkgtest would show their result as "EXPECTED-FAIL" or so and exit
> with 0 (at least for that particular test case)?
Something like that, yes; either that, or report EXPECTED-FAIL or similar
in text, but pretend the test case was skipped when deciding what
exit-status autopkgtest should have? (So in practice it would exit 2)
S
More information about the autopkgtest-devel
mailing list