Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

[Simon McVittie]

Package: autopkgtest
Version: 4.3
Severity: wishlist

As mentioned in recent discussion on debian-devel, I don't think all
autopkgtests are sufficiently reliable, stable or important to be used to
gate CI or block testing migration. (In Debian terms: every test failure
is a bug, but not every bug is release-critical.) However, I don't want
to have to disable unimportant or known-unreliable tests altogether, and
lose the ability to see how their results change over time: I would prefer
to keep running them for the logs, and just ignore the side-effects.

Specifically, if a maintainer is unable to fix a particular unreliable
or broken test, or the non-RC bug that it exposes, I think it's still
correct for that test to continue to run on CI infrastructure, with
its result logged but the usual side-effects of that result ignored.
This can give the maintainer valuable feedback to send to upstream
(for example I can now tell ostree's author that
ostree/test-local-pull.sh.test fails around half the time, but the
other pull tests that sporadically failed have stopped failing since
a thread-safety issue in libsoup was fixed).

With the ability to ignore selected restrictions (#850494), this could
be addressed by a restriction, perhaps something like one of these:

    Restrictions: experimental
    Restrictions: intermittently-fails
    Restrictions: may-fail
    Restrictions: unimportant

possibly accompanied by:

    Features: known-bug=http://bugs.debian.org/1234

Tests with that restriction could perhaps be run normally, but
skipped when determining whether a batch of changes would break testing.

This is similar in concept to the TODO directive in TAP
<https://testanything.org/tap-specification.html>, which marks a
failure as not invalidating the overall success of the test.

    S