Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI

[Simon McVittie]

On Sun, 22 Jan 2017 at 16:55:09 +0100, Martin Pitt wrote:
> > It would also be nice if we had an exit status that marked a test as
> > skipped, perhaps 77 like in Automake - at the moment, a test that
> > probes its environment and determines that it can't run here has to
> > exit 0 and show up as passed. Would you be interested in a patch
> > for this?
> 
> autopkgtest already defines exit code (bit mask) 2 for that, so it should use
> that one. Returning 2 instead of 0 when encountering an exfail test sounds
> again like a bikeshedding exercise to me

Sorry, I didn't make myself clear. What I mean is: suppose I want to
test that some tool (ostree or tar or whatever) can back up and restore
xattrs. I can't test that unless I have some scratch area that lets the
current user set xattrs. So I might write a test like this:

    #!/bin/sh

    touch "$ADTTMP/file"

    if ! setfattr -n does-this-work -v value "$ADTTMP/file" 2>/dev/null; then
        echo "Can't run this test, $ADTTMP doesn't support extended attributes"
        exit 77
    fi

    # ... actual test elided

(Or in a real example I might probe $ADTTMP, /tmp and /var/tmp, and use the
first one that does support xattrs, if any - but you get the idea.)

Or, imagine I want to test a non-http network protocol, perhaps testing
whether my XMPP client can connect to an XMPP server. As we discussed
on #851556, we would expect this to work on ci.debian.net, but not on
Ubuntu's equivalent. I could probe for this with ": | nc xmpp.debian.net 5222"
or something.

With Automake, I would be able to exit 77 to get the test recorded as
having been skipped, but with current autopkgtest I would have to exit 0.

This would hopefully affect autopkgtest's exit status the same way as if
the test had been skipped due to "Restrictions: something-you-dont-support" -
so if one or more tests exit 77, autopkgtest would exit 2.

> How about merging it:
> 
>   Features: exfail=http://bugs.debian.org/1234
> 
> and if a failing test has an "exfail" or "exfail=..." feature it gets
> treated as "skipped" instead.

The reason I suggested a Restriction was that until the test-runner has
been upgraded to an autopkgtest version that knows the new syntax, the test
would be skipped (not run) because of the unknown restriction. But I can
see your point, and your proposed syntax seems good.

    S