Bug#852820: Testsuite-Restrictions field is hard to use

[Ian Jackson]

Iain Lane writes ("Re: Bug#852820: Testsuite-Restrictions field is hard to use"):
> I don't, or not really - see below. I plan on using this field
> externally to choose between a couple of available backends to dispatch
> to when constructing autopkgtest invocations

Yes, I understand that.  But I think this field is too tailored to
your specific use case and too ripe for misuse.

> > https://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/plain/doc/README.package-tests.rst says:
> > Tests which declare unknown restrictions will be skipped. See below
> > for the defined restrictions.
> 
> So it would seem to me (unless I'm missing something in there like an
> exception for x-prefixed items that I can't see ATM) that this would be
> legal and as a result I was not really expecting arbitrary values to be
> present. Again, I'm not going to be doing that, so it's not a problem
> now and this is apparently not currently implemented in autopkgtest
> itself either, since dgit's tests are being run.

Only a few of dgit's tests have these restrictions.  The tests with
those restrictions are not run by Debian's CI.  They can be run in
ad-hoc situations of various kinds.

Declaring test restrictions, using a private bit of the namespace, in
this way, seems like an appropriate way to achieve that effect.

For the avoidance of doubt: I'm not saying that your program would
mishandle correct packages.  (Is it happy with dgit's test suite,
despite the presence of some un-runnable tests?)

But as I say this field seems to invite people to use it for deciding
whether the package is testable, but it is not useable for that
purpose.

To put it another way: why not publish the intersection of the
restrictions ?  Why not also publish the union or intersection of the
features ?  As I wrote:

> > > If we do need more information in Sources.gz then maybe the set of
> > > combinations of restrictions ought to be listed.  But then the
> > > features might be useful too.

ISTM that your real problem is that you don't have an efficient way to
access debian/tests/control.  Perhaps the right answer is to provide
an additional side channel for this information, rather than burdening
the Sources file.  Many many, non-testing-related, programs need to
read Sources.  I think the test metadata is rather too rich to express
compactly.

> > But I certainly agree that this should have probably been discussed
> > more widely, which is something I overlooked, sorry about that. And
> > agree completely with your two points above. So I'll be adding an
> > entry to the FAQ detailing the process to add new information to
> > the .dsc file (and probably to the .changes and other interchange
> > formats).
> 
> Yes, I concede that I should have posted on the list. I discussed with
> Martin first, but that is not enough. Sorry about that.

Thanks.

Ian.