Bug#851556: autopkgtest: please add Restrictions for network access, or Features for lack of network access

[Martin Pitt]

Control: tag -1 wontfix -patch

Hello Simon,

Simon McVittie [2017-01-16  8:47 +0000]:
> There is some controversy over the extent to which it is OK for
> packages to touch the network at build-time, in terms of privacy
> (leaking information), determinism of builds, robustness against network
> services failing and so on. (See also #833503, #850988). autopkgtest's
> Restrictions and Features mechanisms give it the opportunity to do better:
> one interpretation can be the default, and the other can be selected
> by selectively ignoring requirements or requiring features.

The problem with that is that such a restriction cannot be enforced properly
either way, as that's a function of the environment you run autopkgtest in.
This was already discussed several times, last time in
<https://bugs.debian.org/801895>.

> As background for this, I recently implemented tests in libnss-mdns[1]
> which add and remove packages, which was necessary to test the postinst
> and its interactions with libnss-resolve. #786039 suggests that this is
> considered to be OK in autopkgtest.

Yes, both in Debian's and Ubuntu's production setups we offer network access,
although in Ubuntu you need to go through a proxy.

> I attach some initial patches for discussion, based on the assumption that
> Restrictions are the right way to do it.

I'm firmly of the opinion that they aren't, as you can never enforfce it in
autopkgtest. What would be acceptable is to declare this as a "Features:", as
these have only an advisory character (and aren't being used much either).

> Subject: [PATCH 1/2] doc: Define new uses-network restriction

Doing it this way around would break backwards compatibility with all the
existing tests which use the network -- there are a dozen or two, and so far
we've said that network access is okay (and sometimes unavoidable for testing
in a meaningful way).

This is not really a "patch" for this bug, as there is no actual implementation
and testing of this behaviour (and I'm fairly convinced that there can't be),
so untagging.

> +uses-network
> +    The test might contact other machines, so is not suitable to
> +    be run in an environment where privacy is essential.

I don't buy this argument at all, I must say. Running a test (almost)
unavoidably need to access other machines to prepare the test env and install
test dependencies. If you have an environment where you are afraid of some
network logging, don't run tests on it (or use or do development on that
package).

> Subject: [PATCH 2/2] doc: Add restrictions for using or reconfiguring apt
> 
> nss-mdns should use this in its autopkgtests, which exercise
> different orderings for installation of libnss-mdns vs. libnss-resolve,
> and make sure that upgrading from jessie's libnss-mdns works.

This seems unrelated. But this makes no sense at all as a restriction -- you
can't control what the tests do, and they can change packages or call apt (or
dpkg, or rm, or "make install") in any programmatic way. Such a test should add
"Restrictions: breaks-testbed" if it's doing something dangerous, but this is
not practically meaningful as these days pretty much all tests run in a
container or QEMU anyway.

Tests could add this to Features: as an indication to human readers, but it has
no consequence to the machinery.  IMHO, starting to try and categorize tests
like that would quickly end up in a big pile of tags like "installs-files",
"starts-services", "uses-lots-of-ram" and whatnot :-), so I don't want to
encourage that.

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)