autopkgtest: Generic Test Harness / Network Client Service testing

Martin Pitt mpitt at debian.org
Wed Sep 16 07:56:37 UTC 2015


Hello,

sorry for the ridiculously long time to reply -- we discussed that at
debconf already, but for the sake of the mail archives I'll put that
here too.

Christian Seiler [2015-06-26 15:32 +0200]:
> The patch itself contains a file debian/tests/testlib.py, which is
> basically a python library providing all sorts of useful helpers for
> testing daemons etc. But it's also >1000 lines long and there are a
> couple of things in there that I'd like to improve. A quick codesearch
> in Debian revealed that this library has already been copied to 3 other
> Debian packages:
> http://sources.debian.net/src/postfix/2.11.3-1/debian/tests/testlib.py/
> http://sources.debian.net/src/nut/2.7.2-4/debian/tests/testlib.py/
> http://sources.debian.net/src/mailman/1:2.1.20-1/debian/tests/testlib.py/
> 
> Personally, I think having an embedded copy of the whole thing is
> probably a bad idea in the long run - especially if I start improving
> it for the convenience of the package I'm currently working on,
> effectively creating divergent versions of the same piece of software.

Centralization is usually a good thing indeed. But I'm not really
convinced that this particular testlib.py is a good idea -- a lot of
the functions are implemented poorly (eww @ check_port() or
is_kdeinit_running() for example), duplicate already existing
functionality from the standard lib (get_arch(), recursive_vm(),
get_md5(), etc.), encourage bad/unsafe ways of restoring temporary
changes to files (config_replace(), etc.), duplicate already existing
autopkgtest functionality (prepare_source() -- eww eww eww!), and has
lots of little and unnecessary wrappers which just make your test
harder to understand (is_apparmor_loaded()).  It is a rather wild and
unstructured pile of functionality ranging from apparmor to KDE, and
most tests won't ever need most of this functionality.

Making this an official API would mean that this first has to be
essentially rewritten from scratch in a more proper way and leaving
out all the "abstraction for abstraction's sake" wrappers -- in tests
it's almost always better to directly say what you want to do instead
of relying on third-party abstractions which can change underneath
you. It would also need to grow a test suite to guarantee API
stability.

> I think it would make sense to package that - and autopkgtest seems to
> be the best place for that (maybe an additional binary package?). What
> do you think?

TBH, no. This is on a completely different abstraction level.
autopkgtest is a test execution mechanism, and its interface to a test
is a blackbox executable. It should *not* get into the business of
*how* you write your tests -- that's the level of e. g. Python's
unittest, shunit2, JUnit, etc. Conversely, the above kind of "test
utilities" is in no way specific to autopkgtest -- it applies equally
well to tests that you run during "make check", for example. Something
like this would fit much better into e. g. python-testtools, or
perhaps a new python library.

So I'm afraid I'm pretty firmly against this idea. I would not
recommend anyone to use the particular testlib.py due to its poor
quality and overuse of abstraction, and it is not related to
autopkgtest itself.

> Can one tell autopkgtest to provision two VMs? One which is configured
> to provide some generic services - and then a second VM that contains
> the package itself (both can see each other and IP addresses of the own
> and other VMs are available for scripts) in order to test whether it
> works properly against the service that's set up.

We discussed that once specifically with the QEMU runner: There was a
patch to export its own "outside" image into the VM, so that your test
can depend on qemu-system and run a nested VM inside the main one. But
at that time nested KVM was still rather brittle (I think it's much
better these days).

However, that would limit the test to QEMU. These days we have
containers, which are much more widely applicable. For these cases,
would it be enough to put the server end (NFS/iscsi etc.) into a
container that runs in the test environment, and then use that from
the main test environment?

nspawn containers are exceptionally easy/fast when the test
environment is btrfs, but I think they are still reasonably easy to
set up (without an explicit image download, I mean) on other file
systems. And with LXC, and nspawn you can always debootstrap and
apt-get install a small OS for the nested guest.

It becomes more interesting if you need that server for booting the
client -- in this case you'd need to construct a client-side container
instead, and leave the server bits in the main test environment. That
would obviously not work testing for PXE boot (as containers don't do
that), but it should be sufficient for things like NFS or even
open-iscsi?

> If that's not possible, do you think this idea has some merit?

Yes, absolutely.

Thanks,

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)



More information about the autopkgtest-devel mailing list