autopkgtest: Generic Test Harness / Network Client Service testing

Martin Pitt mpitt at debian.org
Sat Oct 31 16:52:44 UTC 2015


Hey Christian,

sorry for the reeeally late answer about the testlib.py, cleaning up
my mailbox on a long flight..

Christian Seiler [2015-09-17 18:34 +0200]:
> To me unit testing frameworks are orthogonal to what autopkgtest does:
> I would put autopkgtest more in the realm of "integration testing"

It is certainly for running integration tests in a defined
environment, but it is not a tool to *write* integration tests.

> Currently, autopkgtest only provides the execution environment, but
> does not provide any helpers for writing tests.

Right, in in the sense of "separation of concerns" and maintaining
flexibility the defined interface is an executable and its exit
status. It should not start to define how tests are written, as there
are a lot of different frameworks for that.

> But for the testing of packages there are several typical actions
> that many people will have repeatedly: checking if a daemon is
> running after installation, checking if a daemon has actually opened
> the appropriate socket, modifying configuration files while keeping
> backups so they may be restored after test completion, interacting
> with debconf, etc. Those types of operations are not operations you
> typically do in unit tests.
> 
> So I do think that something like that could be very useful.

I agree -- there is certainly some utility in generalizing such things
into a test helper library, e. g. an extension of python's unittest or
shunit2. I just claim that this has got nothing to do with
DEP-8/autopkgtest.

So while it certainly might be useful to have such a test helper
library, there is no reason to couple it to autopkgtest, as they do
different things on different levels.

> As said above: after taking a closer look, I agree that testlib.py is
> not it. But that doesn't mean that I think it would be pointless to
> add something along those lines. And it would be something that would
> have to be written from scratch - so I'm not proposing to add something
> immediately - I'm just asking you to reconsider your opinion on this
> issue generally.

Sorry if it sounded differently -- there's nothing to reconsider, we
agree :-). Well, I haven't actually personally felt the need for such
a lib, but other authors of tests certainly have -- after all, someone
wrote this testlib.py thing.

I just don't think it makes any sense to call/bundle this with
the autopkgtest code itself, it should be a separate project. (That
doesn't mean that it can't be discussed/hosted on pkg-autopkgtest of
course, just that it should be a separate git tree/source package).

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)



More information about the autopkgtest-devel mailing list