Fwd: autopkgtest: Generic Test Harness / Network Client Service testing

[Ian Jackson]

Christian Seiler writes ("Fwd: autopkgtest: Generic Test Harness / Network Client Service testing"):
> Dear autopkgtest Maintainers,
> (adding explicit Cc: to Uploaders of autopkgtest)
> 
> I'd like to ping you again, since I didn't receive a reply to my
> previous message (see below). If this isn't the proper venue for
> discussing this, please direct me to where this is more appropriate.

Hi.  Sorry I didn't reply to your mail before.  I'm not very heavily
involved with autopkgtest right now, but I can try to answer your
questions.

> The patch itself contains a file debian/tests/testlib.py, which is
> basically a python library providing all sorts of useful helpers for
> testing daemons etc. But it's also >1000 lines long and there are a
> couple of things in there that I'd like to improve. A quick codesearch
> in Debian revealed that this library has already been copied to 3 other
> Debian packages:
> http://sources.debian.net/src/postfix/2.11.3-1/debian/tests/testlib.py/
> http://sources.debian.net/src/nut/2.7.2-4/debian/tests/testlib.py/
> http://sources.debian.net/src/mailman/1:2.1.20-1/debian/tests/testlib.py/

Erk.

> Personally, I think having an embedded copy of the whole thing is
> probably a bad idea in the long run - especially if I start improving
> it for the convenience of the package I'm currently working on,
> effectively creating divergent versions of the same piece of software.

Yes, absolutely!

> I think it would make sense to package that - and autopkgtest seems to
> be the best place for that (maybe an additional binary package?). What
> do you think?

Yes, probably.

> 2. In order to properly test open-iscsi, one needs to be able to log in
> to an iSCSI target. The patch in bug[1] doesn't really do this by
> default, it only tests logging in to the target if one modifies the
> test script and manually hacks in a host name - which will definitely
> not be generic enough.
> 
> Of course, I could simply install targetcli (from LIO) on the same host
> and log in to localhost, but then it would be really hard to expand the
> automatic tests to situations like whether iSCSI login at boot works
> properly or even having root on iSCSI or the such. I think a couple of
> other services would have similar problems (think NFS etc.) if
> autopktest were to be thoroughly implemented for them.
> 
> Can one tell autopkgtest to provision two VMs? One which is configured
> to provide some generic services - and then a second VM that contains
> the package itself (both can see each other and IP addresses of the own
> and other VMs are available for scripts) in order to test whether it
> works properly against the service that's set up.

There is not currently such a facility.

> If that's not possible, do you think this idea has some merit?

I think this is an interesting idea but it would require substantial
design work to extend the test protocol to support this functionality.

Regards,
Ian.