DEP8 tests using the built package source

[Martin Pitt]

Stephen Kitt [2014-03-19 23:54 +0100]:
> On Wed, 19 Mar 2014 13:49:52 +0100, Andreas Tille <andreas at an3as.eu> wrote:
> > On Wed, Mar 19, 2014 at 11:47:02AM +0100, Jakub Wilk wrote:
> > > Long answer:
> > > 
> > > You can declare that a test needs to be run from a built source
> > > tree. Then the test runner will build the package. But that doesn't
> > > necessarily mean that the built binaries will be used for anything.
> > > 
> > > Now, adt-run(1) has multiple modes of operation. In some of them the
> > > built binaries are used to satisfy tests' dependencies, in others
> > > packages from the archive are used. (This is super confusing. :/)

Well, it depends what you want to do. In CI we *always* want to test
the packages from the actual archive of course. But as a package
maintainer you may want to (and should) run your autopkgtest for an
updated package *before* you upload it to the archive, and you want to
test them with locally built binaries instead.


> Likewise for me. I find it particularly complicated to imagine what variants
> of adt-run should be used to make sure a particular test-suite is suitable
> for the CI platforms (ci.debian.net and Ubuntu's), which basically means
> having to look up the source code of the platforms to figure out what's
> going to happen, or uploading a package and waiting for the results...

Recent versions of autopkgtest now ship the new
/usr/share/doc/autopkgtest/README.running-tests.gz which I hope
explains the use cases and how to run them.

> > I also hope so.  We recently had a discussion about biopython[1] whether
> > to run dh_auto_test or not if autopkgtest exists.  I'm in clear favour
> > of running dh_auto_test and based my arguing on the assumption that
> > autopkgtest is testing the binary packages.  I'd be happy to hear the
> > opinion of the autopkgtest experts about this.
> 
> I'm not an expert, but I do agree with you. If a package includes a
> test-suite which can run during the build, then it might as well be run then,
> since that should avoid uploading broken packages (or flag them very early).

Yes, absolutely. This will also give you coverage on all
architectures, and avoid landing a package even in unstable as failed
tests should fail the build.

autopkgtest does not supersede running upstream tests against the
built tree, it just complements them with testing scenarios that you
can't do during package build.

> To me, autopkgtest test-suites are nice for two things:
> * tests which can't be run within a build (I started looking into all this
>   for libevdev, which includes a test-suite that must be run as root);
> * integration tests which combine multiple binary packages (e.g. Martin's
>   example with the toolchain; I'm also working on a toolchain test-suite for
>   mingw-w64, which would combine the mingw-w64 toolchain and wine to make
>   sure the toolchain is working correctly).

Yes, for these. But also, you can do reverse dependencies testing and
thus ensure that package A still works if one of its dependencies (say
libfoo) gets updated, to prevent libfoo from landing if it breaks API.
Uploading libfoo won't usually cause an upload/build of A otherwise,
so running tests during build won't catch this.

Also, autopkgtests test *packages* as they will appear on an actual
system, unlike tests during build. The latter won't tell you if you
forgot to install some files or put them into the wrong place, or your
package has a missing Depends.

> I'm wondering how all this meshes with Ubuntu's "fast migration" approach:
> I'm guessing it doesn't see build-time tests which are run as part of the
> build... 

It kind of does by rejecting packages which don't build on all
platforms, i. e. if tests during build fail on a particular arch.

> Would it be worth running them as autopkgtest tests *as well* for
> this use case?

If you can, sure. Otherwise a quick smoke test is usually enough. Let
the upstream tests during package build cover the fine details, and
just let autopkgtest check that your package works in general (i. e.
you can start a program, import a Python module, build a simple
5-liner against libfoo-dev, and so on).

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140320/6cba63b0/attachment.sig>