DEP8 tests using the built package source

[Stephen Kitt]

Hi Andreas,

On Wed, 19 Mar 2014 13:49:52 +0100, Andreas Tille <andreas at an3as.eu> wrote:
> On Wed, Mar 19, 2014 at 11:47:02AM +0100, Jakub Wilk wrote:
> > Long answer:
> > 
> > You can declare that a test needs to be run from a built source
> > tree. Then the test runner will build the package. But that doesn't
> > necessarily mean that the built binaries will be used for anything.
> > 
> > Now, adt-run(1) has multiple modes of operation. In some of them the
> > built binaries are used to satisfy tests' dependencies, in others
> > packages from the archive are used. (This is super confusing. :/)
> 
> +1 for confusion.

Likewise for me. I find it particularly complicated to imagine what variants
of adt-run should be used to make sure a particular test-suite is suitable
for the CI platforms (ci.debian.net and Ubuntu's), which basically means
having to look up the source code of the platforms to figure out what's
going to happen, or uploading a package and waiting for the results...

> > I hope that ci.debian.net is configured in such a way it uses binary
> > packages from the archive.
> 
> I also hope so.  We recently had a discussion about biopython[1] whether
> to run dh_auto_test or not if autopkgtest exists.  I'm in clear favour
> of running dh_auto_test and based my arguing on the assumption that
> autopkgtest is testing the binary packages.  I'd be happy to hear the
> opinion of the autopkgtest experts about this.

I'm not an expert, but I do agree with you. If a package includes a
test-suite which can run during the build, then it might as well be run then,
since that should avoid uploading broken packages (or flag them very early).

To me, autopkgtest test-suites are nice for two things:
* tests which can't be run within a build (I started looking into all this
  for libevdev, which includes a test-suite that must be run as root);
* integration tests which combine multiple binary packages (e.g. Martin's
  example with the toolchain; I'm also working on a toolchain test-suite for
  mingw-w64, which would combine the mingw-w64 toolchain and wine to make
  sure the toolchain is working correctly).

I'm wondering how all this meshes with Ubuntu's "fast migration" approach:
I'm guessing it doesn't see build-time tests which are run as part of the
build... Would it be worth running them as autopkgtest tests *as well* for
this use case? Then again, if packages without tests aren't treated
differently, there would be no advantage to making tests' existence explicit
in all cases.

> Moreover I observed another issue with autopkgtest which is quite
> astonishing to me:  In bug #741274 it was reported that the autopkgtest
> would fail and the according log is here:
> 
>    http://ci.debian.net/data/unstable-amd64/packages/p/python-pysam/2014-03-12.log
> 
> The problem is that `make` was not available in the chroot (obviosly)
> which does not sound very reasonable to me.  While I added it to the
> Depends in debian/tests/control I think it is not sensible to assume
> that make exists in a build chroot (it is not in the Build-Depends) but
> trying to build the package somehow and than notice that it is missing.
> I admit that I'm really confused how autopkgtests are working.

This is actually documented, sort of: the documentation for @builddeps@
mentions make as one of the packages which gets pulled in. (Why not
build-essential? Yet somehow gcc is available...)

Regards,

Stephen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140319/a45eb5eb/attachment.sig>