[Pkg-octave-devel] [rafael at debian.org: Re: First try on autopkgtest for OF packages]

[Rafael Laboissière]

[I sent the message below earlier today but it was probably blocked due 
to the oversized attached file.  The said file is available at 
https://wiki.debian.org/Teams/DebianOctaveGroup?action=AttachFile&do=view&target=octave-image-autopkgtest-log.txt]

Rafael

----- Forwarded message from Rafael Laboissière <rafael at debian.org> -----

From: Rafael Laboissière <rafael at debian.org>
Subject: Re: [Pkg-octave-devel] First try on autopkgtest for OF packages
Date: Wed, 30 Aug 2017 16:21:59 +0200
To: pkg-octave-devel at lists.alioth.debian.org
Organization: Debian GNU/Linux
Message-ID: <20170830142159.ho4fijqgzq2dxzuk at laboissiere.net>

* Oliver Heimlich <oheim at posteo.de> [2017-08-30 09:09]:

> On 29.08.2017 10:45, Rafael Laboissière wrote:
>> * Sébastien Villemot <sebastien at debian.org> [2017-08-29 10:28]:
>>
>>> I am also wondering if we should not make test failures fatal 
>>> at build time. This would probably cause a few FTBFS in the 
>>> short term, but in the longer run this would improve the 
>>> quality of the packages.
>>
>> Yes, absolutely.  If a unit test fails, then either the 
>> functionality or the test are wrong and must be fixed.
>
> I totally agree on that one.  The interval package (where it is of 
> particular importance that it produces correct results) has an 
> extensive test suite and the build fails if one of the tests fails.

Ok, with my last commits on the octave-pkg-dev repository, check-pkg has 
now the ability to collect the results of the tests and to exit with an 
error code when any of the tests fails (after running all of them, of 
course).

With this version of octave-pkg-dev installed in the system, the following 
command run from the autopktest branch in the octave-image package:

autopkgtest -B . -- null

produces the log attached below.

Notice that at the end of the log file, there is the following text, 
showing that the autopkgtest fails:

Summary: 1169 tests, 1134 passed, 21 known failures, 0 skipped
/usr/share/cdbs/1/class/octave-pkg.mk:108: recipe for target 
'check-pkg' failed 
make: *** [check-pkg] Error 1 
autopkgtest [11:06:09]: test unit-tests: -----------------------] 
autopkgtest [11:06:09]: test unit-tests:  - - - - - - - - - - results 
- - - - - - - - - - 
unit-tests           FAIL non-zero exit status 2 
autopkgtest [11:06:09]: @@@@@@@@@@@@@@@@@@@@ summary 
unit-tests           FAIL non-zero exit status 2

The code in check-pkg is kind of convoluted.  Take this a proof of concept.

Comments are welcome.

Rafael