[Pkg-octave-devel] First try on autopkgtest for OF packages

[Oliver Heimlich]

On 29.08.2017 10:45, Rafael Laboissière wrote:
> * Sébastien Villemot <sebastien at debian.org> [2017-08-29 10:28]:
> 
>> On Sun, Aug 27, 2017 at 05:04:34PM +0200, Rafael Laboissière wrote:
>>>
>>> Yes, I confirm that it does not catch test failures.  In order to do
>>> that, we must change the check-pkg script, such that it would be able
>>> to collect the failures and optionally exit with an error code if any
>>> of the tests fails.
>>
>> Ok, sounds good. Note that we will need to whitelist some test
>> failures in some packages, which can probably be achieved by patching
>> the .m file and use the expected failure functionality of Octave.
> 
> Yes, I think that patching the file (and eventually forwarding fixes
> upstream) is the right way to go.  The test function of Octave is quite
> powerful.  I looked at it and I think that it should be possible to
> collect the results of tests and allow check-pkg to make use of the
> information.  I have though to have a sound design for the implementation.
> 
>> I am also wondering if we should not make test failures fatal at build
>> time. This would probably cause a few FTBFS in the short term, but in
>> the longer run this would improve the quality of the packages.
> 
> Yes, absolutely.  If a unit test fails, then either the functionality or
> the test are wrong and must be fixed.

I totally agree on that one.  The interval package (where it is of
particular importance that it produces correct results) has an extensive
test suite and the build fails if one of the tests fails.

https://anonscm.debian.org/cgit/pkg-octave/octave-interval.git/tree/debian/check.m

The test suite used to be located in .tst files that are checked by this
script.  With version 3.0.0 the test suite has moved to standard BISTs.
If these are checked by check-pkg with fail on test failure, that'd be
great.

Oliver