[Pkg-octave-devel] First try on autopkgtest for OF packages

[Rafael Laboissière]

* Sébastien Villemot <sebastien at debian.org> [2017-08-29 10:28]:

> On Sun, Aug 27, 2017 at 05:04:34PM +0200, Rafael Laboissière wrote:
>>
>> Yes, I confirm that it does not catch test failures.  In order to do that, 
>> we must change the check-pkg script, such that it would be able to collect 
>> the failures and optionally exit with an error code if any of the tests 
>> fails.
>
> Ok, sounds good. Note that we will need to whitelist some test failures in some 
> packages, which can probably be achieved by patching the .m file and use the 
> expected failure functionality of Octave.

Yes, I think that patching the file (and eventually forwarding fixes 
upstream) is the right way to go.  The test function of Octave is quite 
powerful.  I looked at it and I think that it should be possible to 
collect the results of tests and allow check-pkg to make use of the 
information.  I have though to have a sound design for the 
implementation.

> I am also wondering if we should not make test failures fatal at build 
> time. This would probably cause a few FTBFS in the short term, but in 
> the longer run this would improve the quality of the packages.

Yes, absolutely.  If a unit test fails, then either the functionality or 
the test are wrong and must be fixed.

Rafael