RFR: Britney - autopkgtest integration

[Paul Gevers]

Dear release team,

Since Dec 2016⁰ I have been working together with multiple people
(including nthykier) to prepare the integration of britney with
autopkgtest results. We believe the work has reached a state where full
deployment can be considered. Let me describe how it is designed to work
and what I request from you.

Design
======
We started with autopkgtest policy in the britney2 code as created by
Ubuntu. Ubuntu fully blocks migration of a package from the *-proposed
pocket to the full suite when the package introduces a regression in the
autopkgtest test suite of the package or any of its reverse
dependencies. During autopkgtesting, the new version of the package is
installed in the full suite. Of course hints can be used to ignore
failure or bad tests.

Although I believe we want such a gating behavior in Debian as well for
unstable to testing migration, we worked on implementing an alternative
scheme in parallel. Instead of allowing or blocking migration, this
scheme is decreasing or increasing the time a package has to age. If a
package has an autopkgtest and causes no regression, the package is
rewarded by having to spend less time in unstable before it is
considered a valid candidate. If a package causes regression (whether or
not it has an autopkgtest of its own), its required age is increased.
The additional time can be used to investigate the situation and fix it
(after all, the problem may or may not be with the new version of the
package).

Obviously we didn't teach britney how to run an autopkgtest. Instead, we
adapted the current framework on https://ci.debian.net (which until
recently only ran tests in unstable) to be able to take input from
britney, install the right package from unstable into testing and run
the required tests. (The required changes landed in the debci and
autopkgtest packages.)

Because packages that cause regression can migrate in this scheme, we
implemented a mechanism to reset the baseline of a package's autopkgtest
result. Every package has a special migration-reference trigger that
runs a package autopkgtest in testing (no packages from unstable). Every
DD (or actually, everybody that can use Debian's SSO), can request
retries of any test run, thus the baseline can easily be reset by
maintainers. Ideally, this should be done automatically (I'll work on
something, but I don't consider this blocking for deployment), but to
prevent the complexity of combining the right peaces of information in
britney, we didn't want to implement this in britney itself. When we
move to the gating scheme, this issue becomes less relevant as only
hinted packages get into this situation. (Ubuntu has a bug¹ open on
baseline resetting, I think what we know have is already an improvement
over their situation).

Request
=======
First of all, I'd like to request review of the changes I made to
britney to facilitate the integration with autopkgtest results. My work
lives on the autopkgtest branch of the github repository of britney². I
would appreciate it when you could have a look at the new code and
provide feedback.

Secondly, assuming we can come to deployment of the new britney policy,
I started a gobby document³ to draft an announcement to d-d-a. The
document has already been commented on by nthykier and terceiro (CI),
but I'd like it to be as good as possible.

Shadow britney
==============
I have started a shadow cron job of britney on respighi to see how
everything is behaving. I'll start filing bugs on regressions such that
the current code already has value, but the drawback of a shadow run is
that the information is very poorly discover-able for interested
parties. Also, I expect people to start noticing that ci.d.n is
generating more results, I think it would be great if we can explain
this in one go after britney has the autopkgtest code deployed.

Side note 1
===========
In order to convince myself (and others) of the proper functioning of
the code, I fixed the integration of the github repository with Travis⁴
as the default Travis framework was too old and testing was broken since
some changes last year.

Side note 2
===========
Because I had the hope, which is confirmed by now, that Ubuntu wants to
merge the changes in Debian back into their britney version, I mostly
kept around the code to support their workflow. Some changes are still
needed to limit (or ideally fully mitigate) the delta between Debian and
Ubuntu, as I wasn't always motivated to find a way before the confirmation.

Looking forward to your responses,
Paul

⁰ https://lists.debian.org/debian-release/2016/12/msg00310.html
¹ https://bugs.launchpad.net/britney/+bug/1700668
² https://github.com/Debian/britney2/tree/autopkgtest
³ https://gobby.debian.org/export/Teams/Release/using-autopkgtest
⁴ https://travis-ci.org/Debian/britney2/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20180401/39bfcc24/attachment.sig>