Let autopkgtests be gating for testing migration in Buster: heads-up and brain-dump

[Paul Gevers]

Hi all,

I already started this discussion on IRC (in both #debci and
#debian-release), but for future reference, and some brain-dump I want
to continue on the lists.

As most of you have probably noticed by now, I have taken up the task to
have Debian let debci/autopkgtest results be gating for
unstable-to-testing migration (very similar to what Ubuntu already does
for <release>-proposed-to-<release> migration). I would really love to
see this happen early in the Buster release cycle. I have started a wiki
page¹ to document the requirements and the (outcome of the) discussions.
I'll try to keep that up-to-date with as we progress. As it is a wiki,
feel free to contribute anything that isn't controversial.

As I currently see it, from discussion with multiple of you, there are
three pieces that need to play together in Debian:
1) autopkgtest: the testing platform
2) debci: the Debian CI worker
3) britney: where the migration policy is implemented and enforced

Currently debci is testing packages in unstable. But similar to what
Ubuntu is doing, I think that for this purpose we actually want to test
in testing with only the possible candidate (and if needed it's
dependencies) from unstable. Luckily, autopkgtest is nearly able to do
that, except it needs to support Debian suites instead of only Ubuntu's
"pockets".

Apart from testing testing (no typo), and again copying Ubuntu, I think
we want to run the test suites of the candidate *and* of all the reverse
dependencies that have test suites. This way, you'll see when a
candidate deteriorates testing. For this, debci need some changes: it
needs a testing suite environment and it needs to call autopkgtest with
the additional arguments. To enable debci to know *what* should be
tested, britney must communicate to debci.

Finally, of course, britney needs to be aware of debci results and take
them into account during judgment.

I think the logical order to for me to tackle (I mean, create the code)
this is:
0) figure out how to test all of this without breaking the real
instances (hints more than welcome).
1) fix autopkgtest to enable --apt-suite (next to the current --apt-pocket)
In parallel:
2a) getting a testing suite up for debci and extend debci to be aware of
the additional arguments for autopkgtest
2b) let britney generate a list of tests it would like to perform
2c) align on the transfer mechanism between britney(1) and debci
3) enable debci to swallow the commands from britney
4) enable the policy in britney

Opinions? Other ideas?

If not too many objections, I'll start with 0 and 1. And I sincerely
hope that Antonio wants to help with 2a, but I'd like to hear his
thoughts first. I already had a extensive discussion with pitti and he
is guiding me through the Ubuntu code, which I think serves as a great
example.

I'll probably be back to you way before we are past point 2a.

Paul

PS: as default on Debian mail-lists, no need to CC me, I am subscribed
to the autopkgtest-devel list.

¹ https://wiki.debian.org/debci/britneyIntegration

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20161220/c0452a8b/attachment.sig>