Let autopkgtests be gating for testing migration in Buster: heads-up and brain-dump

[Julien Cristau]

On Sun, Jan  1, 2017 at 10:55:36 +0100, Martin Pitt wrote:

> Hello all,
> 
> Julien Cristau [2016-12-31 17:45 +0100]:
> > > > 2b) let britney generate a list of tests it would like to perform
> > > Is 2b really necessary?  I'm not sure why we would need that.
> > > 
> > Or to be more correct, I'm not sure we would *want* britney to be able
> > to (or have to) do that.
> 
> I find britney the best place for it both 
> 
>   conceptually: it's the piece that makes policy decisions and has overrides
>   for broken/ignored tests
> 
> as well as
> 
>   operationally: it already has knowledge which package groups want to
>   propagate and has an efficient way to compute reverse dependencies and other
>   information such as Testsuite-Triggers: from source packages; and it's the
>   place which has to interpret these results, so it must know which ones to
>   look for anyway.
> 
> It's IMHO not the place of the test execution engine to prescribe which tests
> to run, as that limits its utility to just one use case (and we use Ubuntu's
> engine for lots of other cases that just britney testing migration) and
> duplicates/potentially disagrees with logic and decisions that britney already
> makes.
> 
> TBH I'm a bit surprised by your remark, I would have thought it was quite
> obvious that this should be under the release team's/britney's control -- but
> apparently not then. Where would you like this policy/decisions to live?
> 
I'm happy to have britney consume data from the archive / BTS / piuparts
/ eventually CI.  I'm not so sure about adding a feedback loop where
britney says "this package would be a candidate, but first I'm going to
request a CI test, then wait for some time for that result to be
available, then make it an actual candidate" (at which point maybe the
package set has changed so you're waiting for yet another test result).
And if you're not talking about such a feedback loop, then it seems to
me the CI system could figure out which tests to run on its own.  What
am I missing?

Cheers,
Julien