RFR: Britney - autopkgtest integration

Fri Apr 6 08:10:30 UTC 2018

Hi Raphael

On 05-04-18 14:48, Raphael Hertzog wrote:
> On Sun, 01 Apr 2018, Paul Gevers wrote:
>> (including nthykier) to prepare the integration of britney with
>> autopkgtest results. We believe the work has reached a state where full
>> deployment can be considered. Let me describe how it is designed to work
>> and what I request from you.
> 
> Huge thanks for this work! I'm looking forward to it and I plan to deploy
> it for use by Kali too.

Great.

> Hence I would like to know a bit more how it all ties together with debci.
> I have looked quickly at the code and I saw references to the AMQP
> messaging queue, to a REST API on the debci side, and also to swift.
> Can you quickly describe the workflow between all those components and
> britney?

In contrast to Ubuntu, Debian's britney2 doesn't talk to the outside
world. So britney2 takes input from files and writes output to files. In
the case for autopkgtest integration the following files are relevant:

<state>/debci.json -> this is a file obtained from ci.d.n with the
current status via the API interface of debci
(ci.d.n/documentation/API). britney1 is responsible for getting that
file just before it calls britney2.

<output>/debci.input -> this is the file created by britney2 in the same
format as Ubuntu uses for its AMQP messaging queue to transfer the info
to ci.d.n. britney1 converts this file into the input commands suitable
for the debci.

Antonio and I decided that debci shouldn't care about what anybody
considers as a trigger. So the only thing it cares about is which suite
to use as baseline and which packages (and relevant dependencies) to
take from another suite.

Swift isn't used in the Debian setup, that is how Ubuntu does its thing
(communicates results back to britney2).

So in Debian:
britney2 <-> britney1 <-> debci <-> autopktest
          ^            ^
        files      REST API
in Ubuntu
britney2 <-> debci/?? <-> autopktest
          ^
     AMQP/swift

> How does debci implement the "test package foo/unstable in testing"?
> I'm asking the question because I wonder how it copes with packages
> having gained a dependency on a version of another package that has not
> yet migrated either.

You answer a bit of this yourself later, will answer there.

> Your documentation has this somewhat related paragraph:
> +There are cases where it is required to have multiple packages migrate together
> +to have the test cases pass, e.g. when there was a bug in a regressing test
> +case of a reverse dependency and that got fixed. In that case the test cases
> +need to be triggered with both packages from the source suite in the target
> +suite (again, how this is done depends on the setup).
> 
> Can you let us know how this will be done for the official Debian setup?

I haven't figured out how to do this automatically yet (and neither has
Ubuntu, so apparently this isn't a big issue). In Debian for now this is
less of an issue as we start with aging changes. So stuff migrates in
the end anyways. For now, people with the right knowledge (= the britney
key for ci.d.n) can trigger the right combination manually.

> Another thing that I wonder is whether the autopkgtest policy check will
> be cleared for a given version as soon as it has run or whether it might
> have to run multiple times if some of its dependencies migrate in testing
> before it migrated? (i.e. the test environment evolved during the aging
> period)

Once cleared, it remains cleared, the opposite isn't true. The current
design isn't smart enough to invalidate passes again.

>> Shadow britney
>> ==============
>> I have started a shadow cron job of britney on respighi to see how
>> everything is behaving. I'll start filing bugs on regressions such that
>> the current code already has value, but the drawback of a shadow run is
>> that the information is very poorly discover-able for interested
>> parties. Also, I expect people to start noticing that ci.d.n is
>> generating more results, I think it would be great if we can explain
>> this in one go after britney has the autopkgtest code deployed.
> 
> Looking for such results on ci.debian.net, I found this:
> https://ci.debian.net/packages/g/gnome-photos/testing/amd64/
> 
> Which gives some hints to my former question. It looks like
> autopkgtest is receiving some supplementary option like this
> one:
>     --add-apt-release=unstable --pin-packages=unstable=src:gnome-photos
> (seen in https://ci.debian.net/data/autopkgtest/testing/amd64/g/gnome-photos/98854/log.gz)

Yes, autopktest recently gained functionality to add multiple suites to
the available pool and to use apt pinning to pull in the relevant
packages from the "unstable" suite. If a package has requirements that
are only fulfilled by unstable, they should be pulled in from unstable.
(There may be a bug discovered in Ubuntu recently about autopkgtest
using the wrong resolver to do this as intended, but that is the idea).

> For persons interested in seeing excuses data generated by
> this shadow britney, it seems to be in
> respighi.debian.org:/home/elbrus/data-b2/output/

Yup.

> BTW, the excuses.yaml does contain weird lines with "null":
> 
> (in appstream entry)
>     autopkgtest:
>       appstream:
>         amd64:
>         - RUNNING
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/a/appstream/testing/amd64
>         - null
>         - null

These lines will be filled for other cases, e.g. when the test has
completed. The first one is only relevant for Ubuntu when they are
testing ppa's, the second one will contain a retry URL after completion.

> Also I see the age-requirement bumped to 15 but it would be nice to
> also have somewhere the details of why we got 15 instead of the usual
> value.

Can you explain what you find unclear in the excuses text (check the
html version please)? Do you have a proposal to make it better?
Currently it says: "Required age increased by 10 days because of
autopkgtest" or "Required age decreased by 3 days because of autopkgtest"

Paul

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20180406/b77efd00/attachment.sig>