debci <-> britney communications

[Niels Thykier]

Paul Gevers:
> Hi Niels,
> 

Hi :)

Once again, thanks for volunteering for this. :)

> Today at Debconf, I discussed with Antonio the changes that would need
> to go into debci to discuss with britney. I hope to follow-up later with
> slightly more details, but the following I already want to align with
> you.

Thanks.

For the record, I have a very little insight to the complexity,
requirements and architecture of the ci.d.n side of things.  If I
suggest something that seems "crazy" to you, then I am probably not
aware of the things that you are.

> If I remember well, you (the RT) want brintney1 to do the
> communication and ideally only fetch one file with the results.

Generally, we do one file per suite.  Though for urgencies, we have more
files than that (at a non-trivial cost because we use a dumb http sync).

However, the autopkgtest case is somewhat different than existing cases
as Britney will be influencing the tests (and not consuming
off-the-shelf test results).

> Discussing this we thought to provide the results since the last
> download of results. Do you think that this would work or do you
> consider this too fragile? If so, do you have an better proposal? I
> don't think it makes sense to put all results ever into the file.
> 

My primary concern will be synchronization overhead, results merging
(either side) and "re-synchronization".  From experience, we will
eventually need to re-fetch something because the original fetch failed,
got corrupted, or a test request got lost.

>From my PoV, we can solve the resynchronization in multiple ways:

 * Structure state such that Britney can tell if a test is pending on
   ci.d.n side.  If we lose a result, Britney will tell it won't appear
   in the future and simply reschedule it.
   - This has the advantage of self-healing regardless of whether the
     "request for test" or the "test result" was lost.

 * A replayable "give me the last results since $TS" (within reasonable
   limits)

 * ...

As for the data itself; how large will these files be?  If they are
modest, then perhaps an approach like the urgencies might be decent
(admittedly, I could use more accumulation on the urgencies files).
  The approach these is basically just adding files every now and then.
Britney1 fetches the files exported and deletes them again after X days.
 The exporting side will delete them after Y days (with Y < X as I
recall).  Finally, Britney2 is provided a directory of these files and
merge the files on load.

> Regarding communication from britney2 to debci, is it ok for britney2 to
> do http queries per requested test, or should this also be passed on to
> britney1?

Ideally, no for several reasons:

 * On occasion, people do ad-hoc test runs on release.d.o to see if a
   hint works or what will migrate, etc.  The results are not committed,
   but if we start changing state outside britney, it will have
   undesirable results.

 * I like to avoid having britney2 deal with remote services as it makes
   britney2 simplier and testing can be done trivially without external
   services/processes running.
   - Notably, doing this in Britney2 will eventually require us to
     configure a TLS trust store for connections because DSA managed
     servers have a stricter trust store than most other servers.

 * Ideally britney would run even if ci.d.n is down and still update
   results for other packages.  She might not be able to migrate a lot
   but at least the excuses could be updated.
   - Here, I mean "she would run without me having to disable the
     autopkgtests integration" temporarily.  We "could" fix this by
     "proper error handling" except I know we will never
     "catch^Whandle them all".

Please note that I am open to britney2 having stand-alone utilities that
talk to remove services to assist with this.  I have no intention of
asking you to write complex REST clients in shell (nor any intentions of
maintaining that if I can avoid it).

> If the latter, aren't we delaying communication from britney
> to debci unnecessarily? Any comment on this?
> 
> Paul
> 

What is the concern here?  At first glance a handful of bulk calls at
the end of a 5-10 minute run scheduling all tests seems preferable to me
over a lot of small immediate requests during the run.
  Especially, if we combine it with a synchronization mechanism where
Britney can tell if something has been scheduled or not (so she can
reschedule lost tests).

Related, we are doing hourly runs of Britney now to update excuses
(still only migrations 4 times a day though).  Ideally, the excuses-only
runs would schedule tests to keep the latency down.  If that is not
often enough, we can look into adding more "excuses-only" runs by
playing "crontab kalah" with the release.d.o crontab.

Thanks,
~Niels