Redesigning the autopkgtest controller/workers CI system for Ubuntu and Debian -- call for spec review

[Martin Pitt]

Hey Evan,

Evan Dandrea [2014-04-03 11:53 +0100]:
> Maintaining a mapping of available hardware is a lot of manual
> operational work and doesn't scale particularly well. What happens
> when we have hundreds or thousands of {cloud instances, phones,
> servers, desktops, toaster ovens}?

We don't want to enumerate the particular workers where this runs on,
just the type of environments we want to run the test on. E. g. "any
machine with an NVidia card and a graphical user session".

> What happens when work is submitted before this mapping is in place?
> Does it fail or sit waiting for the mapping to become available?

It would just be run on any platform, as any other test. E. g. if we
upload xorg or Mir without such a mapping, they'd just run in a
minimal environment and just once. With the mapping we would then run
it on three or four different platforms.

This assumes that there are cases where we want to run tests on more
than one default/minimal environment, but not all such tests on all
platforms (we can't afford that, and it would be a giant waste of
resources).

This tag based approach might be too fine-grained, of course. An
alternative would be to just say "we want tests of these packages to
run on *all* available platforms. That might run some tests on some
platforms which we aren't interested in, but that's not such a big
problem I think. Then the mapping reduces to a set of package names,
and we can even include that information in debian/tests/control
("Features: all-platforms" perhaps).

> Perhaps I'm misreading, but I thought we agreed on the first call that
> we need a means of defining hardware constraints in DEP-8. In this
> model, if I try to run a DEP-8 test on a machine that does not match
> those constraints it fails loudly.

We already have testbed constraints in the sense of "this test needs a
container" or "this test needs a full VM or real iron". This is an
inherent property of the test, and thus a restriction.

But "we want to run tests of Xorg/Mir on x86/nvidia, x86/AMD, and a
Google Nexus 4" is not a restriction or inherent property of a test --
you can run it anywhere. It's an expression of where we want this test
to run. So if you think that's too complex, we can just reduce this to
a binary "on any platform" vs. "on all available platforms" decision.

Does that sound better?

Thanks,

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)