[Docker-maint] [pkg-go] Guidance for packaging Docker for Debian

[Arnaud Rebillout]

On 03/15/2018 08:42 PM, Ian Campbell wrote:
> On Thu, 2018-03-15 at 12:52 +0000, Martín Ferrari wrote:
>> I sometimes have kept small dependencies vendored in for
>> convenience..
>> But keeping the whole containerd seems wrong to me.
> OTOH it _might_ not be totally unreasonable to apply that tactic to
> swarmkit which in essence has a single consumer (docker engine).

That's a good point. Here's the dependency tree that bothers me right now

docker-engine
  \_ golang-github-docker-swarmkit
      \_ golang-github-docker-libnetwork

The main problem here are the circular dependencies, since both swarmkit 
and libnetwork import bits from docker-engine. To avoid the circular 
dep, I vendor docker-engine within swarmkit and libnetwork. But it 
doesn't work, because I end up with duplicated bits that are not 
compatible, as described in 
<https://github.com/Masterminds/glide/issues/73>.

To be more specific, both import 'pkg/plugingetter', and the build fails 
with this error:

have Get(string, string, int) 
("github.com/docker/swarmkit/vendor/github.com/docker/docker/pkg/plugingetter".CompatPlugin, 
error)
want Get(string, string, int) 
("github.com/docker/libnetwork/vendor/github.com/docker/docker/pkg/plugingetter".CompatPlugin, 
error)

Actually, this 'pkg' directory is the main cause of circular 
dependencies, and hence it's a big pain point. From the README that you 
can find in moby/pkg:

"pkg/ is a collection of utility packages used by the Docker project 
without being specific to its internals."

These utilities are useful outside of docker, and that's why every other 
docker projects include docker-engine and fish into the pkg directory. 
Hence the circular dependencies.

If we could somehow package this directory as a library, separately from 
docker, then we would have much less  circular dependencies. But since 
this directory is definitely not something stable, that will also bring 
us deeper in an "unresolvable dependencies hell".

So how to solve that ?

One easy solution would be to stop packaging swarmkit and libnetwork, 
and just vendor it in docker-engine. Of course, that makes sense only if 
there's no other consumers for these libraries.

The other solution (for me) is to learn more about Go and the way they 
handle this problem of duplicated vendored bits, and then transpose this 
solution somewhere in the debian packaging files.

It's also possible that I just do it something wrong somewhere, and I 
didn't figure it out yet.

>> In an ideal world, we should try to convince the docker people to use
>> stable APIs (that means using only released non-alpha versions!),
> FWIW the engine uses a non-alpha version in recent releases. It seems
> to be swarmkit (another dep of engine) which is lagging and using the
> alpha version (kind of interesting that that code seems to be ok when
> vendored into moby but apparantly not when standalone, I suppose they
> use different subsets of the API in different ways).

Very interesting point indeed. For more details, here are the versions 
of containerd vendored by docker:

- docker-engine (ie moby): 3fa104f (after v1.0.0)
- docker-swarmkit: 29a4dd7 (somewhere between v1.0.0 alpha3 and alpha4)

Having both in sync would help for sure.

However after giving it a closer look, I think I could patch swarmkit to 
build against containerd v1.0.0. Maybe it's not so hard.

> I think now that containerd v1.0.0 is out and given the stated API
> guarentees containerd is making[0] the problem will be less bad in the
> future at least wrt this particular set of components since there
> ougtn't to be build breakage from using a "too new" version of
> container within a given major release series.

I hope you're right :)

Regards,

   Arnaud