[Docker-maint] [pkg-go] Guidance for packaging Docker for Debian

[Ian Campbell]

On Fri, 2018-03-16 at 08:59 +0700, Arnaud Rebillout wrote:
> On 03/15/2018 08:42 PM, Ian Campbell wrote:
> > On Thu, 2018-03-15 at 12:52 +0000, Martín Ferrari wrote:
> > > I sometimes have kept small dependencies vendored in for
> > > convenience..
> > > But keeping the whole containerd seems wrong to me.
> > 
> > OTOH it _might_ not be totally unreasonable to apply that tactic to
> > swarmkit which in essence has a single consumer (docker engine).
> 
> That's a good point. Here's the dependency tree that bothers me right now
> 
> docker-engine
>   \_ golang-github-docker-swarmkit
>       \_ golang-github-docker-libnetwork
> 
> The main problem here are the circular dependencies, since both swarmkit 
> and libnetwork import bits from docker-engine. To avoid the circular 
> dep, I vendor docker-engine within swarmkit and libnetwork. But it 
> doesn't work, because I end up with duplicated bits that are not 
> compatible, as described in 
> <https://github.com/Masterminds/glide/issues/73>.
> 
> To be more specific, both import 'pkg/plugingetter', and the build fails 
> with this error:
> 
> have Get(string, string, int) 
> ("github.com/docker/swarmkit/vendor/github.com/docker/docker/pkg/plugingetter".CompatPlugin, 
> error)
> want Get(string, string, int) 
> ("github.com/docker/libnetwork/vendor/github.com/docker/docker/pkg/plugingetter".CompatPlugin, 
> error)
> 
> Actually, this 'pkg' directory is the main cause of circular 
> dependencies, and hence it's a big pain point. From the README that you 
> can find in moby/pkg:
> 
> "pkg/ is a collection of utility packages used by the Docker project 
> without being specific to its internals."
> 
> These utilities are useful outside of docker, and that's why every other 
> docker projects include docker-engine and fish into the pkg directory. 
> Hence the circular dependencies.

There was a plan at some point to remove pkg by splitting the stuff
which is truly useful into their own packages and move the rest into
the (often singular) callers or into the main engine code base etc.

I think plugingetter would be in the last category.

One or two bits did get moved/folded but sadly I think that effort has
stalled.

> One easy solution would be to stop packaging swarmkit and libnetwork, 
> and just vendor it in docker-engine. Of course, that makes sense only if 
> there's no other consumers for these libraries.

I'm not aware of any other consumers right now and I think leaving
those two (swarmkit and libnetwork) vendored at the versions given in
moby/moby/vendor.conf unless/until a user shows up would be fine (and
my recomendation, FWIW).

> > > In an ideal world, we should try to convince the docker people to use
> > > stable APIs (that means using only released non-alpha versions!),
> > 
> > FWIW the engine uses a non-alpha version in recent releases. It seems
> > to be swarmkit (another dep of engine) which is lagging and using the
> > alpha version (kind of interesting that that code seems to be ok when
> > vendored into moby but apparantly not when standalone, I suppose they
> > use different subsets of the API in different ways).
> 
> Very interesting point indeed. For more details, here are the versions 
> of containerd vendored by docker:
> 
> - docker-engine (ie moby): 3fa104f (after v1.0.0)
> - docker-swarmkit: 29a4dd7 (somewhere between v1.0.0 alpha3 and alpha4)
> 
> Having both in sync would help for sure.

I prodded some folks yesteday a lo:
    https://github.com/docker/swarmkit/pull/2560

(failing CI right now, but it'll get there)

Ian.