Bug#833407: Please put adt-virt-* binaries back onto PATH

[Martin Pitt]

Hello Ian,

Ian Jackson [2016-08-12 12:06 +0100]:
> But, now, on this question, I am upset.  I designed what I thought was
> a good interface, which would be flexible and extensible enough to
> grow to solve a general problem.

It seems to me the core of the misunderstanding is that apparently you
intended/want the adt-virt-* backends to be a first-class public
interface, whereas I have always considered them internal to
autopkgtest. These backends have never been described, packaged, or
advertised as something independent of autopkgtest, there has never
been a separate project that implemented a new one, and up to today
(when I saw the bug report from Johannes about breaking sbuild) I
wasn't aware that something else even used them directly. Hence the
quick upload to put back the symlinks into /usr/bin/ as a stopgap --
apparently the "proper" fix takes some more discussion.

> My design approach has IMO been vindicated by the fact that there are
> now out-of-tree users of this API.  My design approach has also been
> vindicated by the support it is now receiving from that the author of
> that out-of-tree user.

Yes, agreed.

> I had hoped that when I explained my reasoning, it would become
> apparent that my wider intent hadn't been appreciated and that this
> was simply a mistake.

It certainly was a deliberate decision of me to take these out of
$PATH under the above perspective.(i. e. before I was aware of sbuild,
and that you intended to make the virt backends first-class external
and public interfaces). For sure it was my mistake to not check for
external users of adt-virt-*, as I have always thought of those as
internal private API. There are certainly no (relevant) reverse
dependencies of autopkgtest -- in particular, sbuild does not even
have a Suggests: for it, and all other hits in "apt-cache rdepends
autopkgtest" want the actual test tool, not the virt backends.

> > TBH, I don't consider the autopkgtest virt backends as a generic,
> > useful, and comfortable API for that -- it would need a lot of design
> > work, libraries etc. to become that.
> 
> I feel insulted.  You are dissing my API design.  And we already have
> an example - sbuild - where this was done spontaneously, demonstrating
> that my API is indeed useful.

I didn't mean to insult you, sorry if I did. "Not being comfortable to
use" and "not considered a public API" is far away from "dissing" or
claiming to be "not useful".

I never said that it wasn't useful, just that I *consider* a CLI as
not really that easy/robust to use -- I said it's a lot easier to use
a library. And if you use a CLI anyway, then using "schroot" or
"lxc-attach" directly is not much harder but much simpler to
understand than going through the adt-virt-* indirection.

> Furthermore, there is no reason why language-specific libraries
> couldn't be provided (if desirable) for the making the use of the
> language-neutral fork/exec interface more convenient.

Right. And we already have a Python binding, even though it hasn't had
a stable API -- it hadn't been a separate Python module for a long
time, and since its introduction the API changed several times.

> In contrast, your alternative design approach doesn't seem to have
> been thought through.  And you do not seem to recognise the general
> usefulness of a language-neutral, common API for solving this problem.

I didn't even propose an actual alternative design yet :-)

Maybe/apparently there is a general usefulness of an API that
abstracts away different virtualization backends (in the spirit of
libvirt). The adt-virt-* programs were just never
advertised/packaged/documented to be that, so please accept that I
*was* fairly surprised to see them being used externally. codesearch
has been broken for a while, so I'm not sure whether anything else
uses adt-virt-* right now.

> Are you suggesting that if and when someone writes a backend in a
> compiled language, they should
>  1. file a bug against autopkgtest to ask you to define the
>     right path or search strategy to use; and
>  2. file a bug against all users of adt virt servers to ask them
>     to search the new path ?
> That's obviously not going to be convenient.

*If* someone writes a new backend, I would much rather discuss the
design of that first indeed -- this needs more coordination than just
where to put the binary. I daresay that the chance of this actually
happening is very small anyway (it hasn't happened in more than 10
years), so this is still purely theoretical. I would like to hear a
good rationale for using a different language than the existing ones,
we would need to create conformance tests for the CLI API, and change
the packaging and documentation (it feels wrong and unclean to depend
on "autopkgtest" if you want some library for a virtualization
abstraction, which has nothing to do with what autopkgtest says on the
tin).

> It seemed to me, and it still seems to me, that the best approach for
> this is to think of the virt servers as special purpose command line
> tools.

I'm not completely opposed to putting them back; I just don't think
that this is the most important or relevant piece of that API.

> PATH is littered with tools which are usually invoked by some other
> program, and/or with formulaic names, and or with special calling
> conventions.  (fsck.*, <triplet>-ld, dh_auto_*, sg_*, git-remote-*,
> pkg-config, ....)

Yes, and that's IMHO wrong too. That other guy's clutter is not a
justification and argument for being untidy too :)

> The use of the adt-virt-* prefix avoids trampling all over the
> command namespace.  It seems to me that there is no downside to
> using the command namespace.

I still don't think these belong into $PATH -- these are *not*
commands that an user should run, it's an API that (maybe) *programs*
can run.

> These problems are entirely theoretical, even ridiculous.  There is no
> problem with cluttering tab completion because the virt servers are
> all named `adt-virt-*'.

adt<tab> does show all these things. Now, "adt" is a deprecated name
anyway, but I really do like that autopkg<tab> shows you the programs
that an user is supposed to call, but not the internal bits.

However, I respect that you disagree, and as I said my biggest concern
is not really their location; so moving those to autopkgtest-virt-XXX
into $PATH is okay for me, even though. This isn't worth having a big fight about.

That still leaves the documentation/packaging/API stability
tests/library bindings of course, but I'm happy to review
contributions there.

> >  than the slight inconvenience of calling these by full path for the
> > three people in the world who need to do that once a year.
> 
> I'm shocked to hear you suggest that passing absolute paths is the
> right answer.  It's not a question of "three people [doing] that once
> a year".  If your approach is adopted, these absolute paths will
> become embedded in other programs, which is extremely poor practice
> (and contrary to Debian policy).

Using absolute paths for internal API is common practice --
/usr/{share,lib}/<packagename> is full of examples.

As I said -- please read my replies from my (at least former)
perspective that adt-virt-* are *internal* project APIs.

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)