[buildd-tools-devel] re buildd's resolver and package's build deps

Roger Leigh rleigh at codelibre.net
Mon Feb 28 19:12:00 UTC 2011


On Mon, Feb 28, 2011 at 03:36:47PM +0100, Wouter Verhelst wrote:
> On Tue, Feb 22, 2011 at 05:08:18PM +0000, Roger Leigh wrote:
> > On Mon, Feb 21, 2011 at 07:42:32PM -0600, Raphael Geissert wrote:
> > > I disagree here.
> > > Alternatives in build-* relationships *are* mentioned by policy. In fact, 
> > > there's even an example in section 7.1.
> > 
> > This is correct.  I was thinking about drafting a patch for Policy
> > about this.  Current Policy defines the allowed syntax for
> > Build-Depends.  It does not however, make any mention of existing
> > conventions and best practices, which I feel should be addressed.
> > 
> > The current "internal" build dependency resolver does not make
> > any use of the alternative dependencies.  It always picks the first.
> 
> It was my understanding that there are some corner cases in which it
> would select the second:
> 
> - where one build-dependency (indirectly) depends on a secondly-listed
>   alternative, it could install the second.
> - sbuild will not worry about installing the first alternative if the
>   second has already been installed.
> 
> but I might be mistaken about one or both of the above.

I'm not totally sure either, to be honest.  The internal resolver code
is horrible.  It looks like it still has relics of manual source-deps
entangled in there as well (but I don't dare to touch it since it will
probably break horribly if I delete it).  The alternatives processing
is in Sbuild::InternalResolver::parse_one_srcdep and
filter_dependencies, and it looks like it could well be OK with second
alternatives if installed.

> > · we always have consistent builds (because there's only a single
> >   solution)
> > · we have recommended against using alternative build dependencies
> >   since they were introduced; this was partly because sbuild didn't
> >   support them, but mainly because we want complete consistency
> 
> I've never been aware of such a recommendation.

It's been the general advice we've been giving to maintainers who were
using alternatives and then seeing FTBFS issues due to sbuild not
satisfying those dependencies.  From the sbuild side, the only use of
alternatives we see normally is hideously broken stuff.

> At any rate, if a maintainer would wish to support backports easily
> (ever more relevant now that it's on debian.org systems), having
> alternative build-dependencies would make perfect sense; in those cases,
> sbuild would need to pick one set of dependencies for unstable, and the
> second set for backports.

Agreed.  Note that we now support strict 'first-only' alternatives
handling with the 'apt' and 'aptitude' resolvers.  See the notes for
0.60.0 and 0.60.1 pertaining to resolvers here:

http://git.debian.org/?p=buildd-tools/sbuild.git;a=blob;f=NEWS;h=f4576f0e3b3c0fcd660000abcd93898697246b9a;hb=HEAD

We have made the alternative resolving configurable, so that you can
use strict 'first only' or all alternatives.  It defaults to
'first only', which will give resolution almost entirely like the
internal resolver.

> [...]
> > > There's also no stated guarantee *anywhere* (including release policy) that 
> > > the package's build deps should be consistent, much less the result.
> > 
> > I agree that the documentation is sorely lacking in this regard.
> > It is, however, an  unofficial and unwritten policy.  The need for
> > this is fairly self-explanatory: we don't want builds to vary.
> > Taking one of php5's dependencies as an example:
> > 
> >   libdb-dev (>= 4.7) | libdb4.8-dev | libdb4.6-dev
> > 
> > This dependency permits building against no less than *three* different
> > Berkeley DB versions.  Given that these versions are typically
> > incompatible, imagine if a new upload caused a version change.  It
> > could break all existing databases when the user upgrades and they are
> > no longer readable.  If could even be a downgrade.  The same applies
> > to any other libraries.
> 
> True, but the right solution for this problem is not to remove support
> for alternative build-dependencies; instead, policy should make it clear
> that when a package is built in unstable, no differences in alternatives
> should cause a change in functionality or support for file formats.

Please see #614807: debian-policy: Please document autobuilder-imposed
build-dependency alternative restrictions, which is the current
proposal for documenting the existing behaviour (and it does preserve
the ability for maintainers to use alternatives).  Comments would be
appreciated.

> > I would recommend keeping backports on a separate VCS branch rather than
> > having a single unified package.
> 
> I vehemently disagree with this recommendation. It works now, properly;
> what you suggest requires merging branches every time you wish to update
> the package. That introduces an extra maintenance burden for, IMHO, no
> good reason.

I've been doing this on e.g. a 'lenny-backports' branch, and it's been
a model of simplicity.  I just 'git merge <release-tag>' and it's done
except for updating the changelog (and fixing up the rare conflict).

http://git.debian.org/?p=buildd-tools/schroot.git;a=shortlog;h=refs/heads/schroot-lenny-backport

Certainly more reliable and convenient than doing things by hand.

> Build-dependencies were originally created for the buildd hosts, and
> sbuild is still the main user of build-deps; so if something is
> happening which would break sbuild, then sure, that means we should
> change that.
> 
> But it's important to remember that build deps aren't *just* used by
> sbuild. It is not unsensible to want to build a package on your local
> system; whenever that happens, having just a strict set available of
> packages that can be used to build the package is cumbersome.
> 
> > The need for concrete|virtual is a fundamental deficiency in our
> > package management that's been unaddressed for years (see old -devel
> > discussions).
> 
> Pointer?

The one I could find:
http://lists.debian.org/debian-devel/2006/08/msg01281.html

An ancient proposed solution (probably suboptimal):
http://people.debian.org/~rleigh/virtual-policy_1.dsc

AJ did say back then (though I can't find the mail, sorry), that he was
working on a solution, but I don't know what happened to that.

The main issue here is that every package with a "concrete|virtual"
alternative dep is specifying its own idea of what the ideal
concrete package should be.  That is, each package is dictating what
should be a system-wide policy, and this leads to a lack of uniformity
because there's not necessarily any coordination, and different
maintainers have different preferences.

This was a pain when we changed the default inetd--every package
required updating.  For others, e.g. mail-transport-agent, it's even
more painful (I thought an mta-default was proposed, similar to
virtual-policy above, but can't see it in use yet).  If we had a
centralised list (apt could use them during resolving directly), or
a set of dummy default packages (as virtual-policy), a package could
depend upon just the virtual package, and this would allow the default
to be changed in a single place, pain free.  This would be useful for
- easy customisation of system defaults ("I want postfix as my mta")
- easy customisation of derivative distributions
- easier transitions if there's just one thing to update, then binNMU.

With apt support, we could just ship this in e.g. base-files.  It could
be a simple virtual:concrete mapping:

mail-transport-agent: postfix
inet-superserver: openbsd-inetd
etc.

> [...]
> > Java and DocBook are the two worst offenders WRT build-deps.
> 
> This statement makes no sense at all.
> 
> Both are a clear case of software where there's a spec of what output a
> given input should provide. As such, whatever particular implementation
> you install should not matter; the output will be the same.

It /should/ be the same.  In practice, does this always happen for
sure?  The main issue with these two is the inconsistent use of
alternatives, not just in the build-depends, but in the indirect
dependencies, which leads to an unpredictable outcome which can
vary depending upon dependency ordering.  This is directly related
to the lack of system-wide virtual defaults as discussed above,
leading to each individial package implementing their own, conflicting,
defaults.

> [...]
> > One thing we could do is get sbuild to strip all non-arch-specific
> > alternatives from the Build-Dependsi, so it would never use
> > alternatives, ever.  This is for the "apt" and "aptitude" resolvers,
> > not "internal".
> 
> This could make sense.
> 
> While the above arguments all make sense wrt building locally, there is
> no reason to modify the current behaviour whereby sbuild will only look
> at the first dependency (unless something else has already been
> installed). After all, I do agree that predictability on buildd hosts is
> a good thing...

As I touched on above, this is the way we have gone.  It's configurable,
so it can be turned off for e.g. experimental, backports, where the
additional dependencies have some value.  Because this is actually
properly enforced, unlike for internal (we actually strip out the
unwanted deps), it's actually slightly stricter than internal (though
in practice in the current archive the first dependency is always
satisfiable, so this won't have any impact at present).


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110228/a407d511/attachment-0001.pgp>


More information about the Buildd-tools-devel mailing list