[buildd-tools-devel] Bug#609932: Bug#609932: Bug#609932: sbuild now runs apt-get update automatically while building

[Roger Leigh]

On Fri, Jan 14, 2011 at 01:54:46AM +0100, Cyril Brulebois wrote:
> Roger Leigh <rleigh at codelibre.net> (14/01/2011):
> > The latter should also be coupled with an upgrade to ensure that the
> > baseline state is up-to-date prior to any installations and
> > removals.  This is why the default is also to do an upgrade.
> 
> So by default I get a chroot which gets upgraded behind my back? Great!

Well, I do need to consider the pros and cons of both approaches.
Improving the quality of package builds by ensuring that by default,
people are building against a current package set is one of the pros.
Another is removing a failure mode in which build-dep removal/install
can break the chroot.

Automatically updating the chroot does have the potential to cause
problems if you want all builds to be against the same package set,
but given the frequency distributions such as unstable are updated,
you can't realistically not update for more than a few days before
things will start breaking as packages are removed etc.

So, we need to ask ourselves which use case is most common, most
important, most robust etc.  Some points to consider:

• Should every sbuild user be required to manually update their
  chroot before a build?
• Does automating this reduce the chance of misbuilds and upload of
  packages built against old dependencies?
• Is making sbuild resilient to certain failure modes by default
  more or less important than catering to certain use cases by
  default?
• Is it very common to build packages with sbuild prior to upload?
• Is it very common to deliberately not update the chroot every
  time to ensure consistent rebuilds of the same package?  Are such
  packages subsequently uploaded, or are they rebuilt after updating
  the chroot prior to upload?

As I'm sure you can see, the issue is not exactly black and white.
I made the change to the default after judging that the most common
use of sbuild was for building prior to upload, and that building
against the current archive would improve the quality of packages
uploaded to the archive and would also make sbuild more robust.

> > We don't do a distupgrade by default; that's probably not
> > sufficiently safe.
> 
> What would be the difference? I don't see any.

You won't get any additional installations/removals; there's been
a historical aversion to doing that automatically, though I don't
think that nowadays the chance of trashing the chroot is at all
likely (especially given that we now ensure the installation of
the toolchain in addition to build-deps).  I would prefer to do this
by default if there was consensus to do so; it would automate another
thing buildd admins and sbuild users need to do routinely.

> > With the use of cloned chroots, this is much less of an issue since
> > we just delete the chroot.  But for non-cloned chroots, this means
> > the chroot won't randomly break when apt-get can't cope, and most
> > sbuild users are using non-cloned chroots.
> 
> I don't remember having apt-get break my chroot. I built *some*
> packages though.

This isn't a common scenario.  However, it is a /possible/ scenario,
and the intention is to make sbuild more robust; rather than it
merely being possible, it's now completely impossible since this
failure mode is no longer possible.

Consider what happens if the chroot isn't completely up-to-date, and a
package dependencies have changed between what's installed in the
chroot, and what's available in the archive.  If the package is in
build-conflicts and we remove and reinstall it, that could cause
breakage.

Being up-to-date means that we can guarantee certain operations such
as install/remove or remove/install won't cause breakage.  If the
chroot is outdated, then those guarantees can't be met.

The apt and aptitude resolvers should also increase robustness here.
We now use apt-get rather than dpkg to remove and reinstall packages
for all resolvers, though the specific resolvers should probably be
performing this task eventually, and this also gives us more
resilience to such changes.

> > Note that buildd always asks sbuild to update itself on every run
> > already (--apt-update).
> 
> I don't care about buildd. I care about building packages in a chroot
> I control. This means that if I build package A at a given time, and
> some other host hits the approx caching proxy later, I can have a new
> package version showing in my chroot's cache, and I can't build
> package A anymore if I went offline in the meanwhile, while all
> packages were available in my cache.

OK, I can see this would cause some problems, but I'm not sure that
this should necessarily influence the choice of default sbuild
behaviour, given that the behaviour is entirely configurable.

> Not to mention extra differences which may appear due to newer
> revisions of packages. Lower pertinence of debdiff and the like.
> Looks like a pretty bad plan by default.

When you upload a package, it's going to be built against the current
versions in any case.  To what extent are package list changes
shown by debdiff going to be affected?  In most cases I would suspect
there would be zero differences.

> Also, = 0 seems to be the default, in sbuild.conf, which is also
> misleading.

Quite a number of the sbuild.conf defaults need re-synching with the
current defaults.  Ideally, I'd like to autogenerate the entire
file from the configuration class.  The sticking point is the
mapping between the internal name and the variable name e.g.

  APT_UPGRADE ←→ $apt_upgrade

If we could automatically insert the vars into the symbol table, we
could provide both the help text and variable name directly with the
defaults like so:

        'APT_UPGRADE'                           => {
            DEFAULT => 1,
	    SYMBOL => '$apt_upgrade',
	    DESC => 'APT upgrade.  1 to enable running "apt-get upgrade" at the start of each build, or 0 to disable.'
        },

This would make the configuration much more robust (currently, it needs
three separate bits to be altered: the definition as above, the symbol
definition, and the setting logic.  This would require some perl-fu to
directly alter the symbol table.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110116/c090f4b2/attachment.pgp>