[buildd-tools-devel] Bug#811202: sbuild: order some code differently

[Johannes Schauer]

Control: tag -1 + patch

Hi,

On Sat, 16 Jan 2016 20:04:37 +0100 Johannes Schauer <josch at debian.org> wrote:
>  1. The EXTRA_REPOSITORIES are added to $dummy_archive_list_file in
>     setup_apt_archive() in lib/Sbuild/ResolverBase.pm. But
>     setup_apt_archive() is only called from install_deps() in the
>     lib/Sbuild/*Resolver.pm files. This is too late because this means
>     that the initial update/upgrade step from run_chroot_update() in
>     lib/Sbuild/Build.pm will ignore any extra repositories passed on the
>     command line. This is no problem if the extra repository is only
>     used to install additional packages but it is a problem if the extra
>     repository contains packages with higher versions than the installed
>     essential or build-essential packages. Also see bug #792037

this has been taken care of and is in the sbuild git.

>  2. The function run_chroot_update() in lib/Sbuild/Build.pm should not
>     be run multiple times. Especially not now that there is an external
>     command hook for this step. Running update and upgrade once should
>     be sufficient. Instead though, run_chroot_update() is called again
>     if a new architecture was added in check_architectures(). Thus, the
>     check whether foreign architectures have to be added should be moved
>     to before the original call to run_chroot_update() in
>     run_chroot_session_locked().

This is a tricky one because it creates a chicken-and-egg situation when sbuild
is instructed to download the source package itself. In that situation, the
source package's metadata is required *before* the chroot is updated, so that
extra architectures can be added to the chroot. But sbuild should only download
the source package (including its metadata) *after* the chroot is updated to
pick up the latest source package version. Here is the summery of the attached
patch that fixes this issue but introduces possible problems when sbuild
downloads the source package for the user:

    lib/Sbuild/Build.pm: Avoid doing an sbuild update twice (closes: #811202)
    
    Split the check_architectures() function into two parts. One that checks
    the source package for build dependencies with explicit architecture
    qualifiers and one that checks if the architectures from the source
    package can be produced. The former has to be run *before* the chroot is
    updated, so that any additionally required architectures can be added by
    the update. The latter has to be done *after* the chroot has been
    updated and build-essential is installed because it requires the
    Dpkg::Arch module inside the chroot. This splitting and reordering is to
    avoid calling the chroot update more than once because we want to avoid
    running external commands more than once as we don't want to require
    them to be idempotent.
    
    This would all be no problem if sbuild would not also offer the option
    to download source packages by itself. Ideally, we only want to download
    the source packages *after* the chroot has been updated and after extra
    repositories were added to the internal apt sources. Unfortunately, we
    need the source package's metadata to analyze its build dependencies for
    those with explicit architecture qualification *before* the chroot has
    been updated. Using the output of "apt-cache showsrc" is only a crutch
    because the information of that command might be outdated.
    
    It is a chicken-and-egg problem because downloading the source package
    needs an up-to-date chroot, but updating the chroot needs information
    about the architecture qualifiers which in turn needs up-to-date
    metadata.
    
    This problem can only occur when the user instructs sbuild to download
    the source package itself and the source package saw a recent upload
    such that the metadata the chroot knows about before and after the
    update differs.
    
    In summary, this commit makes sbuild behave better in situations where
    it is to build a given .dsc and external command hooks are to be run. It
    makes it behave worse when it is instructed to download the source
    package itself.
    
    A possible solution to the chicken-and-egg problem would be to add a
    separate step for updating the source package lists only, supply a way
    to add extra source package lists and extra external command hooks for
    exactly that step. I'm not sure the complexity is worth it though.

Anybody knows an elegant solution?

Patch attached.

Thanks!

cheers, josch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-lib-Sbuild-Build.pm-Avoid-doing-an-sbuild-update-twi.patch
Type: text/x-diff
Size: 14448 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20160704/5ab9865b/attachment-0001.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20160704/5ab9865b/attachment-0001.sig>