[buildd-tools-devel] Bug#607228: sbuild pre-build/setup commands

[Wookey]

Package: sbuild
Severity: normal

I have a need of better functionality in this area too. Here is an
example use case and some comments on the current state of the code.

I will supply a patch soon too.

I want to be able to modfy the chroot setup with an extra repository
and dpkg-buildflags.conf files. These are necessary in order for one
standard chroot to support cross-builds for more than one host
architecture (armhf and arm64 in this case).

For obscure reasons
(https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1185896) the arm64
package repo cannot be present during the armhf builds, but in general
the ability to change the repos in use for a build is a common and
reasonable one.

Similarly arm64 needs a dpkg-buildflags.conf file to disable libssp
support until that support is written. Such a file must not be present
for the armhf builds.

So copying in these two files on chroot setup is a neat and simple way
to use one chroot for multiple architectures, and writing a simple
script to do this (which is passed the host architecture as a
parameter) is easy. However the current functionality in the
--pre-build-commands and --chroot-setup-commands options is not
sufficient.

--pre-build-commands is run outside the chroot as root
--chroot-setup-commands is run inside the chroot (as root in ubuntu, as
the build user in debian)

Nomenclature note:
Unless you look at the codebase it's hard to work out from the
documentation when these commands are run, or that some are run inside
and some outside. Also it seems reasonable from the names to think that
'pre-build-commands' are run just before the build phase, and that
'chroot-setup-commands' are run very early on to set-up the chroot.
This is the exact opposite of what actually happens (pre-build is run
right as the chroot is setup, and chroot-setup is run just before the
build phase). Unfortunately these names are exposed in configs, but I think
we should give serious thought to changing them to something users
might get right without looking it up. One wonders if the author got
confused when writing this code, but I guess they were thinking of
'before the whole sbuild job' with 'pre-build'.

Is it too intrusive to change the names?

At the very least much better documentation on this issue is needed.
I'll take a stab at this.

My example:

So I have a simple script "configchrootforarch" which if run inside the
chroot as "configchrootforarch <arch>" will add or remove files as
needed. thus setting --chroot-setup-commands="configchrootforarch
arm64" (or armhf) ought to DTRT, except that a) it's only run as root
on ubuntu and b) it's run _after_ the the apt-get update phase and
before the actual build. This is too late. We need the extra repo in
the update phase so that the arm64 packages area avilable to apt. 

--pre-build-commands is run at the correct time, but is run outside
the chroot and so far as I can see there is no way the script can know
the path to inside the chroot to fiddle with files in there. Am I
missing something here? The /etc/99builddsourceslist file
(http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=99builddsourceslist;att=1;bug=675349
) that roger mentions above seems to have access to lots of sbuild
variables like ${CHROOT_PATH}, but that doesn't seem to work for
commands run in the --pre-build-commands context. That looks like
scripts in /etc/schroot/setup.d. One could conceivably be used to do
this job, but it's hard to make it system wide in the case of adding
repos, and HOST_ARCH is not available (schroot has no
concept of host arches - that's a per-build thing), so I don't really
think that's the way to solve this.

It would also be nice if I could always call it as "configchrootforarch
%a" so the correct host architecture was set automatically.


Solutions:
There are various ways to fix my particular case:
1) Add %p (path to the chroot) substitution so that the called command
in --pre-build-commands can change things inside the chroot. 

2) Add a --pre-build-internal-commands to run at the same time (just
after chroot is set up) but inside the chroot.

3) Move --chroot-setup-commands to operate before the apt-get update
phase (and run as root).

4) Change --pre-build-commands to operate internally (is an external
command at this point useful to anyone? I can't think of anything
offhand, but I assume there are realistic uses)

5) and a little improvement of adding %A (build arch) and %a (host arch)
substitutions so that the build and host architectures can be pased to
scripts. 

Any feedback on which of these would be best would be welcome. 
Unless anyone disagrees I propose to do 2 and 5. I'll also prepare a
patch to change the names around as 2) will make the naming situation
worse!

-- System Information:
Debian Release: 6.0.7
  APT prefers oldstable
  APT policy: (500, 'oldstable')
Architecture: i386 (i686)

Kernel: Linux 2.6.32.33-kvm-i386-20111128-dirty (SMP w/1 CPU core)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash