[buildd-tools-devel] Bug#610689: Bug#610689: sbuild: cross build support

[Neil Williams]

On Sat, 26 Mar 2011 20:06:21 +0000
Roger Leigh <rleigh at codelibre.net> wrote:

> On Fri, Mar 25, 2011 at 05:06:02PM +0000, Hector Oron wrote:
> > 2011/3/10 Roger Leigh <rleigh at codelibre.net>:
> > > For the cross-packages, you can add these to the core build
> > > dependencies list, so they will be installed along with build-essential
> > > etc.  Or (better) you can add a separate XAPT_DEPENDS variable which can
> > > be appended to CORE_DEPENDS in Build.pm prior to installing the core
> > > dependencies.  Does xapt not bring in these packages directly, or do
> > > they always need installing by hand?
> > 
> > xapt downloads the foreign architecture packages using APT with -o
> > APT::Architecture=$HOST_ARCH into a directory controlled by xapt
> > (/var/lib/xapt/, irrc), then throws dpkg-cross into that directory to
> > convert those packages and installs them.
> 
> This is the part that I think can be significantly improved.

Not until we have Multi-Arch-Cross. Indeed, there's no point making
special changes to sbuild ahead of Multi-Arch as the entire
cross-dependency resolution mechanism will be completely thrown away at
that stage.

What you're trying to create in this bug report is a Multi-Arch-Cross
resolver for sbuild and then hacking it to handle dpkg-cross. That is
the wrong way around. Use the hacks which already work with the
dpkg-cross hacks and then implement a *clean* Multi-Arch-Cross resolver
when that becomes possible.

dpkg-cross is fundamentally broken and is completely alien to
Multi-Arch. Anything based on dpkg-cross will need to be completely
re-written for Multi-Arch-Cross, so use the tools which exist (and
which will disappear alongside dpkg-cross) to retain as much isolation
as possible between the sbuild code and the broken dpkg-cross methods.

> sbuild creates a local APT archive to install dependency packages from.
> We can use the same mechanism to allow the dpkg-cross packages to be
> installed.  It could even be used to cache generated packages between
> builds.  We can quite easily refactor the code to allow multiple
> archives (if required).

The problems in this area are:

0: The setup_apt_archive code uses the Dpkg bindings to resolve the
dependencies but these are inherently native (despite taking an arch,
it is difficult to get this right when there are alternatives and
architecture-specific dependencies). These have been re-written in
embuilddeps but all that code will be completely pointless once
Multi-Arch-Cross arrives. There is no point implementing yet another
broken version of it in an XaptResolver for sbuild.

1: embuilddeps has tried to solve these problems but has a
fundamentally different approach to actually installing the packages.
Sbuild::AptResolver and Sbuild::AptitudeResolver use the dummy-package
method to actually do the work of resolving the dependencies within the
specified constraints. embuilddeps resolves the constraints directly.
e.g. constraints of foo (>= 1.2.3) [armel] | bar [!linux-any] | baz
[i386| amd64] are particularly hard to resolve correctly when running
on i386 to cross-build for armel.

2: Cross-dependency resolution requires re-doing the dpkg deps_parse
calculation with an empty dpkg status file and then chasing the
dependencies all the way down because -cross packages do not exist in
the archives. This is why I'm looking at Multi-Arch being the preferred
solution here.

3: dummy-package methods won't work for cross because we must resolve
all the constraints ahead of passing a list of packages to apt so that
these can be downloaded *ahead* of being installed, then converted
using dpkg-cross and installed as packagename-$arch-cross packages.
libfoo-dev_1.2.3_armel.deb becomes libfoo-dev-armel-cross_1.2.3_all.deb

4: The converted packages do not (indeed cannot) exist in any
network-accessible archive - these are all generated on-the-fly until
Multi-Arch makes the whole conversion avoidable.

5: apt and aptitude consistently treat -$arch-cross packages as "local
or obsolete" and consistently get removal parsing wrong when upgrading
existing packages. This means that the list of -$arch-cross packages to
be removed needs to be carefully managed, especially if the cross
toolchain is not to be removed upon every build.

Much easier is for xapt|embuilddeps to put the downloaded and the
converted packages into a directory which sbuild can manage, or for
sbuild to become aware of the current defaults which
are /var/lib/xapt/output for -cross packages and /var/lib/xapt/archives
for the foreign architecture originals.

This will also solve problems of package removal after the build,
albeit that some extra processing is required to skip certain
-cross packages which would otherwise cause the removal of the
cross-building toolchain. (libc6-dev-$arch-cross is a frequent problem
here.) This list of packages does need to be kept updated to cope with
changes in gcc and in eglibc - e.g. the Squeeze chroots will have a
libc-bin-$ARCH-cross package which doesn't exist in Lenny chroots.

Whilst it is certainly a good idea to try and re-use the existing
sbuild support for keeping the downloaded packages available, there are
significant problems with doing this the way that sbuild currently does
it. It would need to be a post-process update, reading in the list from
the xapt directory. (xapt can be told to keep the archives around, it
usually deletes them once all the -cross packages are installed and
this option has also been added to embuilddeps, so the new -k option
passes that down to xapt which keeps the /var/lib/xapt/ tree in place
for subsequent handling.)

I did think about putting that list into a file but the list is simply
the list of .debs which were downloaded into a clean directory - one
directory for the (typically) armel packages and one for the
-armel-cross packages. A simple opendir(); readdir and pattern match
on /^(.*)_.*_($arch|all)\.deb$/ or similar would get the list of package
names and if /var/lib/xapt/* is removed by whatever does the
subsequent package removal handling, then this list will be guaranteed
to be the complete and exact list of all dependencies downloaded as
$host architecture and installed as -$host-cross packages for that
specific build.

Overall, sbuild cross-build support ahead of Multi-Arch-Cross is best
done by leaving as much of the work as possible to tools outside sbuild
- tools which themselves will disappear once Multi-Arch-Cross is in
place. Only at that stage is it worth looking at how the internals of
sbuild can enfold the needs of the Multi-Arch-Cross world which,
currently, is largely opaque.

> > > Well, when we install the build dependencies the package isn't yet
> > > unpacked.  The unpacking is done in the build() function.  But there's
> > > no reason we can't reorder it if we need to.  The unpacking could
> > > be split out of build() into a separate unpack_source() function,
> > > which could be run earlier if xapt/embuilddeps is in use.
> > >
> > > What information does embuilddeps require from the unpacked source
> > > package?  Is this information also available in the .dsc? (We do
> > > have the .dsc information directly to hand).
> > 
> > It is possible to use .dsc information, but embuilddeps does not know
> > how (yet). :-)

It will - in the next release. It's as easy to parse debian/control as
the .dsc. I've got a change in SVN which adds a --dsc option to
embuilddeps.

So sbuild would actually use:
embuilddeps -a $host_arch -k --dsc $dsc_file

No use of -n, no parsing of the output - let embuilddeps call xapt to
download the packages and dpkg-cross to convert and install them, then
parse the list of downloaded and converted packages directly from the
directory listing, handle the removal of the $host-cross converted
packages when appropriate and then tidy up /var/lib/xapt/*

Once we have Multi-Arch-Cross, the entire process folds back into a
standard sbuild/apt|aptitude problem because M-A-C will *have to* allow
dpkg to fix the problems of foreign architecture dependency resolution.
(And it will do a better job of it than either sbuild or embuilddeps
could do themselves.)

> OK.  For the purpose of actually implementing this, I would like to
> have a description of exactly which fields we need for cross building
> from which files in the source or DSC, and what they do and why.  Is
> this all documented somewhere?

There was an idea to separate the "necessary" build depends from the
"ancillary" build depends - the tools like debhelper and automake which
need to be installed but also need to be executable, so need to be the
build arch, not host. This was the xcontrol debacle and it's dead.

Instead, embuilddeps parses the entirety of Build-Depends and
Build-Depends-Indep as well as Build-Depends-Conflicts and resolves
them all for the host architecture. This results in tools like
debhelper being converted by dpkg-cross into empty
debhelper-armel-cross packages but that can't be helped. (We tried to
do that with apt-cross and it resulted in an unfixable RC bug and the
removal of apt-cross.)

It's another reason why this needs to be done properly via
Multi-Arch-Cross, eventually.

So the fields are exactly as per Policy - albeit that the parsing of
those fields for a $host architecture is not currently supported
properly by dpkg or apt.

> > > xapt is doing two jobs it seems:
> > > 1) from the package list, make a list containing all the cross packages
> > >   needed
> > 
> > That is given with simulation option (-n).
> 
> This is with embuilddeps, not xapt?

embuilddeps parses the debian/control file or the .dsc to get a list of
packages. That list is then passed to xapt which downloads the host
architecture binaries, passes those to dpkg-cross which converts and
installs the -cross packages.
 
> This option appears to give a package list, prefixed with
> "/usr/bin/apt-get -y".  This just isn't suitable.
> - it's omitted packages which were already installed; this is not
>   necessary and involves poking around under /var/lib/dpkg.
> - we need to strip off the "/usr/bin/apt-get -y" which might be subject
>   to change and then cause breakage, at some point in the future.
> All we want is the /complete/ package list, without any removal of
> packages or addition of other bits.

The list output by the -n|--simulate option does NOT include any of the
dependencies of the packages listed. The complete package list needs to
be parsed from the listing of /var/lib/xapt/archives (for the
downloaded host architecture packages) and /var/lib/xapt/output/ for
the -$arch-cross packages converted by dpkg-cross.

> > > 2) install them (but it doesn't list them in the list of packages to
> > >   install, which makes removal harder potentially)

See the directory listing.

> > > From the resolver POV, what would be really good if it could just do
> > > (1).  This way, we can just give the expanded list back to the
> > > resolver for it to install. 

The resolver can't handle the conversion of the package name from foo
to foo-armel-cross and the resulting changes in the dependencies from
Depends: bar (>= 1.2.3) to Depends: bar-armel-cross (>= 1.2.3)

No resolver *can* handle that (current, past or future) - this is why we
need Multi-Arch-Cross. xapt doesn't try to handle it. xapt simply gets
the *unconverted* build-dependencies and downloads them, dpkg-cross
converts them and installs. The entire process is based on simply
converting and installing everything that appears to be a .deb
in /var/lib/xapt/archives/ and putting the converted stuff
into /var/lib/xapt/output prior to using dpkg -i *.

> Because we need to clean up after
> > > ourselves, the apt and aptitude resolvers build a dummy
> > > "dependency package" from the dependency list, and then get apt/
> > > aptitude to install it.
> > 
> > This is how old pbuilder used to do it, but it does not work with
> > cross packages as adding dependencies on
> > $foreign_package-$HOST_ARCH-cross won't work, as those packages are
> > output from dpkg-cross and not available from repositories.
> 
> Right.  But with sbuild, as I mentioned above, we can install the
> dpkg-cross-generated packages into a local archive, which *is*
> accessible to apt-get/aptitude, and hence you will then be able to
> use cross packages in the dependencies /directly/.  This will permit
> proper cleanup after a build using the normal mechanisms sbuild uses
> for all builds.

You can copy the converted packages in /var/lib/xapt/output into an
archive, yes. Or you could just use the packages which are already
installed and parse the directory listing to get the list to remove -
always remembering to skip $host-cross packages which would otherwise
cause the removal of the toolchain. This hack too will disappear with
M-A-C.

> This is to say the resolver can do the following:
> - get the cross dependencies from the dsc or unpacked source
> - download the debs
> - run dpkg-cross
> - install the dpkg-cross-generated debs into the local archive
> - add the cross-dependencies to the package build dependencies
> ...
> - run dpkg-buildpackage with the necessary options to cross build
> ...
> 
> We already allow various things to add to the package build deps,
> and so adding cross-deps will be trivial.  All we really need is the
> necessary information for what to parse from the dsc or source package,
> and then how to use dpkg-cross.
> 
> Just to be clear: none of the above uses xapt/embuilddeps, because
> sbuild can, I think, do a better job with its local archive and which
> will integrate much better with our existing cleanup code when it comes
> to removing build deps.

It will have to use embuilddeps because Dpkg::Deps.pm is *not able* to
resolve the build-dependencies for a different $host architecture.

It will need to use either xapt or something almost identical to xapt
(which is not much more than a wrapper to apt-get -o
Apt::get::Architecture=armel -d install $packages which sbuild would
have to do anyway). All that's being changed is the final path. It's
embuilddeps which is the attempt at being clever and resolving
dependencies for a foreign architecture using modifications to
Dpkg::Deb.pm.

Doing this without xapt and embuilddeps is just going to cause more
work because dpkg-cross IS going to be removed and the entire process
will have to be re-written.

> > > We then reverse the process to clean up
> > > after the build.  So ideally, we don't really want xapt to do any
> > > installation: we just want it to give us the package list.  If
> > > this is possible, it might not even be require to have a separate
> > > XaptResolver--we just expand the cross-dependencies and give them
> > > to the regular resolver.

No. There is no point re-doing all the work encoded within xapt and
embuilddeps and then doing it all again when Multi-Arch-Cross arrives.

Do it once - when M-A-C is usable.

> > > If xapt can't do this directly, how complex is the code that
> > > implements (1)?  Would it be possible to split it out as a separate
> > > tool?

By the time you've got the list of converted packages, there is nothing
for the resolver to do except dpkg -i *

> > embuilddeps -a armel -d /path/to/source -n will give you that
> > information, and /path/to/dsc might be in the way.

That's meant to be human-readable, not machine-parseable.

> > We really need to think more about how we are going to do it to be
> > able to fit multiarch in the cooking.

Exactly - use xapt/embuilddeps and then handle M-A-C later.
 
> As I mentioned above, I really don't like this.  It's not giving us the
> unfiltered information we need. 

The unfiltered information you need is simply the directory listing
of /var/lib/xapt/archives and /var/lib/xapt/output

> And it's telling us nothing about what
> xapt is actually going to install (in terms of letting us reverse the
> procedure). 

Those directory listing specify *exactly* what xapt has downloaded and
what dpkg-cross has converted and installed.

> embuilddeps is giving us command lines to do the job.  What we need is
> the information to do that job ourselves.

No. The information needs to come from the results of xapt and
embuilddeps.

-- 


Neil Williams
=============
http://www.linux.codehelp.co.uk/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110414/2c809af7/attachment.pgp>