[buildd-tools-devel] Bug#610689: Bug#610689: sbuild: cross support

[Roger Leigh]

On Sat, Jan 22, 2011 at 09:56:26PM +0000, Hector Oron wrote:
> Hello,
> 
> 2011/1/22 Roger Leigh <rleigh at codelibre.net>:
> > On Fri, Jan 21, 2011 at 12:09:55PM +0000, Hector Oron wrote:
> 
> > I think that the several uses of the --arch option are going to need
> > separating into separate options.  Does the following make sense, or
> > am I missing out anything major?
> 
> > 1) Selection of the build chroot
> >
> >   We look for a chroot named $distribution-$arch-sbuild or
> >   $distribution-$arch
> >
> >   When we are genuinely building for a different arch, then this works.
> >   When we're cross-compiling, we want to build on a chroot using the
> >   host arch (most likely, though you /could/ use a different arch with
> >   qemu and do an emulated cross-build once we have such support!)
> 
> $distribution, as in Debian or derivatives could be guessed using
> dpkg-vendor variables.
> 
> $distribution as in Debian suites probably needs to be passed as argument,
> as it is probably now.

$distribution means the latter (the suite name), as used in
debian/changelog.  If we could read it directly from debian/changelog,
that would be great, but that's not usually possible.  There's a
chicken-and-egg problem with this because we need to know the
distribution in order to download and/or unpack the sources most of
the time, and we can't know that until we have the sources, so it
must be specified.

$arch is referring to the build architecture only (this is what the
patch I just sent uses, though I realise I have confused host and build
in my patch, I'll swap them around).

> $arch, it is splitted by dpkg-architecture into DEB_BUILD_* and
> DEB_HOST_*.
> 
> Let's use m68k as an example, in case we do native build:
>   DEB_BUILD_ARCH (m68k) = DEB_HOST_ARCH (m68k)
> 
> if we do a cross-build
>   DEB_BUILD_ARCH (amd64|i386) != DEB_HOST_ARCH (m68k)
> we need to use DEB_BUILD_ARCH chroot and follow DEB_HOST_ARCH
> *.changes file.

Hmm, I think I've confused host and build in my patch, I'll swap them
around.

> if we do an emulated build
>   DEB_BUILD_ARCH (amd64|i386) = DEB_HOST_ARCH (amd64|i386) -- out of the chroot
>   DEB_BUILD_ARCH (m68k) = DEB_HOST_ARCH (m68k) -- within the chroot

Yes.  From the point of view of sbuild, it shouldn't need to worry about
emulated builds; it should be entirely transparent and treated no
differently than normal native builds.

> I do not think we need a new variable on dpkg-architecture, as this is same
> as doing a native build but picking a foreign architecture, which we could use
> current --arch switch on sbuild. If --arch is used, dpkg
> --print-architecture checks
> should probably need to be bypassed.

It should work with sbuild today (once schroot does this).  The check
should still succeed when --arch is used (after all, it's a sanity
check to ensure we have the correct build arch in the chroot).  If
"dpkg --print-architecture" returns the emulated arch in the chroot
(and since it's the native m68k package I would assume it should return
the correct value), then that should be sufficient.  Is this how qemu
works?  Would it return m68k on amd64 in an m68k chroot with qemu?

> > 2) Selection of the host architecture
> >
> >   Linked to (1).  We are assuming that the build and host architectures
> >   are the same, so --host is implicitly setting both.
> >
> > I think we should add two additional options:
> >  --host   select host arch
> >  --build  select build arch
> > (or whatever are most commonly used in existing Debian tools; these are
> > from autoconf configure) and have "--arch=foo" be equivalent to
> > "--host=foo --build=foo".
> >
> > The chroot selection will use the host arch only.  So this will
> > typically be the "real" host arch, but it will also allow "native"
> > (non-cross) building using qemu as well as kernel personalities (i386
> > on amd64 etc, which is already supported).  When we come to run
> > dpkg-buildpackage, we will then pass the build arch which will result
> > in a cross-build if this differs from the host arch.
> >
> > Hopefully this will implement support fairly cleanly and flexibly.
> 
> On the previous options described above, we might want to develop some
> syntax in the control files which can mark packages to be able to be cross
> built or built emulated. Sbuild should decode that information from control
> file information, then set accordingly --host, --build or whatever we call them,
> but I am afraid this naming might help confusing things much more with build,
> host and target.

What's the use case for such information?

sbuild can certainly extract such information from the control file.  It
couldn't set --host/--build though surely?; (like the --distribution, this
would need specifying by the user/buildd running sbuild since they would
be build-specific depending upon the arch one wants to build for.)

Information telling us that cross-compilation works (or not) would allow
for aborting builds that are pointless.  OTOH, such information might
better belong in databases like Packages-arch-specific.

> (( Info: I was going to suggest the use of  '--cross $arch --arch $_arch'
> which implies building on a chroot of architecture $_arch and passing
> dpkg-buildpackage '-a$arch' in the build. Then, in the case of emulated build,
> we could explicitly tell sbuild by other option '--emulate' (or else).
> But maybe your suggestion is cleaner ))

I think --build/--host match the existing usage in dpkg-architecture
and autoconf etc, so may possibly be clearer, but I'm by no means an
expert!

--emulate should never be necessary should it?  If the emulation is
completely transparent, why would we need to be informed about it, and
what could we do with the information?

> > Following on from that there's the issue of resolver support and
> > use of xapt.  Some examples of how xapt would be used would help
> > here; I'm a bit unsure of if it's used directly as an apt-get
> > wrapper, or if it's a multi-step process.  Understanding the
> > workflow would be most useful.
> 
> xapt reads build dependencies of package and then it downloads those
> for the cross architecture, passing the packages to dpkg-cross which
> puts headers and libraries ready to be used by the cross tools.
> It can be called directly from command-line (before running the build).
> All this trickery should go away after multiarch bits are in place which
> should be able to provide libraries and headers cleanly and transparently
> for foreign architectures.

Could I possibly see an example of its command-line usage and output?
One thing we would need to do it know what it installed/removed so we
can reverse it post-build.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110122/c5432d34/attachment.pgp>