[buildd-tools-devel] [GIT] sbuild branch user/josch/unshare updated. release/sbuild-0.66.0-5-gac50af2

Sun Dec 13 13:10:27 UTC 2015

Hello fellow sbuild people :)

for comments, ideas and critique I'd like to share a rather large-ish
experimental proof-of-concept commit with you that I just pushed to the branch
user/josch/unshare in the sbuild repository:

Quoting Johannes Schauer (2015-12-13 12:49:08)
> - Log -----------------------------------------------------------------
> commit ac50af22e58ca66695cdb0ab4e2690e3095330ad
> Author: Johannes 'josch' Schauer <josch at mister-muffin.de>
> Date:   Thu Dec 3 22:07:46 2015 +0100
> 
>     proof of concept for new sbuild backends (adt-virt-* and uchroot/user-unshare)
> 
> -----------------------------------------------------------------------

The commit adds three things:

1. Decouple chroot from host
----------------------------

For the existing chroot modes (schroot and sudo) sbuild was allowed to make the
assumption that the chroot is mounted in a file system that the sbuild process
has read/write access to. This made things like copying things into or out of
the chroot or reading and writing files in the chroot very trivial.
Unfortunately this stops sbuild from supporting alternative backends like a
chroot that can only be accessed over network via ssh or that runs in a qemu
virtual machine or inside an lxc container. So I identified (hopefully) all the
instances where sbuild assumes it can directly access the chroot and replaced
them with code that allows the chosen chroot mode to decide in which way sbuild
can communicate with the chroot. This effectively eliminates the 'Location'
variable from Build.pm and also makes the <<CHROOT>> placeholder in the build
log superfluous as the path to the chroot is never used directly anywhere
anymore.

Theoretically this change should not break anything (minus hooks). In practice
I probably introduced some regressions, so more testing is needed with more
input than I tested with so far.

The only practically unavoidable regression of this decoupling that I can think
of is the %SBUILD_CHROOT_DIR placeholder for external sbuild hooks. This is not
meaningful anymore because the chroot location became transparent. The location
will also not be practical anymore when one thinks of chroots in virtual
machines or on remote ssh servers. To still allow the same functionality as was
earlier provided by %SBUILD_CHROOT_DIR I plan to add a variable containing a
command that external scripts can run to execute an arbitrary command inside
the chroot and then be able to communicate with it via pipes.

It would be appreciated if you could test whether normal schroot based
operation still works for you without issues with my patch.

2. Add adt-virt-* backend
-------------------------

Using the decoupling explained in the last section I was able to add a new
backend which is able to make use of the adt-virt-* interface from the
autopkgtest package. This means that sbuild is now able to do a build in an lxc
container, on a networked ssh server or in a qemu virtual machine using the
same interface. For example, the following will demo its functionality using
the adt-virt-schroot backend (the current default):

$ PERL5LIB=$PWD/lib bin/sbuild --chroot-mode=adt -c unstable-amd64-sbuild -d unstable mypkg.dsc

By using the --adt-virt-server option you can choose other adt backends like
lxc, qemu or ssh if you have set these up beforehand.

If you feel adventurous, please test the new adt-virt-* backend.

3. Building packages without superuser privileges (even without setuid root)
----------------------------------------------------------------------------

In fact adding the adt-virt-* backend was just a convenient side-effect for me.
My main motivation to decouple the chroot from the host was to implement a
sbuild backend which would not require superuser privileges to build packages.
Too often it happened to me that for example after a failed or aborted sbuild
run, schroot didn't properly clean up after itself and I'm left with tons of
superfluous mounts on my system. I consider this not so much a schroot bug but
a bug in the whole concept of requiring superuser privileges for building
packages. I do not see why the rest of my system (for example my /proc/mounts)
should be influenced in any way if all I want to do is compile my software in a
chroot. This should be possible without becoming superuser and potentially
doing something dangerous, ever. Unfortunately Linux is still really bad at
this, probably partly because many just accept to just sudo everything. The
good news is, that with Linux namespaces I can now do mount and chroot without
actually being root or being able to influence the rest of my system. I used to
ramble about this in the past [1] and I finally wanted to put my money where my
mouth is.

Another nice side effect of using Linux namespaces as an sbuild backend is,
that we get restricted network access for free. By unsharing the network
namespace as well, dpkg-buildpackage will only be able to access the loopback
interface. This makes superuser-requiring iptable entries or the like a thing
of the past.

So this third addition is a new chroot mode that I call uchroot (name subject
to change). I found uchroot fitting because I suspected that schroot is called
like that because it is chroot via setuid. Uchroot on the other hand is chroot
with user namespaces, that's where the U comes from. The program has a similar
interface to schroot but is very much a proof of concept. In the background it
makes use of the user-unshare program which I first introduced here [2].

To use the uchroot mode, do the following:

 - copy bin/uchroot and bin/user-unshare somewhere into your $PATH
 - mkdir ~/.uchroot
 - touch ~/.uchroot/config # this stays empty for now
 - mkdir ~/.uchroot/chroot.d
 - and put the following into ~/.uchroot/chroot.d/unstable-amd64-sbuild:

Name: unstable-amd64-sbuild
Type: tarball
Path: /srv/chroot/unstable-amd64.tar.gz
Aliases:
 UNRELEASED
Unpack: /tmp

The config file is in deb822 format to allow a structured config file while at
the same time not add more dependencies to sbuild (Dpkg::Control is used for
parsing). Change the Path field to a fitting chroot. Notice that this tarball
must not contain any device nodes because those cannot be created inside an
unshared user namespace for security reasons.

You can then build packages like this:

$ PERL5LIB=$PWD/lib bin/sbuild --chroot-mode=uchroot -d unstable mypkg.dsc

Should this chroot mode show to make sense, I plan to also adapt
sbuild-createchroot so that it can be run without superuser privileges as well.
This is currently not possible with debootstrap (see #698347 and #731802) so it
might be done using multistrap instead (I demonstrated with polystrap how
multistrap can be run without superuser privileges even with support for
foreign architectures through qemu-user [3]).

This third mode is *highly* experimental. Neither do I know whether I want to
keep the interface for uchroot as it is nor if I want to keep a separate
user-unshare program. David Lechner got the idea to use the code to write a
program with an interface similar to proot but using Linux namespaces instead
of ptrace to become pseudo-root. That sounds like a good idea but it misses the
management of tarballs that schroot is able to do.

4. Conclusions
--------------

Decoupling the host from the chroot as explained in the first section allows
lots of useful stuff, so I would like to get the respective changes into the
next sbuild release. For that, more testing would be needed though, so that my
changes do not break too many systems.

Support for adt-virt-* backend sounds like the next desirable feature which
would already make sbuild much more powerful than it is now.

The third point is still very much of a pipe dream as I yet have to figure out
how to implement everything correctly with an interface that makes sense and
with all the error checks that are not present yet in uchroot (or however that
program will end up being called). I'm also hesitant to make uchroot its own
package because that would mean more maintenance from my side. On the other
hand, sbuild itself is probably not the best location for it either.
Suggestions and help are very much welcome!

Finally it is needless to say that the patch needs lots of work still to make
it absolute water tight in all three areas. I'm looking forward to your
replies!

cheers, josch

[1] https://blog.mister-muffin.de/2014/01/11/why-do-i-need-superuser-privileges-when-i-just-want-to-write-to-a-regular-file/
[2] https://blog.mister-muffin.de/2015/10/25/unshare-without-superuser-privileges/
[3] https://gitlab.mister-muffin.de/josch/polystrap
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20151213/8950746f/attachment.sig>