[Buildd-tools-devel] schroot: union filesystem based schroots

[Roger Leigh]

Jan-Marek Glogowski <glogow at fbihome.de> writes:

>> > It's a first implementation based on the code for LVM snapshots.
>> > It's not well tested yet (just Etch and unionfs 1.4), but I want to get
>> > some feedback, especially how to solve the following problem:
>> >
>> > Old unionfs crashs, if you change any of it's branches not throught
>> > unionfs while unionfs is overlaying them. Therefore I need some lock when
>> > someone wants to enter the parent changeroot, while sessions are running.
>> > This should block new sessions from startup, and should block the parent
>> > until all sessions are closed.
>>
>> It should be possible to create some form of lockfile to do this.  We
>> already have lock facilities (sbuild/sbuild-lock.h), but these are for
>> either files or block devices, rather than mounts.  As a result, I
>> don't think they would be appropriate.  One mechanism would be to look
>> at existing sessions, where we can lock the session files, but it
>> would still be slightly racy I think.
>
> I was thinking of creating locks in /var/lock, like
>
> /var/lock/<session>.lock
> /var/lock/<chroot-name>-manage.lock
>
> Here comes the masterplan for the algo...
>
>  1. get manage.lock
>  2. look for <chroot-name>-source-<session>.lock
>
>  2.1 -e <chroot-name>-source-<session>.lock
>
>  2.1.1 remove manage.lock
>  2.1.2 wait x and check for CTRL+C
>  2.1.3 if break and request is source chroot
>
>  2.1.3.1 remove <chroot-name>-source-<session>.lock
>  2.1.3.2 exit
>
>  2.1.4 goto 1
>
>  2.2 ! -e <chroot-name>-source-<session>.lock
>
>  2.2.1 I'm source or session?
>
>  2.2.1.1 I'm source
>  2.2.1.2 create <chroot-name>-source-<session>.lock
>  2.2.1.3 -e <chroot-name>-(*=^source).lock
>
>  2.2.1.3.1 yes -> goto 2.1.1
>  2.2.1.3.1 no -> goto 2.2.2
>
>  2.2.1 create <session>.lock
>  2.2.2 remove manage.lock
>  2.2.3 start chroot session
>
>  2.2.4.1.1 fails - remove manage.lock
>
>  2.2.4.2.1 ok - do chroot
>  2.2.4.2.2 remove lock on exit
>
> I think this scheme could be generally used, also to make non-session
> chroots more secure to be used by multiple users.
>
> Own locks can always be removed without the manage.lock.

This looks sensible, though I need to give it a bit more thought.
Once consideration is how to do lock recovery if things get messed up.
As an example, the current file and device locks are completely
dependent upon an existing process to own them.  This is intrinsic to
file (fcntl/lockf) byte region locks, and is done in liblockdev for
device locks (pid in lock file).  This lets us detect if a lockfile
exists but can be safely removed (e.g. due to a crash or manual
intervention).  We will also need to tie the lock to something, though
I guess the lifetime of a chroot session would be good enough.

> From evaluating all the ideas, I think it's a good idea to move the union
> stuff into source. Maybe we can't think of "good" examples for the
> combination of lvm-snapshot or file and fs-union. Maybe it wouldn't be
> useful with default combination, but someone can add a completely custom
> branch definition, with multiple rw branches to store central logfile in a
> second rw branch (which could also be done via mount-defaults), but even
> if layering doesn't make much sense, it should not hurd (exept for the
> branch change outside of union crash, which might not be valid anymore -
> it was just a BUG in the unionfs 1.4 manpage).
>
> Next time I'll export my git-tree on my homepage, so you can clone my
> branch from there, or should I send more patches to this list?

Being able to clone it (or add as a "remote" and merge directly) would
be useful, and give me an excuse to try out git features I've never
had the chance to try out before!  But, I would still like us to send
patches to the list--it makes discussing things a bit easier I think,
and lets everyone else reading the list see what's going on.

>> Other minor suggestions/questions:
>>
>> Should mount not be called with -p ?

Sorry, that should have been "mkdir".

>> Is it our place to modprobe kernel modules?  Or, should be admin load
>> the modules specifically?
>
> The script just tries to modprobe the fs-module, if it can't find it in
> /proc/filesystems. I can add a flag "auto-modprobe", but on the other hand
> its possible to explicitly select the fs to try for every chroot
> (fs-type). For me its convenience for non-root users and I don't have to
> add explicit loading to /etc/modules after a reboot.

I'm just thinking of security issues here, as well as not screwing
with policies the admin may have implemented; kmod should autoload any
missing modules on demand--there should never be a need for user-level
code to be modprobing things on a properly configured system, IMHO.

>> Why does gettextize need adding to bootstrap?  We already call
>> autopoint to set it up.  Also, why does the gettext version change in
>> configure.ac?--through I have no objection to using a later version.
>
> On my first checkout I didn't saw the bootstrap file and calling auto* was
> missing some stuff. Maybe it's not needed anymore. I really don't care
> about the gettext version.

I've updated the minimum version to 0.16, since it removes some
annoying (but harmless) warnings.

>> overlay_base: While a temporary directory is OK, we also need to
>> consider that the sessions should (by default) get preserved over a
>> reboot.  As a result, we might want to create /var/lib/schroot/overlay
>> for the purpose.  If having it on a tmpfs is important, we can allow
>> the user to customise this in schroot.conf.
>
> Preserving the session should work, as long as the branch inodes didn't
> change. Not sure what happens, if the unions virtual inodes wouldn't match
> the real inodes anymore...

OK.  This is one thing we will need to test.

> - I'll split the patch and move my "Etch control and bootstrap" into a
>   second patch and also remove the uneeded configure.ac changes.

OK.

> - I'll merge the "fs-union" code into the chroot_source, as I can't think
>   of any impossible combination. This will add the "fs-union-type" "none"
>   as a default, so we won't have union overlays as default.

Thinking a bit more about this, I think it still makes sense as a
separate module, even if you do derive it from chroot_source.  I also
want it to run on non-Linux kernels, e.g. BSD, and doing it this way
means we can simply not compile the file, and #ifdef its use in other
files.

> The only missing piece of the plan is the "moment", when the union is
> mounted. Currently it's before any other mount but this won't work with
> lvm, which will cover the whole unionfs.

Well, the LVM code doesn't mount anything.  If you run it after 05lvm
but before 10mount, you can set up the mount and overlay.  I think we
should probably split 10mount at this point, so that it just mounts
filesystems /inside/ the chroot (/proc, /sys, /home etc.) and make
mounting of the chroot happen earlier.  This will give us the
flexibility to set up union mounts (or not).

> Probably we can mark mounts before and after union? An other idea would be
> to mount everything ro before union, everything rw after. As for fstab,
> default would be rw, but this - again won't work with lvm-snapshot...
> Any idea for a good solution?

Sorry, but I'm not entirely clear on what you are describing here.  I
can see that mounting the base filesystem r/o is needed, but I'm not
sure why fstab is involved--the filesystems inside the chroot would be
mounted after the unionfs setup, so I'm not sure what fstab would have
to do with it (do you mean /etc/fstab or the per-chroot fstab?).


BTW, I branched off a stable branch of schroot (schroot-1.2) for
Lenny, which is now basically frozen.  So this would go on the master
branch for Lenny+1.  Once it's all working and tested, we could then
merge it onto the schroot-1.2 branch and make a point release.  This
is such a nice feature, I'd like to do that if possible.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url : http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20080123/0b2dee00/attachment.pgp