[Buildd-tools-devel] schroot: union filesystem based schroots

[Jan-Marek Glogowski]

> Jan-Marek Glogowski <glogow at fbihome.de> writes:
>
> >> > It's a first implementation based on the code for LVM snapshots.
> >> > It's not well tested yet (just Etch and unionfs 1.4), but I want to get
> >> > some feedback, especially how to solve the following problem:
> >> >
> >> > Old unionfs crashs, if you change any of it's branches not throught
> >> > unionfs while unionfs is overlaying them. Therefore I need some lock when
> >> > someone wants to enter the parent changeroot, while sessions are running.
> >> > This should block new sessions from startup, and should block the parent
> >> > until all sessions are closed.

[stripped locking scheme]

> This looks sensible, though I need to give it a bit more thought.
> Once consideration is how to do lock recovery if things get messed up.
> As an example, the current file and device locks are completely
> dependent upon an existing process to own them.  This is intrinsic to
> file (fcntl/lockf) byte region locks, and is done in liblockdev for
> device locks (pid in lock file).  This lets us detect if a lockfile
> exists but can be safely removed (e.g. due to a crash or manual
> intervention).  We will also need to tie the lock to something, though
> I guess the lifetime of a chroot session would be good enough.

The "pid-file-lock" is exactly the way I wanted to implement these
file-locks using liblockfile.

> > From evaluating all the ideas, I think it's a good idea to move the union
> > stuff into source. Maybe we can't think of "good" examples for the
> > combination of lvm-snapshot or file and fs-union. Maybe it wouldn't be
> > useful with default combination, but someone can add a completely custom
> > branch definition, with multiple rw branches to store central logfile in a
> > second rw branch (which could also be done via mount-defaults), but even
> > if layering doesn't make much sense, it should not hurd (exept for the
> > branch change outside of union crash, which might not be valid anymore -
> > it was just a BUG in the unionfs 1.4 manpage).
> >
> > Next time I'll export my git-tree on my homepage, so you can clone my
> > branch from there, or should I send more patches to this list?
>
> Being able to clone it (or add as a "remote" and merge directly) would
> be useful, and give me an excuse to try out git features I've never
> had the chance to try out before!  But, I would still like us to send
> patches to the list--it makes discussing things a bit easier I think,
> and lets everyone else reading the list see what's going on.
>
> >> Other minor suggestions/questions:
> >>
> >> Should mount not be called with -p ?
>
> Sorry, that should have been "mkdir".

Yep - changed that.

> >> Is it our place to modprobe kernel modules?  Or, should be admin load
> >> the modules specifically?
> >
> > The script just tries to modprobe the fs-module, if it can't find it in
> > /proc/filesystems. I can add a flag "auto-modprobe", but on the other hand
> > its possible to explicitly select the fs to try for every chroot
> > (fs-type). For me its convenience for non-root users and I don't have to
> > add explicit loading to /etc/modules after a reboot.
>
> I'm just thinking of security issues here, as well as not screwing
> with policies the admin may have implemented; kmod should autoload any
> missing modules on demand--there should never be a need for user-level
> code to be modprobing things on a properly configured system, IMHO.

Ok - so I'll simply check after mount, if the fs exists in
/proc/filesystems, so I can distinguish, if the failed mount is a branch
config problem or simply an unsupported fs-type, which will trigger the
fallback code for the next fs union type.

> >> overlay_base: While a temporary directory is OK, we also need to
> >> consider that the sessions should (by default) get preserved over a
> >> reboot.  As a result, we might want to create /var/lib/schroot/overlay
> >> for the purpose.  If having it on a tmpfs is important, we can allow
> >> the user to customise this in schroot.conf.
> >
> > Preserving the session should work, as long as the branch inodes didn't
> > change. Not sure what happens, if the unions virtual inodes wouldn't match
> > the real inodes anymore...
>
> OK.  This is one thing we will need to test.
>
> > - I'll split the patch and move my "Etch control and bootstrap" into a
> >   second patch and also remove the uneeded configure.ac changes.
>
> OK.
>
> > - I'll merge the "fs-union" code into the chroot_source, as I can't think
> >   of any impossible combination. This will add the "fs-union-type" "none"
> >   as a default, so we won't have union overlays as default.
>
> Thinking a bit more about this, I think it still makes sense as a
> separate module, even if you do derive it from chroot_source.  I also
> want it to run on non-Linux kernels, e.g. BSD, and doing it this way
> means we can simply not compile the file, and #ifdef its use in other
> files.

There is no linux specific stuff in the sourcecode, except for the
supported fs-type list. So for BSD you would change the enum FSTypes and
the known_types string list and that's it. I don't think these lists will
change often, so don't need to be configureable, as each type needs
support in the setup script anyway. Or am I missing your point?

> > The only missing piece of the plan is the "moment", when the union is
> > mounted. Currently it's before any other mount but this won't work with
> > lvm, which will cover the whole unionfs.
>
> Well, the LVM code doesn't mount anything.  If you run it after 05lvm
> but before 10mount, you can set up the mount and overlay.  I think we
> should probably split 10mount at this point, so that it just mounts
> filesystems /inside/ the chroot (/proc, /sys, /home etc.) and make
> mounting of the chroot happen earlier.  This will give us the
> flexibility to set up union mounts (or not).
>
> > Probably we can mark mounts before and after union? An other idea would be
> > to mount everything ro before union, everything rw after. As for fstab,
> > default would be rw, but this - again won't work with lvm-snapshot...
> > Any idea for a good solution?
>
> Sorry, but I'm not entirely clear on what you are describing here.  I
> can see that mounting the base filesystem r/o is needed, but I'm not
> sure why fstab is involved--the filesystems inside the chroot would be
> mounted after the unionfs setup, so I'm not sure what fstab would have
> to do with it (do you mean /etc/fstab or the per-chroot fstab?).

I was refering to /etc/schroot/mount-defaults (or $FSTAB). I was thinking
of the situation, that someone wants to protect some mounts from changes
(mounted before fs-union) and some not (mounted after fs-union).

> BTW, I branched off a stable branch of schroot (schroot-1.2) for
> Lenny, which is now basically frozen.  So this would go on the master
> branch for Lenny+1.  Once it's all working and tested, we could then
> merge it onto the schroot-1.2 branch and make a point release.  This
> is such a nice feature, I'd like to do that if possible.

Jan-Marek