[buildd-tools-devel] Bug#593516: Bug#593516: schroot: race condition in /proc/mounts can break cleanup

Roger Leigh rleigh at codelibre.net
Thu Aug 19 19:21:31 UTC 2010


On Wed, Aug 18, 2010 at 05:12:35PM -0400, Greg Price wrote:
> schroot suffers from a race condition where if two schroot sessions
> are ended concurrently, one of them may fail to clean up properly.  On
> some intensely schroot-using build machines, I see this failure
> regularly.  The symptom looks like this:
> 
>   E: 10mount: umount:
> /var/lib/schroot/mount/somechroot-source-bd8cdb9f-611a-4991-869c-e479fa673ec7:
> device is busy.
> 
> and then the 'schroot' command exits with failure.
> 
> The issue is that /proc/mounts is unreliable.  It turns out that when
> you read it concurrently with a umount, you can end up missing records
> for mounts that have nothing to do with the ones that were unmounted.
> Arguably this is a kernel bug, but there is no simple fix, so
> /proc/mounts will definitely remain unreliable at least for squeeze.

This is rather annoying, but thanks for bringing it to my attention.

> Currently I'm working around the issue with a flock around the body of
> do_umount_all().  That's easy and is enough to address the problem
> when no other umounts are happening on the system.

If you would like this fix including, I'll be happy to accept a
patch implementing this.  Are you doing this in the shell script?
If it's possible to directly lock /proc/mounts with fcntl this could
be addressed directly in schroot-listmounts using sbuild::file_lock,
which wraps fcntl.

Has a bug report been filed against the Linux kernel or on the
kernel mailing list?  A fix in the kernel would be ideal, and they
should probably also be notified.

> A real fix would
> probably have to be for schroot to record for itself the list of
> mounts it makes inside a session's tree, and rely on that list instead
> of on /proc/mounts.

This wouldn't be robust enough, I'm afraid.  Processes inside the
chroot (or outside, for that matter) could mount (and umount)
filesystems after initial setup.  Mounts might also occur in
additional custom setup scripts.  As a result, we really do need
to get a definitive list of mounts at the session end in order to
clean up correctly.

At least on Linux, one other approach would be to make use of
per-process namespaces which would TTBOMK allow us to skip the
umounts completely since it's automatically cleaned up when the
last process in the namespace exits.  This has only recently
become viable without significant rearchitecting of schroot
though (IIRC it's now possible to connect to an existing namespace
without a parent-child relationship, though I'm not familiar with
the details).


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20100819/6f325a0b/attachment.pgp>


More information about the Buildd-tools-devel mailing list