[buildd-tools-devel] Bug#593516: schroot: race condition in /proc/mounts can break cleanup

[Greg Price]

Package: schroot
Version: 1.3.2-1
Severity: normal

Hello,

schroot suffers from a race condition where if two schroot sessions
are ended concurrently, one of them may fail to clean up properly.  On
some intensely schroot-using build machines, I see this failure
regularly.  The symptom looks like this:

  E: 10mount: umount:
/var/lib/schroot/mount/somechroot-source-bd8cdb9f-611a-4991-869c-e479fa673ec7:
device is busy.

and then the 'schroot' command exits with failure.

The issue is that /proc/mounts is unreliable.  It turns out that when
you read it concurrently with a umount, you can end up missing records
for mounts that have nothing to do with the ones that were unmounted.
Arguably this is a kernel bug, but there is no simple fix, so
/proc/mounts will definitely remain unreliable at least for squeeze.

Consequently because schroot-listmounts relies on /proc/mounts, it too
is unreliable.  So when do_umount_all() in etc/setup.d/10mount does
what amounts to this (but with better reporting and error handling):

    "$LIBEXEC_DIR/schroot-listmounts" -m "$base" | xargs -rn1 umount

the schroot-listmounts may skip a record if another process is ending
another schroot session, or unmounting something for any other reason.
Then one of the internal mounts, like .../tmp or .../dev/pts, remains
mounted, and the attempt to unmount the base directory fails.

Currently I'm working around the issue with a flock around the body of
do_umount_all().  That's easy and is enough to address the problem
when no other umounts are happening on the system.  A real fix would
probably have to be for schroot to record for itself the list of
mounts it makes inside a session's tree, and rely on that list instead
of on /proc/mounts.

Cheers,
Greg