[buildd-tools-devel] Bug#749897: Can't end session if daemon starting after a schroot session is created uses a private mount namespace.

Wookey wookey at wookware.org
Tue Aug 2 18:30:14 UTC 2016


Source: schroot
Version: 1.6.10-1+b1
Followup-For: Bug #749897

OK. I think I have the same problem as Mike reports in this bug.

I use schroot a lot - it's great.

However I do have problems with dead chroots collecting, and being very hard to tidy up.

Currently I have: (schroot -la)
chroot:jessie-amd64-sbuild
chroot:trusty-amd64-sbuild
chroot:unstable-amd64-sbuild
chroot:vivid-amd64-sbuild
session:jessie-amd64-sbuild-3d1e5ddb-1ebb-4ebe-b22e-4986db9007a4
session:jessie-amd64-sbuild-436b01b7-5b83-4677-b6bb-fad7d7207dc6
session:jessie-amd64-sbuild-85e5f4a1-2d38-4cd5-9491-607892d02143
session:rebootstrap
session:unstable-amd64-sbuild-0b2a2880-3035-43a6-bcc6-eee08ac8b766
session:unstable-amd64-sbuild-0d9bd969-9a85-4c3e-bff9-6717171e2bea
session:unstable-amd64-sbuild-0e62e5f3-801e-40d1-8606-dc0b69a4fcb9
session:unstable-amd64-sbuild-4865bdd5-86ac-4e86-8b38-4787191a3ed1
session:unstable-amd64-sbuild-4e5d1f1c-02b1-4900-b72e-2726d4c1e422
session:unstable-amd64-sbuild-513ce5bc-d33b-4bbb-840e-29248850b02f
session:unstable-amd64-sbuild-5c5bee45-1ea0-4e24-a054-db2c554c7781
session:unstable-amd64-sbuild-5e47eef3-7f8a-43c8-9811-b29c599957f0
session:unstable-amd64-sbuild-6b8aeae9-34ef-4861-9d4c-7a9cd24a6ca3
session:unstable-amd64-sbuild-6d593108-2398-4b83-8ccf-09fe2b4b3f25
session:unstable-amd64-sbuild-8a8a5234-9e64-45dd-aa5d-4339ab73b4bc
session:unstable-amd64-sbuild-91f36ce2-e46c-4f1b-bc77-10657d7eed41
session:unstable-amd64-sbuild-b50831d9-652f-40c5-accf-96a06714870d
session:unstable-amd64-sbuild-c2b25759-aa57-4031-93e3-3860d55fe203
session:unstable-amd64-sbuild-dcbb2fb4-6161-47c8-bb3d-3e25b242859c
session:unstable-amd64-sbuild-f1171bad-f606-4e9b-be2b-77d19a34b555
session:unstable-amd64-sbuild-fc898033-a427-4b0d-bfc8-9a5dc446e776
source:jessie-amd64-sbuild
source:trusty-amd64-sbuild
source:unstable-amd64-sbuild
source:vivid-amd64-sbuild

That's _after_ doing 
schroot --end-session --all-sessions

and also rebooting in the hope of getting rid of leftover 'mounts' preventing tidy-up.
Unfortunately schroot's automatic init script which tries to recover
all current sessions rather gets in the way here.

Let's take one example (a named chroot as it's easier to read):
sudo schroot --end-sessieon -c session:rebootstrap
update-binfmts: unable to open /var/lib/schroot/mount/rebootstrap/bin/sh: No such file or directory
rmdir: failed to remove '/var/lib/schroot/mount/rebootstrap': Device or resource busy

The updatebinfmts thing does nt really matter, it's the 2nd error
which is preventing tidy-up. This comes from the 10mount script in /etc/schroot/setup.d/10mount

Now the thing is that I can't work out _what_ is making that resource
'busy'. Anyone got any clues?

It is not mounted according to mount (or /etc/mounts)
sudo lsof /var/lib/schroot/mount/rebootstrap
reports nothing.
sudo lsof +D /var/lib/schroot/mount/rebootstrap
(which should recurse) reports nothing 
sudo fuser /var/lib/schroot/mount/rebootstrap
also reports nothing.

I am deeply perplexed as to what is actually stopping rmdir tidying up that directory.

If I reboot, then this session is recovered, and is mounted again:
$ mount  | grep rebootstrap
/dev/mapper/vg_root-lv_root on /var/lib/schroot/mount/rebootstrap type ext4 (rw,relatime,errors=remount-ro,data=ordered)
proc on /var/lib/schroot/mount/rebootstrap/proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /var/lib/schroot/mount/rebootstrap/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devpts on /var/lib/schroot/mount/rebootstrap/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /var/lib/schroot/mount/rebootstrap/dev/shm type tmpfs (rw,relatime)
/dev/mapper/vg_root-lv_work on /var/lib/schroot/mount/rebootstrap/home type ext4 (rw,relatime,data=ordered)
/dev/mapper/vg_root-lv_root on /var/lib/schroot/mount/rebootstrap/build type ext4 (rw,relatime,errors=remount-ro,data=ordered)

schroot -r -c session:rebootstrap
gets me back into the chroot

sudo schroot --end-session -c session:rebootstrap
still fails as described above.

So what is going on? How do I get rid of all those chroots (which all
seem to have this problem of not wanting to go away)

---------------

So, having discovered this bug I have some new things to check:
This machine is using systemd. 
On my other machine which is still using init, I do not see this
problem. (I still get 'leftover' schroots, but they can be tidied)


Stuart says: 
For those wondering how to get rid of schroot sessions that schroot
doesn't seem to be able to clean up, shutdown the daemons involved,
schroot -e --all- sessions, restart the daemons

OK. but how do I find out what that daemons are?

There are 200 processes listed on this machine. I don't know which is
the offending one.

Aha. Important to do 
sudo readlink /proc/*/ns/mnt | sort | uniq -c
not 
readlink /proc/*/ns/mnt | sort | uniq -c
as the latter only reveals one mount namespace.
the former shows 
    201 mnt:[4026531840]
      1 mnt:[4026531856]
      1 mnt:[4026532308]
      1 mnt:[4026532323]
      1 mnt:[4026532409]

and tracking those via 
sudo ls -l /proc/*/ns/mnt | grep $mountspace
ps -p $PID
I find that 
   47 ?        00:00:00 kdevtmpfs
  976 ?        00:00:00 cupsd
 1341 ?        00:00:00 colord
 3633 ?        00:00:00 rtkit-daemon

are the offending processes.
stopping the services does indeed allow me to clean-up
session:rebootstrap (and other sessions)

As Stuart says, you have to remove the affected chroot, before
restarting the demons. It is not necessary to kill/stop kdevtmpfs

Zdenek's script is a little heavy-handed. I normally want to be able
to clean up specific chroots, not all of them at once. And that's what
schroot needs to do too. The obvious thing is to stop any services in
private mount space, before cleaning up. Should they be restarted? Are
they associated with one of the chroots, or with the outer system?

I don't yet understand why these things have anything to do with
chroot mounts. I have cups running on my machine anyway. It has
nothing to do with schroot chroots, so why does its use of a private
mount space (if started by systemd) get in the way of removing the
chroot mount point?

Anyway, thank you to previous bug reporters for teaching me about
'mount namespaces' of which I was totally unware until today.

Now we just have to work out what to do about them (other then 'don't
use systemd, it's too bloody clever by half).

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/



More information about the Buildd-tools-devel mailing list