jenkins-debian-glue: recursive bind mount problem with parallel builds
Philipp Hahn
hahn at univention.de
Thu Aug 27 13:04:33 UTC 2015
Hi Michael, cc:pbuilder-maint,
hope you made it back from DebConf15 allright - thank you for your talk
about Continuous Delivery.
I'm currently looking at a problem with jenkins-debian-glue: I build
multiple packages in parallel, as my host has multiple CPUs. I have the
strange phenomenon, that the 2nd cowbuilder environment seems to get
mounted inside the 1st and prevent cleanly unmounting it:
$ ps axfu
...
> jenkins 8544 0.0 0.0 18596 36 ? S Aug25 0:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -
> jenkins 8553 0.5 12.6 2535184 507312 ? Sl Aug25 16:11 \_ /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1
> jenkins 8804 0.0 0.0 13864 3584 ? S 14:18 0:00 \_ /bin/bash /usr/bin/build-and-provide-package
> root 10948 0.0 0.0 40592 3324 ? S 14:18 0:00 | \_ sudo DIST= ARCH=amd64 ADT=skip cowbuilder --buildresult /srv/jenkins/workspace/ucs-3.2-G00R05C02P1-master/in8-config-binaries/architecture/amd64/binaries/ --bui
> root 10949 0.0 0.0 4232 1308 ? S 14:18 0:00 | \_ cowbuilder --buildresult /srv/jenkins/workspace/ucs-3.2-G00R05C02P1-master/in8-config-binaries/architecture/amd64/binaries/ --build /srv/jenkins/workspace/u
> root 10973 0.0 0.0 13292 2556 ? S 14:18 0:00 | \_ /bin/bash /usr/sbin/pbuilder build --debbuildopts -sa --hookdir /usr/share/jenkins-debian-glue/pbuilder-hookdir/ --bindmounts /tmp/apt-8804 /var/cache/p
> root 10974 0.1 0.0 13708 3492 ? S 14:18 0:00 | \_ /bin/bash /usr/lib/pbuilder/pbuilder-buildpackage --debbuildopts -sa --hookdir /usr/share/jenkins-debian-glue/pbuilder-hookdir/ --bindmounts /tmp/ap
> root 12395 0.0 0.0 11800 2764 ? S 14:18 0:00 | \_ /bin/bash /usr/lib/pbuilder/pbuilder-satisfydepends --control /srv/jenkins/workspace/ucs-3.2-G00R05C02P1-master/in8-config-binaries/architecture
> root 12955 5.7 1.3 173324 54948 ? Sl 14:18 0:01 | \_ aptitude -y --without-recommends -o APT::Install-Recommends=false -o Aptitude::ProblemResolver::StepScore=100 -o Aptitude::ProblemResolver::
> root 12969 3.1 0.1 23936 7088 ? D 14:18 0:00 | \_ /usr/bin/dpkg --status-fd 113 --unpack --auto-deconfigure /var/cache/apt/archives/bsdmainutils_8.0.13.11.201104201320_amd64.deb /var/cac
> jenkins 9148 0.0 0.0 13864 3588 ? S 14:18 0:00 \_ /bin/bash /usr/bin/build-and-provide-package
> root 11323 0.0 0.0 40592 3440 ? S 14:18 0:00 \_ sudo DIST= ARCH=amd64 ADT=skip cowbuilder --buildresult /srv/jenkins/workspace/ucs-3.2-G00R05C02P1-master/in8-soap-transport-binaries/architecture/amd64/binarie
> root 11327 0.0 0.0 4232 1472 ? S 14:18 0:00 \_ cowbuilder --buildresult /srv/jenkins/workspace/ucs-3.2-G00R05C02P1-master/in8-soap-transport-binaries/architecture/amd64/binaries/ --build /srv/jenkins/wor
> root 11618 0.0 0.0 13296 2636 ? S 14:18 0:00 \_ /bin/bash /usr/sbin/pbuilder build --debbuildopts -sa --hookdir /usr/share/jenkins-debian-glue/pbuilder-hookdir/ --bindmounts /tmp/apt-9148 /var/cache/p
> root 11621 0.0 0.0 13712 3432 ? S 14:18 0:00 \_ /bin/bash /usr/lib/pbuilder/pbuilder-buildpackage --debbuildopts -sa --hookdir /usr/share/jenkins-debian-glue/pbuilder-hookdir/ --bindmounts /tmp/ap
> root 12873 0.0 0.0 11800 2692 ? S 14:18 0:00 \_ /bin/bash /usr/lib/pbuilder/pbuilder-satisfydepends --control /srv/jenkins/workspace/ucs-3.2-G00R05C02P1-master/in8-soap-transport-binaries/arch
> root 12960 6.1 1.3 115052 54724 ? Sl 14:18 0:01 \_ aptitude -y --without-recommends -o APT::Install-Recommends=false -o Aptitude::ProblemResolver::StepScore=100 -o Aptitude::ProblemResolver::
> root 12990 2.9 0.1 23988 7140 ? D 14:18 0:00 \_ /usr/bin/dpkg --status-fd 113 --unpack --auto-deconfigure /var/cache/apt/archives/bsdmainutils_8.0.13.11.201104201320_amd64.deb /var/cac
$ pidof cowbuilder
> 11327 10949
$ mount | grep --color /var/cache/pbuilder/build
> /proc on /var/cache/pbuilder/build/cow.10949/proc type proc (rw,relatime)
> tmpfs on /var/cache/pbuilder/build/cow.10949/run/shm type tmpfs (rw,relatime)
> /dev/pts on /var/cache/pbuilder/build/cow.10949/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> /dev/md0 on /var/cache/pbuilder/build/cow.10949/tmp/apt-8804 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/md0 on /var/cache/pbuilder/build/cow.10949/var/cache/pbuilder/build type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /proc on /var/cache/pbuilder/build/cow.11327/proc type proc (rw,relatime)
> /proc on /var/cache/pbuilder/build/cow.10949/var/cache/pbuilder/build/cow.11327/proc type proc (rw,relatime)
> tmpfs on /var/cache/pbuilder/build/cow.11327/run/shm type tmpfs (rw,relatime)
> tmpfs on /var/cache/pbuilder/build/cow.10949/var/cache/pbuilder/build/cow.11327/run/shm type tmpfs (rw,relatime)
> /dev/pts on /var/cache/pbuilder/build/cow.11327/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> /dev/pts on /var/cache/pbuilder/build/cow.10949/var/cache/pbuilder/build/cow.11327/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> /dev/md0 on /var/cache/pbuilder/build/cow.11327/tmp/apt-9148 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/md0 on /var/cache/pbuilder/build/cow.10949/var/cache/pbuilder/build/cow.11327/tmp/apt-9148 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/md0 on /var/cache/pbuilder/build/cow.11327/var/cache/pbuilder/build type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/md0 on /var/cache/pbuilder/build/cow.10949/var/cache/pbuilder/build/cow.11327/var/cache/pbuilder/build type ext4 (rw,relatime,errors=remount-ro,data=ordered)
In /usr/bin/build-and-provide-package:536 I see this:
> local BINDMOUNTS="/tmp/apt-$$ /var/cache/pbuilder/build ${USER_BINDMOUNTS:-}"
Why is it needed to mount BUILDPLACE recursively?
It was added by
<https://github.com/mika/jenkins-debian-glue/commit/5930014a1d894a0e2e89da3d9e5af897f96733c8>
Reading
<https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt>
I think the pbuilder mounts should use "mount --make-private".
I have this work-around added to my /etc/pbuilderrc:
> mount () {
> case "$1" in
> -obind) /bin/mount --make-private "$@" ;;
> *) /bin/mount "$@"
> esac
> }
This seems to work as I only have this mounted now:
> /proc on /var/cache/pbuilder/build/cow.20354/proc type proc (rw,relatime)
> tmpfs on /var/cache/pbuilder/build/cow.20354/run/shm type tmpfs (rw,relatime)
> /dev/pts on /var/cache/pbuilder/build/cow.20354/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> /dev/md0 on /var/cache/pbuilder/build/cow.20354/tmp/apt-18355 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/md0 on /var/cache/pbuilder/build/cow.20354/var/cache/pbuilder/build type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /proc on /var/cache/pbuilder/build/cow.20370/proc type proc (rw,relatime)
> tmpfs on /var/cache/pbuilder/build/cow.20370/run/shm type tmpfs (rw,relatime)
> /dev/pts on /var/cache/pbuilder/build/cow.20370/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> /dev/md0 on /var/cache/pbuilder/build/cow.20370/tmp/apt-19482 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> /dev/md0 on /var/cache/pbuilder/build/cow.20370/var/cache/pbuilder/build type ext4 (rw,relatime,errors=remount-ro,data=ordered)
Thanks for jenkins-debian-glue.
Philipp <mailto:pmhahn at debian.org>
PS: I haven't yet filed a bug against pbuilder @ Debian.
--
Philipp Hahn
Open Source Software Engineer
Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
hahn at univention.de
http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
More information about the Pbuilder-maint
mailing list