Bug#858669: cp: cannot create hard link [...]: Invalid cross-device link + rm: cannot remove [...]/dev/ptmx': Device or resource busy

[Michael Prokop]

Package: pbuilder
Version: 0.228.6~bpo8+1
Severity: important

Hi,

since the upgrade to pbuilder v0.228.6~bpo8+1 I noticed problems
related to the build environment's ptmx mount point, it looks like a
regression of #841935, at least for certain environments.

Problem #1
----------

I get the following error during builds on a
Debian/jessie system (running kernel 3.16.0-4-amd64):

| + sudo DIST= ARCH=amd64 cowbuilder --update --basepath /var/cache/pbuilder/base-sid-amd64.cow --configfile=/tmp/tmp.3HyEOBo1jh
|  -> Copying COW directory
|   forking: rm -rf /var/cache/pbuilder/build/cow.29599
|   forking: cp -al /srv/pbuilder/base-sid-amd64.cow /var/cache/pbuilder/build/cow.29599
| cp: cannot create hard link ‘/var/cache/pbuilder/build/cow.29599/dev/ptmx’ to ‘/srv/pbuilder/base-sid-amd64.cow/dev/ptmx’: Invalid cross-device link
| Failed cowcopy.

This is caused by the /dev/ptmx mountpoint being left behind:

| % mount
| [...]
| devpts on /srv/pbuilder/base-sid-amd64.cow/dev/ptmx type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)

Here /var/cache/pbuilder is a symlink to /srv/pbuilder:

| % ls -la /var/cache/pbuilder
| lrwxrwxrwx 1 root root 13 May  7  2012 /var/cache/pbuilder -> /srv/pbuilder

The same problem also exists if /var/cache/pbuilder is residing as a
whole inside a tmpfs mountpoint.

When I'm downgrading to pbuilder 0.228.5~bpo8+1 (or 0.215+nmu3 from
jessie) the problem disappears.

Problem #2
----------

On another Debian/jessie system (running kernel 4.8.0-0.bpo.2-amd64)
I get:

| % gbp buildpackage --git-pbuilder
| gbp:info: Creating zsh_5.3.1.orig.tar.gz from 'zsh-5.3.1'
| gbp:info: Building with (cowbuilder) for sid
| Building with cowbuilder for distribution sid
| + pdebuild --buildresult ../ --pbuilder cowbuilder --debbuildopts '' -- --basepath /var/cache/pbuilder/base.cow
| [...]
| I: Copying COW directory
| I: forking: rm -rf /var/cache/pbuilder/build/cow.10985
| I: forking: cp -al /data/pbuilder/base-sid-amd64.cow /var/cache/pbuilder/build/cow.10985
| I: removed stale ilistfile /var/cache/pbuilder/build/cow.10985/.ilist
| I: forking: chroot /var/cache/pbuilder/build/cow.10985 cowdancer-ilistcreate /.ilist 'find . -xdev -path ./home -prune -o \( \( -type l -o -type f \) -a -links +1 -print0 \) | xargs -0 stat --format '%d %i ''
| I: Invoking pbuilder
| I: forking: pbuilder build --debbuildopts  --debbuildopts  --buildplace /var/cache/pbuilder/build/cow.10985 --buildresult /home/grml/git-debian --extrapackages eatmydata --no-targz --internal-chrootexec 'chroot /var/cache/pbuilder/build/cow.10985 cow-shell' /home/grml/git-debian/zsh_5.3.1-3.dsc
| [...]
| I: mounting /dev/pts filesystem
| I: redirecting /dev/ptmx to /dev/pts/ptmx
| I: policy-rc.d already exists
| [...]
| I: forking: rm -rf /var/cache/pbuilder/build/cow.10985
| rm: cannot remove '/var/cache/pbuilder/build/cow.10985/dev/ptmx': Device or resource busy

And here's a symlink for /var/cache/pbuilder present as well:

| %  ls -la /var/cache/pbuilder
| lrwxrwxrwx 1 root root 14 Dec  2  2013 /var/cache/pbuilder -> /data/pbuilder

Downgrading to pbuilder 0.228.5~bpo8+1 also helps here.

Approach #1
-----------

When switching from the symlink /var/cache/pbuilder to a bind mount
instead (`mount --bind /data/pbuilder /var/cache/pbuilder`) it still
fails, this time with a more concrete error message at least though:

| E: Something (/var/cache/pbuilder/build/cow.24056/dev/ptmx) is still mounted under /var/cache/pbuilder/build/cow.24056; unmount and remove /var/cache/pbuilder/build/cow.24056 manually

The /dev/ptmx mountpoint is indeed left behind:

| % grep pbuilder /proc/mounts
| /dev/mapper/x230-data /var/cache/pbuilder ext4 rw,relatime,discard,data=ordered 0 0
| devpts /var/cache/pbuilder/build/cow.24056/dev/ptmx devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
| devpts /data/pbuilder/build/cow.24056/dev/ptmx devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0

So this doesn't really solve the issue.

Approach #2
-----------

Updating the debootstrap version and rebuilding the base.cow.
When switching from jessie's debootstrap 1.0.67 to the debootstrap
version from jessie-backports (1.0.86~bpo8+1) and *rebuilding* the
build environment the issue disappears as well.

Instead of the character device for /dev/ptmx:

  crw-rw-rw- 1 root tty 5, 2 Dec 30  2013 /var/cache/pbuilder/base-jessie-amd64.cow/dev/ptmx

I now get the symlink instead:

  lrwxrwxrwx 1 root root 8 Mar 25 00:37 /var/cache/pbuilder/base-jessie-amd64.cow/dev/ptmx -> pts/ptmx

... and the issue no longer exists.
As a starting point pbuilder should at least depend on
deboostrap (>=1.0.86~bpo8+1~) then?

I'm afraid this will affect more users with existing chroots once
they upgrade to the pbuilder from stretch (I noticed and reproduced
this issue on three different environments already).

regards,
-mika-