[buildd-tools-devel] Bug#708844: buildd-mail: cannot move packages > 50Mb correctly to upload directory

[Michael Cree]

Package: buildd
Version: 0.63.2-1.1
Severity: normal

Dear Maintainer,

buildd-mail fails to correctly move packages larger than 50Mb to the upload
directory.  It chomps such packages to exactly 50Mb thus corrupting the build
because of data loss.

As an example here is the build directory before signing the changes files:

buildd at imago:~$ ls -lFS build|head
total 130984
-rw-rw-r-- 1 buildd buildd 63001774 May 19 03:41 ghc_7.4.1-4+b1_alpha.deb
-rw-rw-r-- 1 buildd buildd 55189872 May 19 03:50 ghc-prof_7.4.1-4+b1_alpha.deb
-rw-rw-r-- 1 buildd buildd 13422968 May 19 03:51 ghc-haddock_7.4.1-4+b1_alpha.deb
-rw-rw-r-- 1 buildd buildd  1486780 May 19 04:16 dssp-dbg_2.1.0-1_alpha.deb
-rw-rw-r-- 1 buildd buildd   582434 May 16 21:26 gcal_3.6.3-1_alpha.deb
-rw-rw-r-- 1 buildd buildd   221738 May 19 04:16 dssp_2.1.0-1_alpha.deb
-rw-rw-r-- 1 buildd buildd    60728 May 19 04:09 libgtkglextmm-x11-1.2-dev_1.2.0-6_alpha.deb
-rw-rw-r-- 1 buildd buildd    44088 May 19 03:55 librelp-dev_1.0.7-1_alpha.deb
-rw-rw-r-- 1 buildd buildd    38800 May 19 04:09 libgtkglextmm-x11-1.2-0_1.2.0-6_alpha.deb

There is plenty of room on the disks (20GB) for shifting packages to upload.
I sign the received changes file by mail and then see in daemon.log:

May 19 08:26:38 buildd-mail[2354]: Log reply from [censored_email_address]
database is deprecated, please use 'arch' instead.
cp: writing '/home/buildd/upload/ghc_7.4.1-4+b1_alpha.deb': File too large
cp: failed to extend '/home/buildd/upload/ghc_7.4.1-4+b1_alpha.deb': File too large
May 19 08:26:43 buildd-mail[2354]: Cannot copy ghc_7.4.1-4+b1_alpha.deb to /home/buildd/upload/
cp: writing '/home/buildd/upload/ghc-prof_7.4.1-4+b1_alpha.deb': File too large
cp: failed to extend '/home/buildd/upload/ghc-prof_7.4.1-4+b1_alpha.deb': File too large
May 19 08:26:44 buildd-mail[2354]: Cannot copy ghc-prof_7.4.1-4+b1_alpha.deb to /home/buildd/upload/
May 19 08:26:44 buildd-mail[2354]: Moved ghc_7.4.1-4+b1 to upload
May 19 08:26:44 buildd-mail[2354]: Error: Could not move all files to upload dir.

And receive the following mail message in reply to the signed changes file:

To: [censored_email_address]
Subject: Re: Log for successful build of ghc_7.4.1-4+b1 on alpha (dist=unstable)

Your mail could not be processed:
Could not move all files to upload dir.
[rest of message includes signed changes file.]


In the upload directory is the following:

buildd at imago:~$ ls -lFS upload | head
total 112640
-rw------- 1 buildd buildd 50000000 May 19 08:26 ghc_7.4.1-4+b1_alpha.deb
-rw------- 1 buildd buildd 50000000 May 19 08:26 ghc-prof_7.4.1-4+b1_alpha.deb
-rw------- 1 buildd buildd 13422968 May 19 08:26 ghc-haddock_7.4.1-4+b1_alpha.deb
-rw------- 1 buildd buildd  1486780 May 19 08:23 dssp-dbg_2.1.0-1_alpha.deb
-rw------- 1 buildd buildd   221738 May 19 08:23 dssp_2.1.0-1_alpha.deb
-rw------- 1 buildd buildd    60728 May 19 08:24 libgtkglextmm-x11-1.2-dev_1.2.0-6_alpha.deb
-rw------- 1 buildd buildd    44088 May 19 08:23 librelp-dev_1.0.7-1_alpha.deb
-rw------- 1 buildd buildd    38800 May 19 08:24 libgtkglextmm-x11-1.2-0_1.2.0-6_alpha.deb
-rw------- 1 buildd buildd    27222 May 19 08:23 librelp0_1.0.7-1_alpha.deb

And the upload fails because of corrupted package files.

Cheers
Michael Cree.

-- System Information:
Debian Release: 7.0
  APT prefers unreleased
  APT policy: (500, 'unreleased'), (500, 'unstable')
Architecture: alpha

Kernel: Linux 3.8.11-titan-p+ (SMP w/3 CPU cores)
Locale: LANG=en_NZ.UTF-8, LC_CTYPE=en_NZ.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages buildd depends on:
ii  adduser                         3.113+nmu3
ii  cron                            3.0pl1-124
ii  dupload                         2.7.0
ii  libsbuild-perl                  0.63.2-1.1
ii  libyaml-tiny-perl               1.51-2
ii  perl                            5.14.2-21
ii  postfix [mail-transport-agent]  2.10.0-3
ii  sbuild                          0.63.2-1.1

buildd recommends no packages.

Versions of packages buildd suggests:
pn  wanna-build  <none>

-- Configuration Files:
/etc/cron.d/buildd changed:
5,20,35,50   *            *   *   *    buildd   /usr/bin/buildd-watcher

/etc/schroot/setup.d/25nssdatabases changed:
set -e
if [ -f "/etc/schroot/setup.d/20nssdatabases" ]; then
    # We're dealing with a schroot from squeeze.  20nssdatabases will
    # handle everything for us.  This script is only needed with
    # schroot from lenny.  So exit now.
    exit 0
fi
if [ -f "$CHROOT_SCRIPT_CONFIG" ]; then
    . "$CHROOT_SCRIPT_CONFIG"
elif [ "$2" = "ok" ]; then
    echo "script-config file '$CHROOT_SCRIPT_CONFIG' does not exist"
    exit 1
fi
if [ "$AUTH_VERBOSITY" = "verbose" ]; then
  VERBOSE="--verbose"
fi
dup_nss()
{
    if [ "$AUTH_VERBOSITY" = "verbose" ]; then
	echo "Copying $1 database to $2"
    fi
    getent "$1" > "$2"
}
if [ $1 = "setup-start" ] || [ $1 = "setup-recover" ]; then
    if [ -n "$NSSDATABASES" ]; then
	if [ -f "$NSSDATABASES" ]; then
	    while read db; do
		if echo "$db" | egrep -q '^(#|$)' ; then
                    continue
		fi
                # Device and inode
		dr=$(/usr/bin/stat --format="%d %i" "/etc/$db")
                # This one can fail since it might not exist yet
		dc=$(/usr/bin/stat --format="%d %i" "${CHROOT_PATH}/etc/$db" 2>/dev/null || :)
                # If the database inside and outside the chroot is the
                # same, it's very likely that dup_nss would blank the
                # database, so skip it.
		if [ "$dr" = "$dc" ]; then
		    echo "warning: $db files '/etc/$db' and '${CHROOT_PATH}/etc/$db' are the same file; skipping" 1>&2
                    continue;
		fi
		dup_nss "$db" "${CHROOT_PATH}/etc/$db"
	    done < "$NSSDATABASES"
	else
	    echo "nssdatabases file '$NSSDATABASES' does not exist"
	    exit 1
	fi
    fi
fi


-- no debconf information