[Debian-ports-devel] [PATCH] Add automatic by-hand processing of d-i's raw-installer images

[Aurelien Jarno]

Hi,

On 2017-02-11 21:56, James Clarke wrote:
> ---
> 
> Hi,
> Since debian-installer has now been fixed for (some) ports, the next
> upload will see the buildds uploading installer images. Currently
> mini-dak does not process these; this patch intends to function
> similarly to dak's auto by-hand processing of d-i images. The
> archive-byhand-di was taken from scripts/debian/byhand-di and slightly
> modified to use mini-dak's config.
> 
> I have tested this with a local mini-dak setup and it seems to work,
> unpacking to the installer-$arch directory. However, I expect there are
> things I have missed, and this may not be implemented in the way you
> would like. Comments?

Thanks for the patch and sorry for the delay. I haven't tried the
patch, but after review it seems overall good. My main concern is that
we should try to catch errors earlier than in the queue_accepted()
function. See below for more detailed comments.


> diff --git a/bin/archive-byhand-di b/bin/archive-byhand-di
> new file mode 100755
> index 0000000..6443a4d
> --- /dev/null
> +++ b/bin/archive-byhand-di
> @@ -0,0 +1,111 @@
> +#!/bin/bash
> +
> +. archive-lib
> +
> +set -u
> +set -e
> +set -o pipefail
> +
> +if [ $# -lt 5 ]; then
> +	echo "Usage: $0 filename version arch changes_file suite"
> +	exit 1
> +fi
> +
> +TARBALL="$1"	# Tarball to read, compressed with gzip
> +VERSION="$2"
> +ARCH="$3"
> +CHANGES="$4"	# Changes file for the upload
> +SUITE="$5"
> +
> +error() {
> +	echo "$*"
> +	exit 1
> +}

I think this should be converted to use the log function, so that the
reason for the failure is logged. It won't be propagated to the reject
mail, but at least it will be available somewhere.

[...]

> diff --git a/bin/archive-queue b/bin/archive-queue
> index 1268e68..f5eff0d 100755
> --- a/bin/archive-queue
> +++ b/bin/archive-queue

[...]

> @@ -102,7 +108,7 @@ verify_gpg_signature ()
>  verify_dpkg_signature ()
>  {
>    local archive_file=$1
> -  local files=`fetch_secure_files < $archive_file | grep '\.u\?deb$'`
> +  local files=`fetch_secure_files < $archive_file | grep '\.u\?deb$|\.tar\.gz$'`

This won't work, dpkg-sig can't verify .tar.gz file (ok in practice
dpkg-sig is not used, but well let's not break it).

[...]

> +queue_process_autobyhand_d_i_images ()
> +{
> +  local changes_file=$1
> +  local archive_file=$2
> +  # Get full file path, since archive-byhand-di runs with CWD as bin_dir
> +  local file=$(readlink -f $3)
> +  local version=`fetch_field "Version" < $archive_file`
> +  local target_arches=`strip_gpg < $changes_file | fetch_field "Architecture"`
> +  local arch=`filter_real_arches $target_arches`
> +  local suite=`fetch_field "Distribution" < $archive_file`
> +
> +  if test `echo $arch | wc -w` -ne 1 ; then
> +    log queue "queue_process_autobyhand_d_i_images ${archive_file##*/} $file changes has arches != 1: $arch"
> +    return 1
> +  fi
> +
> +  if ! (cd $bin_dir && ./archive-byhand-di $file $version $arch $changes_file $suite); then
> +    return 1
> +  fi
> +

I understand that you would like to diverge as few as possible from the
original dak version of byhand-di, but what would be the cost of not
requiring changing CWD? I would prefer if possible that all scripts are
called consistently.

> @@ -212,6 +292,18 @@ queue_accepted ()
>    local files_install="$files $changes_file $archive_file"
>    local suite=`fetch_field "Distribution" < $archive_file`
>  
> +  for file in $files; do
> +    if queue_is_autobyhand $changes_file $archive_file $file; then
> +      if queue_process_autobyhand $changes_file $archive_file $file; then
> +        log queue "queue_process_autobyhand_success ${archive_file##*/} $file"
> +      else
> +        script_error "queue_accepted" "$?"
> +        log queue "queue_process_autobyhand_failed ${archive_file##*/} $file"
> +        return 1
> +      fi
> +    fi
> +  done
> +
>    files_owner_perms $files_install
>  
>    if mv $files_install $accepted_dir; then

We should probably check for possible errors earlier than in
queue_accepted(), at least for the ones which can occur due to issues in
the incoming files. Otherwise it means that no ACCEPTED or REJECTED mail
is sent, and that the files stay in the uncheck directory.

Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien at aurel32.net                 http://www.aurel32.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/debian-ports-devel/attachments/20170411/31e3666a/attachment.sig>