[devscripts] 01/01: debpatch: New script, "Apply a debdiff to a Debian source package"
Ximin Luo
infinity0 at debian.org
Fri Nov 25 17:33:54 UTC 2016
This is an automated email from the git hooks/post-receive script.
infinity0 pushed a commit to branch pu/debpatch
in repository devscripts.
commit a8237eb963cb7b0d6e0e818ad70589765ceb93ba
Author: Ximin Luo <infinity0 at debian.org>
Date: Fri Nov 25 18:33:34 2016 +0100
debpatch: New script, "Apply a debdiff to a Debian source package"
---
README | 7 ++
conf.default.in | 4 +
debian/changelog | 4 +
debian/control | 4 +
debian/copyright | 16 +++
po4a/devscripts-po4a.conf | 2 +
scripts/Makefile | 2 +-
scripts/debdiff.1 | 1 +
scripts/debpatch | 311 ++++++++++++++++++++++++++++++++++++++++++++++
scripts/debpatch.1 | 108 ++++++++++++++++
10 files changed, 458 insertions(+), 1 deletion(-)
diff --git a/README b/README
index 575d3d4..8c9ca11 100644
--- a/README
+++ b/README
@@ -97,6 +97,13 @@ And now, in mostly alphabetical order, the scripts:
ability to install the package with a very short command is very
useful when troubleshooting packages.
+- debpatch: Apply unified diffs of two Debian source packages, such as those
+ generated by debdiff, to a target Debian source package. Any changes to
+ debian/changelog are dealt with specially, to avoid the conflicts that
+ changelog diffs typically produce when applied naively. May be used to check
+ that old patches still apply to newer versions of those packages.
+ [python3-unidiff, quilt]
+
- debpkg: A wrapper for dpkg used by debi to allow convenient testing
of packages. For debpkg to work, it needs to be made setuid root,
and this needs to be performed by the sysadmin -- it is not
diff --git a/conf.default.in b/conf.default.in
index e043136..48633dd 100644
--- a/conf.default.in
+++ b/conf.default.in
@@ -280,6 +280,10 @@
# debc recognises the DEBRELEASE_DEBS_DIR variable; see debrelease
# below for more information.
+##### debpatch
+#
+# No variables currently
+
##### debpkg
#
# No variables currently
diff --git a/debian/changelog b/debian/changelog
index ae2b897..2fa6913 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -16,6 +16,10 @@ devscripts (2.16.10) UNRELEASED; urgency=medium
+ Require a '--' between debuild options and debian/rules target so we
know where options end. (Closes: #845566)
+ [ Ximin Luo ]
+ * debpatch:
+ + New script, Apply a debdiff to a Debian source package. (Closes: #845659)
+
-- Afif Elghraoui <afif at debian.org> Wed, 23 Nov 2016 23:50:46 -0800
devscripts (2.16.9) unstable; urgency=medium
diff --git a/debian/control b/debian/control
index 9c64a0b..523a3e8 100644
--- a/debian/control
+++ b/debian/control
@@ -66,6 +66,7 @@ Recommends: apt,
patchutils,
python3-debian (>= 0.1.15),
python3-magic,
+ python3-unidiff,
sensible-utils,
strace,
unzip,
@@ -98,6 +99,7 @@ Suggests: adequate,
mozilla-devscripts,
mutt,
piuparts,
+ quilt,
ratt,
reprotest,
ssh-client,
@@ -149,6 +151,8 @@ Description: scripts to make the life of a Debian Package maintainer easier
added and removed files. Use the diffoscope package for deep comparisons.
[wdiff, patchutils]
- debi: install a just-built package
+ - debpatch: apply unified diffs of two Debian source packages, such as those
+ generated by debdiff, to a target source package [python3-unidiff, quilt]
- debpkg: dpkg wrapper to be able to manage/test packages without su
- debrepro: reproducibility tester for Debian packages [faketime,
diffoscope, disorderfs]
diff --git a/debian/copyright b/debian/copyright
index a4a78e5..ea20cfe 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -81,6 +81,22 @@ License: GPL-3+
On Debian systems, the complete text of the GNU General Public License
version 3 can be found in the /usr/share/common-licenses/GPL-3 file.
+Files: scripts/debpatch*
+Copyright: 2016 Ximin Luo <infinity0 at debian.org>
+License: GPL-3+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 3 of the License, or
+ (at your option) any later version.
+ .
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ General Public License for more details.
+ .
+ On Debian systems, the complete text of the GNU General Public License
+ version 3 can be found in the /usr/share/common-licenses/GPL-3 file.
+
Files: doc/suspicious-source.1
doc/wrap-and-sort.1
scripts/devscripts/*
diff --git a/po4a/devscripts-po4a.conf b/po4a/devscripts-po4a.conf
index 7061e06..56629f1 100644
--- a/po4a/devscripts-po4a.conf
+++ b/po4a/devscripts-po4a.conf
@@ -44,6 +44,8 @@
$lang:$lang/debdiff.$lang.1 add_$lang:?add_$lang/translator_man.add
[type:man] ../scripts/debi.1 \
$lang:$lang/debi.$lang.1 add_$lang:?add_$lang/translator_man.add
+[type:man] ../scripts/debpatch.1 \
+ $lang:$lang/debpatch.$lang.1 add_$lang:?add_$lang/translator_man.add
[type:man] ../scripts/debpkg.1 \
$lang:$lang/debpkg.$lang.1 add_$lang:?add_$lang/translator_man.add
[type:pod] ../scripts/debrepro.pod \
diff --git a/scripts/Makefile b/scripts/Makefile
index 931143a..7c4494e 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -23,7 +23,7 @@ BC_BUILD_DIR:=bash_completion
COMPLETION = $(patsubst %.bash_completion,$(BC_BUILD_DIR)/%,$(COMPL_FILES))
COMPL_DIR := $(shell pkg-config --variable=completionsdir bash-completion)
PKGNAMES:=wnpp-alert wnpp-check mk-build-deps rmadison mass-bug debsnap dd-list build-rdeps who-uploads transition-check getbuildlog dcontrol grep-excuses rc-alert whodepends dget pts-subscribe pts-unsubscribe debcheckout
-PYTHON3_SCRIPTS:=sadt suspicious-source wrap-and-sort
+PYTHON3_SCRIPTS:=debpatch sadt suspicious-source wrap-and-sort
GEN_MAN1S += debrepro.1 devscripts.1 mk-origtargz.1 uscan.1
diff --git a/scripts/debdiff.1 b/scripts/debdiff.1
index bf3c77f..6ea0628 100644
--- a/scripts/debdiff.1
+++ b/scripts/debdiff.1
@@ -232,6 +232,7 @@ Normally the exit value will be 0 if no differences are reported and 1
if any are reported. If there is some fatal error, the exit code will
be 255.
.SH "SEE ALSO"
+.BR debpatch (1),
.BR diffstat (1),
.BR dpkg-deb (1),
.BR interdiff (1),
diff --git a/scripts/debpatch b/scripts/debpatch
new file mode 100755
index 0000000..fd861e3
--- /dev/null
+++ b/scripts/debpatch
@@ -0,0 +1,311 @@
+#!/usr/bin/python3
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# See file /usr/share/common-licenses/GPL-3 for more details.
+#
+"""
+Apply a debdiff to a Debian source package.
+
+It handles d/changelog hunks specially, to avoid conflicts.
+
+Depends on dpkg-dev, devscripts, python3-unidiff, quilt.
+"""
+
+import argparse
+import hashlib
+import io
+import logging
+import os
+import random
+import unidiff
+import shutil
+import subprocess
+import sys
+import tempfile
+
+dirname = os.path.dirname
+basename = os.path.basename
+C = subprocess.check_call
+
+# this can be any valid value, it doesn't appear in the final output
+DCH_DUMMY_TAIL = "\n -- debpatch dummy tool <infinity0 at debian.org> Thu, 01 Jan 1970 00:00:00 +0000\n\n"
+TRY_ENCODINGS = ["utf-8", "latin-1"]
+DISTRIBUTION_DEFAULT = "experimental"
+
+def parse_dch(dch_str, *args):
+ return subprocess.run(
+ ["dpkg-parsechangelog", "-l-", "-c1"] + list(args),
+ input=dch_str,
+ check=True,
+ universal_newlines=True,
+ stdout=subprocess.PIPE,
+ ).stdout.rstrip()
+
+def read_dch(dch_str):
+ dch = {}
+ for i in ("Version", "Distribution", "Urgency", "Maintainer"):
+ dch[i] = parse_dch(dch_str, "-S"+i)
+ dch["Changes"] = "".join(parse_dch(dch_str, "-SChanges").splitlines(True)[3:])
+ return dch
+
+def is_dch(path):
+ return (basename(path) == 'changelog'
+ and basename(dirname(path)) == 'debian'
+ and dirname(dirname(dirname(path))) == '')
+
+def hunk_lines_to_str(hunk_lines):
+ return "".join(map(lambda x: str(x)[1:], hunk_lines))
+
+def read_dch_patch(dch_patch):
+ if len(dch_patch) > 1:
+ raise ValueError("don't know how to deal with d/changelog patch that has more than one hunk")
+ hunk = dch_patch[0]
+ source_str = hunk_lines_to_str(hunk.source_lines()) + DCH_DUMMY_TAIL
+ target_str = hunk_lines_to_str(hunk.target_lines())
+ # here we assume the debdiff has enough context to see the previous version
+ # this should be true all the time in practice
+ source_version = parse_dch(source_str, "-SVersion")
+ target = read_dch(target_str)
+ return source_version, target
+
+def apply_dch_patch(source_file, current, patch_name, old_version, target, dry_run):
+ # Do not change this text, unless you also add logic to detect markers from
+ # previously-released versions.
+ marker = "Patch %s applied by debpatch(1)." % patch_name
+ if marker in current["Changes"]:
+ logging.info("patch %s already applied to d/changelog", patch_name)
+ return target["Version"]
+
+ dch_args = []
+ dch_env = dict(os.environ)
+
+ if target["Distribution"] == "UNRELEASED":
+ # UNRELEASED causes hard-to-reason-about behaviours in dch, let's avoid that
+ newdist = current["Distribution"] if current["Distribution"] != "UNRELEASED" else DISTRIBUTION_DEFAULT
+ logging.info("using distribution '%s' instead of 'UNRELEASED'", newdist)
+ target["Distribution"] = newdist
+
+ if not old_version or not target["Version"].startswith(old_version):
+ logging.warn("don't know how to reapply version-change %s to %s" %
+ (old_version, target["Version"]))
+ version = subprocess.check_output(["sh", "-c",
+ "EDITOR=cat dch -n 2>/dev/null | dpkg-parsechangelog -l- -SVersion"
+ ]).decode("utf-8").rstrip()
+ logging.warn("using version %s based on `dch -n`; feel free to make me smarter", version)
+ else:
+ version_suffix = target["Version"][len(old_version):]
+ version = current["Version"] + version_suffix
+ logging.info("using version %s based on suffix %s", version, version_suffix)
+
+ if dry_run:
+ return version
+
+ dch_args += ["-v", version]
+ dch_args += ["--force-distribution", "-D", target["Distribution"]]
+ dch_args += ["-u", target["Urgency"]]
+ if "Maintainer" in target:
+ dch_env["DEBEMAIL"] = target["Maintainer"]
+ del dch_env["DEBFULLNAME"]
+
+ changes = target["Changes"]
+ if changes.lstrip().startswith("["):
+ changes = "\n" + changes
+
+ token = "DEBPATCH PLACEHOLDER %s DELETEME" % random.randint(0, 2**64)
+ shutil.copy(source_file, source_file + ".debpatch.bak")
+ try:
+ C(["dch", "-c", source_file] + dch_args + [token])
+ C(["dch", "-c", source_file, "-a", marker], )
+ C(["sed", "-e", "/%s/c\\\n%s" % (token, changes.replace("\n", "\\\n")), "-i", source_file])
+ except:
+ os.rename(source_file, source_file + ".debpatch.err")
+ logging.warn("failed to patch %s", source_file)
+ logging.warn("half-applied changes in %s", source_file + ".debpatch.err")
+ logging.warn("current working directory is %s", os.getcwd())
+ os.rename(source_file + ".debpatch.bak", source_file)
+ raise
+ else:
+ os.unlink(source_file + ".debpatch.bak")
+
+def call_patch(patch_str, *args, check=True, **kwargs):
+ return subprocess.run(
+ ["patch", "-p1"] + list(args),
+ input=patch_str,
+ universal_newlines=True,
+ check=check,
+ **kwargs)
+
+def check_patch(patch_str, *args, **kwargs):
+ return call_patch(patch_str,
+ "--dry-run", "-f", "--silent",
+ *args,
+ check=False,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ **kwargs).returncode == 0
+
+def apply_patch_str(patch_name, patch_str):
+ if check_patch(patch_str, "-N"):
+ call_patch(patch_str)
+ logging.info("patch %s applies!", patch_name)
+ elif check_patch(patch_str, "-R"):
+ logging.info("patch %s already applied", patch_name)
+ else:
+ call_patch(patch_str, "--dry-run")
+ raise ValueError("patch %s doesn't apply!", patch_name)
+
+def debpatch(patch, patch_name, args):
+ if len(patch_name) > 60:
+ # this messes with our dch "already applied" logic detection, sorry
+ raise ValueError("pick a shorter patch name; sorry")
+
+ # don't change anything if...
+ dry_run = args.target_version
+
+ changelog = list(filter(lambda x: is_dch(x.path), patch))
+ if not changelog:
+ logging.info("no debian/changelog in patch: %s" % args.patch_file)
+ old_version = None
+ target = {
+ "Version": None,
+ "Distribution": DISTRIBUTION_DEFAULT,
+ "Urgency": "low",
+ "Changes": " * Rebase patch %s." % patch_name,
+ }
+ elif len(changelog) > 1:
+ raise ValueError("more than one debian/changelog patch???")
+ else:
+ patch.remove(changelog[0])
+ old_version, target = read_dch_patch(changelog[0])
+
+ if args.source_version:
+ if old_version:
+ print(old_version)
+ return
+
+ if not dry_run:
+ apply_patch_str(patch_name, str(patch))
+
+ # only apply d/changelog patch if the rest of the patch applied
+ with open(args.changelog) as fp:
+ current = read_dch(fp.read())
+ new_version = apply_dch_patch(args.changelog, current, patch_name, old_version, target, dry_run)
+ if args.target_version:
+ print(new_version)
+ return
+
+ if args.repl:
+ import code
+ code.interact(local=locals())
+
+def main(args):
+ parser = argparse.ArgumentParser(
+ description='Apply a debdiff to a Debian source package')
+ parser.add_argument('-v', '--verbose', action="store_true",
+ help='Output more information')
+ parser.add_argument('-c', '--changelog', default='debian/changelog',
+ help='Path to debian/changelog; default: %(default)s')
+ parser.add_argument('--repl', action="store_true",
+ help="Run the python REPL after processing.")
+ parser.add_argument('--source-version', action="store_true",
+ help="Don't apply the patch; instead print out the version of the "
+ "package that it is supposed to be applied to, or nothing if the patch "
+ "does not specify a source version.")
+ parser.add_argument('--target-version', action="store_true",
+ help="Don't apply the patch; instead print out the new version of the "
+ "package debpatch(1) would generate, when the patch is applied to the "
+ "the given target package, as specified by the other arguments.")
+ parser.add_argument('orig_dsc_or_dir', nargs='?', default=".",
+ help="Target to apply the patch to. This can either be an unpacked "
+ "source tree, or a .dsc file. In the former case, the directory is "
+ "modified in-place; in the latter case, a second .dsc is created. "
+ "Default: %(default)s")
+ parser.add_argument('patch_file', nargs='?', default="/dev/stdin",
+ help="Patch file to apply, in the format output by debdiff(1). "
+ "Default: %(default)s")
+ group1 = parser.add_argument_group('Options for .dsc patch targets')
+ group1.add_argument('--no-clean', action="store_true",
+ help="Don't clean temporary directories after a failure, so you can "
+ "examine what failed.")
+ group1.add_argument('--quilt-refresh', action="store_true",
+ help="If the building of the new source package fails, try to refresh "
+ "patches using quilt(1) then try building it again.")
+ group1.add_argument('-d', '--directory', default=None,
+ help="Extract the .dsc into this directory, which won't be cleaned up "
+ "after debpatch(1) exits. If not given, then it will be extracted to a "
+ "temporary directory.")
+ args = parser.parse_args(args)
+ #print(args)
+
+ if args.verbose:
+ logging.getLogger().setLevel(logging.DEBUG)
+
+ with open(args.patch_file, 'rb') as fp:
+ data = fp.read()
+ for enc in TRY_ENCODINGS:
+ try:
+ patch = unidiff.PatchSet(data.splitlines(keepends=True), encoding=enc)
+ break
+ except:
+ if enc == TRY_ENCODINGS[-1]:
+ raise
+ else:
+ continue
+
+ patch_name = '%s:%s' % (
+ basename(args.patch_file),
+ hashlib.sha256(data).hexdigest()[:20 if args.patch_file == '/dev/stdin' else 8])
+ quiet = args.source_version or args.target_version
+ dry_run = args.source_version or args.target_version
+ stdout = subprocess.DEVNULL if quiet else None # user can redirect stderr themselves
+
+ # change directory before applying patches
+ if os.path.isdir(args.orig_dsc_or_dir):
+ os.chdir(args.orig_dsc_or_dir)
+ debpatch(patch, patch_name, args)
+ elif os.path.isfile(args.orig_dsc_or_dir):
+ parts = os.path.splitext(os.path.basename(args.orig_dsc_or_dir))
+ if parts[1] != ".dsc":
+ raise ValueError("unrecognised patch target: %s" % args.orig_dsc_or_dir)
+ extractdir = args.directory if args.directory else tempfile.mkdtemp()
+ if not os.path.isdir(extractdir):
+ os.makedirs(extractdir)
+ try:
+ builddir = os.path.join(extractdir, parts[0]) # dpkg-source doesn't like existing dirs
+ C(["dpkg-source", "-x", "--skip-patches", args.orig_dsc_or_dir, builddir], stdout=stdout)
+ origdir = os.getcwd()
+ os.chdir(builddir)
+ debpatch(patch, patch_name, args)
+ if dry_run:
+ return
+ os.chdir(origdir)
+ try:
+ C(["dpkg-source", "-b", builddir])
+ except subprocess.CalledProcessError:
+ if args.quilt_refresh:
+ C(["sh", "-c", """
+set -ex
+export QUILT_PATCHES=debian/patches
+while quilt push; do quilt refresh; done
+"""
+ ], cwd=builddir)
+ C(["dpkg-source", "-b", builddir])
+ else:
+ raise
+ finally:
+ cleandir = builddir if args.directory else extractdir
+ if args.no_clean:
+ logging.warn("you should clean up temp files in %s", cleandir)
+ else:
+ shutil.rmtree(cleandir)
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/scripts/debpatch.1 b/scripts/debpatch.1
new file mode 100644
index 0000000..8af6fb7
--- /dev/null
+++ b/scripts/debpatch.1
@@ -0,0 +1,108 @@
+.\" Copyright (c) 2016, Ximin Luo <infinity0 at debian.org>
+.\"
+.\" This program is free software; you can redistribute it and/or
+.\" modify it under the terms of the GNU General Public License
+.\" as published by the Free Software Foundation; either version 3
+.\" of the License, or (at your option) any later version.
+.\"
+.\" This program is distributed in the hope that it will be useful,
+.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
+.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+.\" GNU General Public License for more details.
+.\"
+.\" See file /usr/share/common-licenses/GPL-3 for more details.
+.\"
+.TH DEBPATCH 1 "Debian Utilities" "DEBIAN"
+
+.SH NAME
+debpatch \- apply a debdiff to a Debian source package
+
+.SH SYNOPSIS
+.B debpatch
+[options] [orig_dsc_or_dir] [patch_file]
+.br
+.B debpatch
+[options] < [patch_file]
+
+.SH DESCRIPTION
+.B debpatch
+takes a \fIpatchfile\fR that describes the differences between two Debian
+source packages \fIold\fR and \fInew\fR, and applies it to a target Debian
+source package \fIorig\fR.
+.PP
+\fIorig\fR could either be the same as \fIold\fR or it could be different.
+\fIpatchfile\fR is expected to be a unified diff between two Debian source
+trees, as what
+.BR debdiff (1)
+normally generates.
+.PP
+Any changes to \fIdebian/changelog\fR are dealt with specially, to avoid the
+conflicts that changelog diffs typically produce when applied naively. The
+exact behaviour may be tweaked in the future, so one should not rely on it.
+.PP
+If \fIpatchfile\fR does not apply to \fIorig\fR, even after the special-casing
+of \fIdebian/changelog\fR, no changes are made and
+.BR debpatch (1)
+will exit with a non-zero error code.
+
+.SH ARGUMENTS
+.TP
+orig_dsc_or_dir
+Target to apply the patch to. This can either be an unpacked source tree, or a
+\[char46]dsc file. In the former case, the directory is modified in\-place; in
+the latter case, a second .dsc is created. Default: \fI.\fP
+.TP
+patch_file
+Patch file to apply, in the format output by
+.BR debdiff (1).
+Default:
+\fI\,/dev/stdin\/\fP
+
+.SH OPTIONS
+.TP
+\fB\-h\fR, \fB\-\-help\fR
+show this help message and exit
+.TP
+\fB\-v\fR, \fB\-\-verbose\fR
+Output more information
+.TP
+\fB\-c\fR CHANGELOG, \fB\-\-changelog\fR CHANGELOG
+Path to debian/changelog; default: debian/changelog
+.TP
+\fB\-\-repl\fR
+Run the python REPL after processing.
+.TP
+\fB\-\-source\-version\fR
+Don't apply the patch; instead print out the version of the package that it is
+supposed to be applied to, or nothing if the patch does not specify a source
+version.
+.TP
+\fB\-\-target\-version\fR
+Don't apply the patch; instead print out the new version of the package
+.BR debpatch (1)
+would generate, when the patch is applied to the the given target
+package, as specified by the other arguments.
+.SS "For .dsc patch targets:"
+.TP
+\fB\-\-no\-clean\fR
+Don't clean temporary directories after a failure, so you can examine what
+failed.
+.TP
+\fB\-\-quilt\-refresh\fR
+If the building of the new source package fails, try to refresh patches using
+.BR quilt (1)
+then try building it again.
+.TP
+\fB\-d\fR DIRECTORY, \fB\-\-directory\fR DIRECTORY
+Extract the .dsc into this directory, which won't be cleaned up after
+.BR debpatch (1)
+exits. If not given, then it will be extracted to a temporary directory.
+
+.SH AUTHORS
+\fBdebpatch\fR and this manual page were written by Ximin Luo
+<infinity0 at debian.org>
+.PP
+Both are released under the GNU General Public License, version 3 or later.
+
+.SH SEE ALSO
+.BR debdiff (1)
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/collab-maint/devscripts.git
More information about the devscripts-devel
mailing list