[SCM] ci-tooling packaging branch, master, updated. 89cf10e6c9c3e4a3d5287b506a769049cc721fe1
Harald Sitter
apachelogger-guest at moszumanska.debian.org
Tue May 19 08:52:13 UTC 2015
Gitweb-URL: http://git.debian.org/?p=pkg-kde/ci-tooling.git;a=commitdiff;h=89cf10e
The following commit has been merged in the master branch:
commit 89cf10e6c9c3e4a3d5287b506a769049cc721fe1
Author: Harald Sitter <sitter at kde.org>
Date: Tue May 19 10:51:22 2015 +0200
implement very hackish dsc mangling until we get proper control writing
dsc mangling edits a dsc's Architecture field to replace the 'any'
wildcard with a specific architecture list as per the data/kci.yaml info.
launchpad in turn only will build the source on the listed architectures.
this is to explicitly prevent extra_architectures from building unless
enabled (e.g. armhf would not get stable builds nor would it get
application builds).
for a job to opt into arm building it needs to add the relevant arch to
env ENABLED_EXTRA_ARCHITECTURES.
only extra architectures that are both in the data/kci.yml file AND in the
env variable can be enabled. adding an arch to either of the two alone will
have no impact whatsoever. this is to have a double safe guard against
screwing up and have 600 armhf builds triggered.
---
kci/builder.rb | 27 ++++++-
lib/debian/dsc_arch_twiddle.rb | 76 ++++++++++++++++++
.../test_debian_dsc_arch_twiddle/test_any/my.dsc | 16 ++++
.../test_any_all/my.dsc | 16 ++++
.../test_i386_amd64/my.dsc | 16 ++++
.../test_i386_amd64_all/my.dsc | 16 ++++
.../test_multiline/my.dsc | 17 ++++
.../test_no_arch/my.dsc | 15 ++++
.../test_randomextraarch/my.dsc | 16 ++++
.../test_randomextraarch_all/my.dsc | 16 ++++
.../test_too_many/dsc-arch_1ubuntu1.dsc | 16 ++++
.../test_too_many/dsc-arch_1ubuntu2.dsc | 16 ++++
test/test_debian_dsc_arch_twiddle.rb | 90 ++++++++++++++++++++++
13 files changed, 352 insertions(+), 1 deletion(-)
diff --git a/kci/builder.rb b/kci/builder.rb
index af3d368..2d2763d 100755
--- a/kci/builder.rb
+++ b/kci/builder.rb
@@ -7,6 +7,7 @@ require 'timeout'
require_relative 'lib/ci/build_version'
require_relative 'lib/debian/changelog'
+require_relative 'lib/debian/dsc_arch_twiddle'
require_relative 'lib/cmake_parser'
require_relative 'lib/kci'
require_relative 'lib/lint/control'
@@ -170,11 +171,35 @@ end
# dpkg-buildpackage
Dir.chdir('build/source/') do
system('update-maintainer')
- unless system("dpkg-buildpackage -S -k#{KEYID}")
+ unless system('dpkg-buildpackage -us -uc -S')
abort 'Failed to build source package'
end
end
+# Mangle dsc to not do ARM builds unless explicitly enabled.
+# With hundreds of distinct sources on CI, building all of them on three
+# architectures, of which one is not too commonly used, is extremly excessive.
+# Instead, by default we only build on the common architectures with extra
+# architectures needing to be enabled explicitly.
+# To achieve this mangle the Architecture field in the control file.
+# If it contains an uncommon arch -> remove it -> if it is empty now, abort
+# If it contains any -> replace with !uncommon
+# This is a cheapster hack implementation to avoid having to implement write
+# support in Debian control.
+begin
+ Debian::DSCArch.twiddle!('build/')
+rescue Debian::DSCArch::Error => e
+ # NOTE: this can raise a number of errors and we want them all to be fatal
+ # to prevent unhandled dscs or completely empty architectures from
+ # getting uploaded.
+ abort e
+end
+
+# Sign
+Dir.chdir('build/source/') do
+ abort 'Failed to sign the source.' unless system("debsign -k#{KEYID}")
+end
+
# Upload
def timeout_spawn(cmd, timeout)
pid = Process.spawn(cmd, pgroup: true)
diff --git a/lib/debian/dsc_arch_twiddle.rb b/lib/debian/dsc_arch_twiddle.rb
new file mode 100644
index 0000000..5bcc075
--- /dev/null
+++ b/lib/debian/dsc_arch_twiddle.rb
@@ -0,0 +1,76 @@
+require_relative '../kci'
+
+module Debian
+ # Mangle dsc to not do ARM builds unless explicitly enabled.
+ # With hundreds of distinct sources on CI, building all of them on three
+ # architectures, of which one is not too commonly used, is extremly excessive.
+ # Instead, by default we only build on the common architectures with extra
+ # architectures needing to be enabled explicitly.
+ # To achieve this mangle the Architecture field in the control file.
+ # If it contains an uncommon arch -> remove it -> if it is empty now, abort
+ # If it contains any -> replace with !uncommon
+ # This is a cheapster hack implementation to avoid having to implement write
+ # support in Debian::Control|DSC.
+ class DSCArch
+ class Error < Exception; end
+ class EmptyError < Error; end
+ class CountError < Error; end
+ class MultilineError < Error; end
+
+ def self.enabled_architectures
+ enabled_architectures = KCI.architectures.dup
+ extras = KCI.extra_architectures
+ env_extras = ENV.fetch('ENABLED_EXTRA_ARCHITECTURES') { nil }
+ extras.each do |extra|
+ next unless env_extras && !env_extras.empty?
+ next unless env_extras.split(' ').include?(extra)
+ enabled_architectures << extra
+ end
+ enabled_architectures
+ end
+
+ # Twiddles the Architecture field of a DSC file to only list enabled arches.
+ def self.twiddle!(directory_with_dsc)
+ dsc = nil
+ Dir.chdir(directory_with_dsc) do
+ dsc = Dir.glob('*.dsc')
+ unless dsc.size == 1
+ fail CountError, "Not exactly one dsc WTF -> #{dsc}"
+ end
+ dsc = File.expand_path(dsc[0])
+ end
+
+ enabled_arches = enabled_architectures
+
+ saw_architecture = false
+ line_after = false
+ lines = File.read(dsc).lines
+ lines.collect! do |line|
+ if line_after && line.start_with?(' ')
+ fail MultilineError, 'Line after Architecture starts with space.'
+ end
+ line_after = false
+
+ match = line.match(/^Architecture: (.*)$/)
+ next line unless match
+ arches = match[1].split(' ')
+ arches.collect! do |arch|
+ next enabled_arches if arch == 'any'
+ next arch if arch == 'all'
+ next nil unless enabled_arches.include?(arch)
+ arch
+ end
+ arches.flatten!
+ arches.compact!
+ arches.uniq!
+ fail EmptyError, "Ripped all arches out of '#{line}'" if arches.empty?
+ saw_architecture = true
+ line_after = true
+ "Architecture: #{arches.join(' ')}
"
+ end
+ File.write(dsc, lines.join)
+ return if saw_architecture
+ fail EmptyError, 'There apparently was no Architecture field!'
+ end
+ end
+end
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_any/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_any/my.dsc
new file mode 100644
index 0000000..2dee80b
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_any/my.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: any
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_any_all/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_any_all/my.dsc
new file mode 100644
index 0000000..5074910
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_any_all/my.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: any all
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_i386_amd64/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_i386_amd64/my.dsc
new file mode 100644
index 0000000..23a8b01
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_i386_amd64/my.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: i386 amd64
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_i386_amd64_all/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_i386_amd64_all/my.dsc
new file mode 100644
index 0000000..7084204
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_i386_amd64_all/my.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: i386 amd64 all
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_multiline/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_multiline/my.dsc
new file mode 100644
index 0000000..2ef5299
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_multiline/my.dsc
@@ -0,0 +1,17 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: any
+ all
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_no_arch/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_no_arch/my.dsc
new file mode 100644
index 0000000..ac97340
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_no_arch/my.dsc
@@ -0,0 +1,15 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_randomextraarch/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_randomextraarch/my.dsc
new file mode 100644
index 0000000..68fc119
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_randomextraarch/my.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: randomextraarch
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_randomextraarch_all/my.dsc b/test/data/test_debian_dsc_arch_twiddle/test_randomextraarch_all/my.dsc
new file mode 100644
index 0000000..a117800
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_randomextraarch_all/my.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: randomextraarch all
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_too_many/dsc-arch_1ubuntu1.dsc b/test/data/test_debian_dsc_arch_twiddle/test_too_many/dsc-arch_1ubuntu1.dsc
new file mode 100644
index 0000000..2dee80b
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_too_many/dsc-arch_1ubuntu1.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: any
+Version: 1ubuntu1
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ b4404e68dd17c01b1785207ed7b658ccceab525e 732 dsc-arch_1ubuntu1.tar.xz
+Checksums-Sha256:
+ 7a1ab2a2c0795606e0b0c9595d35626d2129d91d2df6ae86417949a1b2a4debd 732 dsc-arch_1ubuntu1.tar.xz
+Files:
+ e9b673639180378b528e5cd1b4dc0876 732 dsc-arch_1ubuntu1.tar.xz
diff --git a/test/data/test_debian_dsc_arch_twiddle/test_too_many/dsc-arch_1ubuntu2.dsc b/test/data/test_debian_dsc_arch_twiddle/test_too_many/dsc-arch_1ubuntu2.dsc
new file mode 100644
index 0000000..60a2afb
--- /dev/null
+++ b/test/data/test_debian_dsc_arch_twiddle/test_too_many/dsc-arch_1ubuntu2.dsc
@@ -0,0 +1,16 @@
+Format: 3.0 (native)
+Source: dsc-arch
+Binary: dsc-arch
+Architecture: !amd64
+Version: 1ubuntu2
+Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss at lists.ubuntu.com>
+Standards-Version: 3.9.6
+Build-Depends: debhelper (>= 9)
+Package-List:
+ dsc-arch deb kde optional arch=any
+Checksums-Sha1:
+ 838ffec28cfee1ee7912791e3d9992cd6632fb21 744 dsc-arch_1ubuntu2.tar.xz
+Checksums-Sha256:
+ d703f4e933f22137dc867905d0a7166bfdc84ed3a73f5b0568b78e63c2717b55 744 dsc-arch_1ubuntu2.tar.xz
+Files:
+ b1901cdd789e94b51dc7dfb43715bdd1 744 dsc-arch_1ubuntu2.tar.xz
diff --git a/test/test_debian_dsc_arch_twiddle.rb b/test/test_debian_dsc_arch_twiddle.rb
new file mode 100644
index 0000000..1fa018f
--- /dev/null
+++ b/test/test_debian_dsc_arch_twiddle.rb
@@ -0,0 +1,90 @@
+require 'fileutils'
+
+require_relative '../lib/debian/dsc_arch_twiddle'
+require_relative 'lib/testcase'
+
+# Test debian/source/format
+module Debian
+ class DSCArchTwiddleTest < TestCase
+ def teardown
+ ENV.delete('ENABLED_EXTRA_ARCHITECTURES')
+ end
+
+ def tmpdirdata
+ File.join(Dir.pwd, File.basename(data))
+ end
+
+ # Note: data is a method that resolves the fixture path from the
+ # calling test_ method name. So we can not set up from setup.
+ def copy_data
+ FileUtils.cp_r(Dir.glob(data), Dir.pwd)
+ end
+
+ def assert_twiddles(expected_str, env: [])
+ ENV.delete('ENABLED_EXTRA_ARCHITECTURES')
+ ENV['ENABLED_EXTRA_ARCHITECTURES'] = env.join(' ') unless env.empty?
+ copy_data
+ DSCArch.twiddle!(tmpdirdata)
+ lines = File.read("#{tmpdirdata}/my.dsc").lines
+ assert_equal(expected_str, lines[3])
+ end
+
+ def test_too_many
+ copy_data
+ assert_raise Debian::DSCArch::CountError do
+ DSCArch.twiddle!(tmpdirdata)
+ end
+ end
+
+ def test_any
+ assert_twiddles("Architecture: amd64 i386
")
+ assert_twiddles("Architecture: amd64 i386 armhf
", env: %w(armhf))
+ end
+
+ def test_any_all
+ assert_twiddles("Architecture: amd64 i386 all
")
+ assert_twiddles("Architecture: amd64 i386 armhf all
", env: %w(armhf))
+ end
+
+ def test_i386_amd64
+ # Note: order is as in input (which is non-alpahabetical)
+ assert_twiddles("Architecture: i386 amd64
")
+ assert_twiddles("Architecture: i386 amd64
", env: %w(armhf))
+ end
+
+ def test_i386_amd64_all
+ # Note: order is as in input (which is non-alpahabetical).
+ assert_twiddles("Architecture: i386 amd64 all
")
+ assert_twiddles("Architecture: i386 amd64 all
", env: %w(armhf))
+ end
+
+ def test_randomextraarch_all
+ assert_twiddles("Architecture: all
")
+ # Not in KCI data, won't be allowed even if in ENV...
+ assert_twiddles("Architecture: all
", env: %w(randomextraarch))
+ end
+
+ def test_randomextraarch
+ assert_raise DSCArch::EmptyError do
+ copy_data
+ DSCArch.twiddle!(tmpdirdata)
+ end
+ end
+
+ def test_no_arch
+ assert_raise DSCArch::EmptyError do
+ copy_data
+ DSCArch.twiddle!(tmpdirdata)
+ end
+ end
+
+ def test_multiline
+ # Line after Architecture starts with a space, indicating a folded field
+ # but we can't handle folding right now, so we expect a fail.
+ assert_raise DSCArch::MultilineError do
+ copy_data
+ DSCArch.twiddle!(tmpdirdata)
+ end
+ end
+ end
+end
--
ci-tooling packaging
More information about the pkg-kde-commits
mailing list