[Reproducible-commits] [blog] 01/01: Second draft of reprotest week 8
Ceridwen
ceridwen-guest at moszumanska.debian.org
Mon Jul 18 19:24:47 UTC 2016
This is an automated email from the git hooks/post-receive script.
ceridwen-guest pushed a commit to branch master
in repository blog.
commit 276a3eaa275c702b8c940cb959a48e39efa1472d
Author: Ceridwen <ceridwenv at gmail.com>
Date: Mon Jul 18 15:24:42 2016 -0400
Second draft of reprotest week 8
---
drafts/people/ceridwen/reprotest_week8.mdwn | 144 +++++++++++++++++++++++++++-
1 file changed, 139 insertions(+), 5 deletions(-)
diff --git a/drafts/people/ceridwen/reprotest_week8.mdwn b/drafts/people/ceridwen/reprotest_week8.mdwn
index 1a6453b..bb3c06a 100644
--- a/drafts/people/ceridwen/reprotest_week8.mdwn
+++ b/drafts/people/ceridwen/reprotest_week8.mdwn
@@ -1,11 +1,145 @@
-[[!meta title="Preparing for the second release"]]
+[[!meta title="Preparing for the second release of reprotest"]]
[[!meta date="Mon, 18 Jul 2016 11:51:31 -0400"]]
[[!tag reproducible_builds Debian reprotest Outreachy]]
Author: ceridwen
-I now have working test environments set up for null, schroot, and
-qemu. After fixing some bugs, null and qemu now test clean! schroot
-still has a permission error related to disorderfs.
+I now have working test environments set up for null (no container,
+build on the host system), [schroot](https://wiki.debian.org/Schroot),
+and [qemu](https://wiki.debian.org/QEMU). After fixing some bugs,
+null and qemu now pass all their tests!
+
+schroot still has a permission error related to disorderfs. Since the
+same code works for null and qemu and for schroot when disorderfs is
+disabled, it's something specific to disorderfs and/or its combination
+with schroot. The following is debug output that shows `ls` for the
+build directory on the testbed before and after the mock build, and
+`stat` for both the build directory and the mock build artifact
+itself. The first control run, without disorderfs, succeeds:
+
+ test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
+ total 20
+ drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+ -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+ -rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
+ -rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
+ -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['sh', '-ec', 'cd /tmp/autopkgtest.5oMipL/control/ ;\n python3 mock_build.py ;\n'], kind short, sout raw, serr pipe, env ['LANG=en_US.UTF-8', 'HOME=/nonexistent/first-build', 'VIRTUAL_ENV=~/code/reprotest/.tox/py35', 'PATH=~/code/reprotest/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PYTHONHASHSEED=559200286', 'TZ=GMT+12']
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
+ total 20
+ drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+ -rw-r--r-- 1 root root 0 Jul 18 15:06 artifact
+ -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+ -rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
+ -rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
+ -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
+ File: '/tmp/autopkgtest.5oMipL/control/'
+ Size: 4096 Blocks: 8 IO Block: 4096 directory
+ Device: 56h/86d Inode: 1351634 Links: 3
+ Access: (0755/drwxr-xr-x) Uid: ( 1000/ user) Gid: ( 1000/ user)
+ Access: 2016-07-18 15:06:31.105915342 -0400
+ Modify: 2016-07-18 15:06:31.089915352 -0400
+ Change: 2016-07-18 15:06:31.089915352 -0400
+ Birth: -
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/control/artifact'], kind short, sout raw, serr raw, env []
+ File: '/tmp/autopkgtest.5oMipL/control/artifact'
+ Size: 0 Blocks: 0 IO Block: 4096 regular empty file
+ Device: fc01h/64513d Inode: 40767795 Links: 1
+ Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
+ Access: 2016-07-18 15:06:31.089915352 -0400
+ Modify: 2016-07-18 15:06:31.089915352 -0400
+ Change: 2016-07-18 15:06:31.089915352 -0400
+ Birth: -
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: sending command to testbed: copyup /tmp/autopkgtest.5oMipL/control/artifact /tmp/tmpw_mwks82/control_artifact
+ schroot: DBG: executing copyup /tmp/autopkgtest.5oMipL/control/artifact /tmp/tmpw_mwks82/control_artifact
+ schroot: DBG: copyup_shareddir: tb /tmp/autopkgtest.5oMipL/control/artifact host /tmp/tmpw_mwks82/control_artifact is_dir False downtmp_host /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52//tmp/autopkgtest.5oMipL
+ schroot: DBG: copyup_shareddir: tb(host) /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/control/artifact is not already at destination /tmp/tmpw_mwks82/control_artifact, copying
+ test.py: DBG: got reply from testbed: ok
+
+That last bit indicates that copy command for the build artifact from
+the testbed to a temporary directory on the host succeeded. This is the
+debug output from the second run, with disorderfs enabled:
+
+ test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
+ total 20
+ drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+ -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+ -rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
+ -rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
+ -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['sh', '-ec', 'cd /tmp/autopkgtest.5oMipL/disorderfs/ ;\n umask 0002 ;\n linux64 --uname-2.6 python3 mock_build.py ;\n'], kind short, sout raw, serr pipe, env ['LC_ALL=fr_CH.UTF-8', 'CAPTURE_ENVIRONMENT=i_capture_the_environment', 'HOME=/nonexistent/second-build', 'VIRTUAL_ENV=~/code/reprotest/.tox/py35', 'PATH=~/code/reprotest/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin/i_capture_the_p [...]
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
+ total 20
+ drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+ -rw-r--r-- 1 root root 0 Jul 18 15:06 artifact
+ -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+ -rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
+ -rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
+ -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
+ File: '/tmp/autopkgtest.5oMipL/disorderfs/'
+ Size: 4096 Blocks: 8 IO Block: 4096 directory
+ Device: 58h/88d Inode: 1 Links: 3
+ Access: (0755/drwxr-xr-x) Uid: ( 1000/ user) Gid: ( 1000/ user)
+ Access: 2016-07-18 15:06:31.201915291 -0400
+ Modify: 2016-07-18 15:06:31.185915299 -0400
+ Change: 2016-07-18 15:06:31.185915299 -0400
+ Birth: -
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/disorderfs/artifact'], kind short, sout raw, serr raw, env []
+ File: '/tmp/autopkgtest.5oMipL/disorderfs/artifact'
+ Size: 0 Blocks: 0 IO Block: 4096 regular empty file
+ Device: 58h/88d Inode: 7 Links: 1
+ Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
+ Access: 2016-07-18 15:06:31.185915299 -0400
+ Modify: 2016-07-18 15:06:31.185915299 -0400
+ Change: 2016-07-18 15:06:31.185915299 -0400
+ Birth: -
+ test.py: DBG: testbed command exited with code 0
+ test.py: DBG: sending command to testbed: copyup /tmp/autopkgtest.5oMipL/disorderfs/artifact /tmp/tmpw_mwks82/experiment_artifact
+ schroot: DBG: executing copyup /tmp/autopkgtest.5oMipL/disorderfs/artifact /tmp/tmpw_mwks82/experiment_artifact
+ schroot: DBG: copyup_shareddir: tb /tmp/autopkgtest.5oMipL/disorderfs/artifact host /tmp/tmpw_mwks82/experiment_artifact is_dir False downtmp_host /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52//tmp/autopkgtest.5oMipL
+ schroot: DBG: copyup_shareddir: tb(host) /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/disorderfs/artifact is not already at destination /tmp/tmpw_mwks82/experiment_artifact, copying
+ schroot: DBG: cleanup...
+ schroot: DBG: execute-timeout: schroot --run-session --quiet --directory=/ --chroot jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52 --user=root -- rm -rf -- /tmp/autopkgtest.5oMipL
+ rm: cannot remove '/tmp/autopkgtest.5oMipL/disorderfs': Device or resource busy
+ schroot: DBG: execute-timeout: schroot --quiet --end-session --chroot jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52
+ Unexpected error:
+ Traceback (most recent call last):
+ File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 708, in mainloop
+ command()
+ File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 646, in command
+ r = f(c, ce)
+ File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 584, in cmd_copyup
+ copyupdown(c, ce, True)
+ File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 469, in copyupdown
+ copyupdown_internal(ce[0], c[1:], upp)
+ File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 494, in copyupdown_internal
+ copyup_shareddir(sd[0], sd[1], dirsp, downtmp_host)
+ File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 408, in copyup_shareddir
+ shutil.copy(tb, host)
+ File "/usr/lib/python3.5/shutil.py", line 235, in copy
+ copyfile(src, dst, follow_symlinks=follow_symlinks)
+ File "/usr/lib/python3.5/shutil.py", line 114, in copyfile
+ with open(src, 'rb') as fsrc:
+ PermissionError: [Errno 13] Permission denied: '/var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/disorderfs/artifact'
+
+`ls` shows that the artifact is created in the right place. However,
+when reprotest tries to copy it from the testbed to the host, it gets
+a permission error. The traceback is coming from `virt/schroot`, and
+it's a Python `open()` call that's failing. Note that the permissions
+are wrong for the second run, but that's expected because my schroot
+is stable so the umask bug isn't fixed yet. I expect to see the umask
+test fail, though, not a crash in every test where the build
+succeeds.
After a great deal of effort, I isolated the bug that was causing the
process to hang not to my code or autopkgtest's code, but to
@@ -24,7 +158,7 @@ Here is my current to-do list:
directory, which fails if disorderfs is mounted, so the script needs
to unmount it before that happens.)
-* Find and fix the schroot/disorderfs/autopkgtest permission error bug.
+* Find and fix the schroot/disorderfs permission error bug.
* Convert my notes on setting up for the tests into something useful
for users.
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/reproducible/blog.git
More information about the Reproducible-commits
mailing list