[Reproducible-commits] [blog] 01/01: Second draft of reprotest week 8

Ceridwen ceridwen-guest at moszumanska.debian.org
Mon Jul 18 19:24:47 UTC 2016


This is an automated email from the git hooks/post-receive script.

ceridwen-guest pushed a commit to branch master
in repository blog.

commit 276a3eaa275c702b8c940cb959a48e39efa1472d
Author: Ceridwen <ceridwenv at gmail.com>
Date:   Mon Jul 18 15:24:42 2016 -0400

    Second draft of reprotest week 8
---
 drafts/people/ceridwen/reprotest_week8.mdwn | 144 +++++++++++++++++++++++++++-
 1 file changed, 139 insertions(+), 5 deletions(-)

diff --git a/drafts/people/ceridwen/reprotest_week8.mdwn b/drafts/people/ceridwen/reprotest_week8.mdwn
index 1a6453b..bb3c06a 100644
--- a/drafts/people/ceridwen/reprotest_week8.mdwn
+++ b/drafts/people/ceridwen/reprotest_week8.mdwn
@@ -1,11 +1,145 @@
-[[!meta title="Preparing for the second release"]]
+[[!meta title="Preparing for the second release of reprotest"]]
 [[!meta date="Mon, 18 Jul 2016 11:51:31 -0400"]]
 [[!tag reproducible_builds Debian reprotest Outreachy]]
 Author: ceridwen
 
-I now have working test environments set up for null, schroot, and
-qemu.  After fixing some bugs, null and qemu now test clean!  schroot
-still has a permission error related to disorderfs.
+I now have working test environments set up for null (no container,
+build on the host system), [schroot](https://wiki.debian.org/Schroot),
+and [qemu](https://wiki.debian.org/QEMU).  After fixing some bugs,
+null and qemu now pass all their tests!
+
+schroot still has a permission error related to disorderfs.  Since the
+same code works for null and qemu and for schroot when disorderfs is
+disabled, it's something specific to disorderfs and/or its combination
+with schroot.  The following is debug output that shows `ls` for the
+build directory on the testbed before and after the mock build, and
+`stat` for both the build directory and the mock build artifact
+itself.  The first control run, without disorderfs, succeeds:
+
+    test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
+    total 20
+    drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+    -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+    -rwxr--r-- 1 user user  175 Jun  3 15:42 mock_failure.py
+    -rw-r--r-- 1 user user  252 Jun 14 16:06 template.ini
+    -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['sh', '-ec', 'cd /tmp/autopkgtest.5oMipL/control/ ;\n python3 mock_build.py ;\n'], kind short, sout raw, serr pipe, env ['LANG=en_US.UTF-8', 'HOME=/nonexistent/first-build', 'VIRTUAL_ENV=~/code/reprotest/.tox/py35', 'PATH=~/code/reprotest/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PYTHONHASHSEED=559200286', 'TZ=GMT+12']
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
+    total 20
+    drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+    -rw-r--r-- 1 root root    0 Jul 18 15:06 artifact
+    -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+    -rwxr--r-- 1 user user  175 Jun  3 15:42 mock_failure.py
+    -rw-r--r-- 1 user user  252 Jun 14 16:06 template.ini
+    -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
+      File: '/tmp/autopkgtest.5oMipL/control/'
+      Size: 4096      	Blocks: 8          IO Block: 4096   directory
+    Device: 56h/86d	Inode: 1351634     Links: 3
+    Access: (0755/drwxr-xr-x)  Uid: ( 1000/    user)   Gid: ( 1000/    user)
+    Access: 2016-07-18 15:06:31.105915342 -0400
+    Modify: 2016-07-18 15:06:31.089915352 -0400
+    Change: 2016-07-18 15:06:31.089915352 -0400
+     Birth: -
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/control/artifact'], kind short, sout raw, serr raw, env []
+      File: '/tmp/autopkgtest.5oMipL/control/artifact'
+      Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
+    Device: fc01h/64513d	Inode: 40767795    Links: 1
+    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
+    Access: 2016-07-18 15:06:31.089915352 -0400
+    Modify: 2016-07-18 15:06:31.089915352 -0400
+    Change: 2016-07-18 15:06:31.089915352 -0400
+     Birth: -
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: sending command to testbed: copyup /tmp/autopkgtest.5oMipL/control/artifact /tmp/tmpw_mwks82/control_artifact
+    schroot: DBG: executing copyup /tmp/autopkgtest.5oMipL/control/artifact /tmp/tmpw_mwks82/control_artifact
+    schroot: DBG: copyup_shareddir: tb /tmp/autopkgtest.5oMipL/control/artifact host /tmp/tmpw_mwks82/control_artifact is_dir False downtmp_host /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52//tmp/autopkgtest.5oMipL
+    schroot: DBG: copyup_shareddir: tb(host) /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/control/artifact is not already at destination /tmp/tmpw_mwks82/control_artifact, copying
+    test.py: DBG: got reply from testbed: ok
+
+That last bit indicates that copy command for the build artifact from
+the testbed to a temporary directory on the host succeeded.  This is the
+debug output from the second run, with disorderfs enabled:
+
+    test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
+    total 20
+    drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+    -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+    -rwxr--r-- 1 user user  175 Jun  3 15:42 mock_failure.py
+    -rw-r--r-- 1 user user  252 Jun 14 16:06 template.ini
+    -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['sh', '-ec', 'cd /tmp/autopkgtest.5oMipL/disorderfs/ ;\n umask 0002 ;\n linux64 --uname-2.6 python3 mock_build.py ;\n'], kind short, sout raw, serr pipe, env ['LC_ALL=fr_CH.UTF-8', 'CAPTURE_ENVIRONMENT=i_capture_the_environment', 'HOME=/nonexistent/second-build', 'VIRTUAL_ENV=~/code/reprotest/.tox/py35', 'PATH=~/code/reprotest/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin/i_capture_the_p [...]
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
+    total 20
+    drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
+    -rw-r--r-- 1 root root    0 Jul 18 15:06 artifact
+    -rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
+    -rwxr--r-- 1 user user  175 Jun  3 15:42 mock_failure.py
+    -rw-r--r-- 1 user user  252 Jun 14 16:06 template.ini
+    -rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
+      File: '/tmp/autopkgtest.5oMipL/disorderfs/'
+      Size: 4096      	Blocks: 8          IO Block: 4096   directory
+    Device: 58h/88d	Inode: 1           Links: 3
+    Access: (0755/drwxr-xr-x)  Uid: ( 1000/    user)   Gid: ( 1000/    user)
+    Access: 2016-07-18 15:06:31.201915291 -0400
+    Modify: 2016-07-18 15:06:31.185915299 -0400
+    Change: 2016-07-18 15:06:31.185915299 -0400
+     Birth: -
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/disorderfs/artifact'], kind short, sout raw, serr raw, env []
+      File: '/tmp/autopkgtest.5oMipL/disorderfs/artifact'
+      Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
+    Device: 58h/88d	Inode: 7           Links: 1
+    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
+    Access: 2016-07-18 15:06:31.185915299 -0400
+    Modify: 2016-07-18 15:06:31.185915299 -0400
+    Change: 2016-07-18 15:06:31.185915299 -0400
+     Birth: -
+    test.py: DBG: testbed command exited with code 0
+    test.py: DBG: sending command to testbed: copyup /tmp/autopkgtest.5oMipL/disorderfs/artifact /tmp/tmpw_mwks82/experiment_artifact
+    schroot: DBG: executing copyup /tmp/autopkgtest.5oMipL/disorderfs/artifact /tmp/tmpw_mwks82/experiment_artifact
+    schroot: DBG: copyup_shareddir: tb /tmp/autopkgtest.5oMipL/disorderfs/artifact host /tmp/tmpw_mwks82/experiment_artifact is_dir False downtmp_host /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52//tmp/autopkgtest.5oMipL
+    schroot: DBG: copyup_shareddir: tb(host) /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/disorderfs/artifact is not already at destination /tmp/tmpw_mwks82/experiment_artifact, copying
+    schroot: DBG: cleanup...
+    schroot: DBG: execute-timeout: schroot --run-session --quiet --directory=/ --chroot jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52 --user=root -- rm -rf -- /tmp/autopkgtest.5oMipL
+    rm: cannot remove '/tmp/autopkgtest.5oMipL/disorderfs': Device or resource busy
+    schroot: DBG: execute-timeout: schroot --quiet --end-session --chroot jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52
+    Unexpected error:
+    Traceback (most recent call last):
+      File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 708, in mainloop
+        command()
+      File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 646, in command
+        r = f(c, ce)
+      File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 584, in cmd_copyup
+        copyupdown(c, ce, True)
+      File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 469, in copyupdown
+        copyupdown_internal(ce[0], c[1:], upp)
+      File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 494, in copyupdown_internal
+        copyup_shareddir(sd[0], sd[1], dirsp, downtmp_host)
+      File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 408, in copyup_shareddir
+        shutil.copy(tb, host)
+      File "/usr/lib/python3.5/shutil.py", line 235, in copy
+        copyfile(src, dst, follow_symlinks=follow_symlinks)
+      File "/usr/lib/python3.5/shutil.py", line 114, in copyfile
+        with open(src, 'rb') as fsrc:
+    PermissionError: [Errno 13] Permission denied: '/var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/disorderfs/artifact'
+    
+`ls` shows that the artifact is created in the right place.  However,
+when reprotest tries to copy it from the testbed to the host, it gets
+a permission error.  The traceback is coming from `virt/schroot`, and
+it's a Python `open()` call that's failing.  Note that the permissions
+are wrong for the second run, but that's expected because my schroot
+is stable so the umask bug isn't fixed yet.  I expect to see the umask
+test fail, though, not a crash in every test where the build
+succeeds.
 
 After a great deal of effort, I isolated the bug that was causing the
 process to hang not to my code or autopkgtest's code, but to
@@ -24,7 +158,7 @@ Here is my current to-do list:
   directory, which fails if disorderfs is mounted, so the script needs
   to unmount it before that happens.)
 
-* Find and fix the schroot/disorderfs/autopkgtest permission error bug.
+* Find and fix the schroot/disorderfs permission error bug.
 
 * Convert my notes on setting up for the tests into something useful
   for users.

-- 
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/reproducible/blog.git



More information about the Reproducible-commits mailing list