[Reproducible-commits] [blog] 01/01: First draft for reprotest week 4

[Ceridwen]

This is an automated email from the git hooks/post-receive script.

ceridwen-guest pushed a commit to branch master
in repository blog.

commit 0b1d56dcb8093afbb7aa626203d68fb098c3705e
Author: Ceridwen <ceridwenv at gmail.com>
Date:   Mon Jun 20 18:21:09 2016 -0400

    First draft for reprotest week 4
---
 drafts/people/ceridwen/reprotest_week4.mdwn | 99 +++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)

diff --git a/drafts/people/ceridwen/reprotest_week4.mdwn b/drafts/people/ceridwen/reprotest_week4.mdwn
new file mode 100644
index 0000000..9abc6cd
--- /dev/null
+++ b/drafts/people/ceridwen/reprotest_week4.mdwn
@@ -0,0 +1,99 @@
+[[!meta title="Reorganizing the autopkgtest code"]]
+[[!meta date="Mon, 20 Jun 2016 14:30:02 -0400"]]
+[[!tag reproducible_builds Debian reprotest Outreachy]]
+Author: ceridwen
+
+I've spent most of the past week reverse-engineering autopkgtest, with
+some help from the maintainer, [Martin
+Pitt](https://www.piware.de/tag/autopkgtest/).
+
+As I outlined [last
+week](https://reproducible.alioth.debian.org/blog/posts/people/ceridwen/reprotest_week3/),
+autopkgtest is composed of a main command-line program and other
+command-line programs `chroot`, `lxd`, `schroot`, `null`, `schroot`,
+and `qemu`, which live in a directory virt/.  (I've followed
+autopkgtest 4.0 in dropping the `adt-virt-` prefix and other name
+conventions.)  Autopkgtest's directory structure looks like:
+
+    autopkgtest/
+        lib/
+			adt_testbed.py
+			adtlog.py
+			VirtSubproc.py
+		runner/
+		setup-commands/
+		ssh-setup/
+		tests/
+		tools/
+		virt/
+			chroot
+			lxc
+			lxd
+			null
+			qemu
+			schroot
+			ssh
+
+Reprotest's directory structure looks like:
+
+    reprotest/
+        reprotest/
+	    __init__.py
+            lib/
+                adt_testbed.py
+                adtlog.py
+                VirtSubproc.py
+	        runner/
+            tools/
+            virt/
+                chroot
+                lxc
+                lxd
+                null
+                qemu
+                schroot
+                ssh
+	    tests/
+
+Autopkgtest doesn't contain any `__init__.py` files, which means that
+from the perspective of Python none of its components are Python
+packages.  To enable the various components of autopkgtest test to
+import each other as
+[implicit namespace packages](https://www.python.org/dev/peps/pep-0420/),
+there's a lot of explicit `sys.path` mutation code in the various
+files like `sys.path.insert(0, '/usr/share/autopkgtest/python')` and
+`sys.path.insert(0,
+os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+'lib'))`.  For my design goals, there are two problems with this.
+First, setuptools makes it very, very difficult to distribute code
+that isn't in Python packages, so this approach more or less doesn't
+work with PyPi at all.  Second, different OSes have different
+directory layouts for Python packages, so an absolute path like
+/usr/share/autopkgtest/python isn't portable.
+
+The main code of reprotest at the moment lives in its `__init__.py`.
+The autopkgtest CLI of the main runner is specific to running tests,
+so my entry point is going to be `adt_testbed`.  My first attempt to
+refactor this involved putting an empty `__init__.py` in lib/ to turn
+into a package so that I could import the modules in the files in lib/
+using code like `from reprotest.lib import adtlog`.  This works fine
+for the files in lib/ and makes `adt_testbed` importable in the main
+`__init__.py`.  However, the executables in virt/, which are invoked
+by `subprocess` calls in adt_testbed.py, *also* need to import modules
+from lib/ for the various `adtlog` functions and the timeout functions
+in `VirtSubproc`.  I haven't been able to find a simple modification
+of `sys.path` that allows the new Python processes launched from virt/
+to properly import `reprotest.lib` as a package, which means I'm
+looking at some kind of larger refactoring, either changing all the
+`subprocess` calls in `adt_testbed` so they work with `reprotest.virt`
+as a subpackage or reorganizing the rest of the code somehow.
+
+It's still not clear to me why autopkgtest uses its own special logging
+module rather than the standard library's `logging` module.  I thought
+it might have something to do with combining the logging from a
+process and its subprocesses, but because the Python subprocesses are
+importing their own instances of the `adtlog` module, this isn't the
+case.  Another problem related to logging and debugging is that the
+programs in virt/ interact through stdin and stdout, which means that
+I haven't been able to use simple prints to help work out their state
+during execution.

-- 
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/reproducible/blog.git