[Reproducible-commits] [blog] 01/01: Draft of week 6 report

Wed Jul 6 21:10:29 UTC 2016

This is an automated email from the git hooks/post-receive script.

ceridwen-guest pushed a commit to branch master
in repository blog.

commit 1769c094d1c2b00f4b6f76a420f2326b4f2fdfc3
Author: Ceridwen <ceridwenv at gmail.com>
Date:   Wed Jul 6 17:10:24 2016 -0400

    Draft of week 6 report
---
 drafts/people/ceridwen/reprotest_week6.mdwn | 77 +++++++++++++++++++----------
 1 file changed, 50 insertions(+), 27 deletions(-)

diff --git a/drafts/people/ceridwen/reprotest_week6.mdwn b/drafts/people/ceridwen/reprotest_week6.mdwn
index 23fa12f..4aec6d5 100644
--- a/drafts/people/ceridwen/reprotest_week6.mdwn
+++ b/drafts/people/ceridwen/reprotest_week6.mdwn
@@ -3,11 +3,11 @@
 [[!tag reproducible_builds Debian reprotest Outreachy]]
 Author: ceridwen
 
-I got another reply from Martin Pitt, I hope I'm not trying his
-patience too much.  It's clear that he has a very different mental
-model of how this should work than I did.  I'll illustrate by
-borrowing my previous example.  I know that schroot has the following
-behavior:
+With some more help from Martin Pitt, it became clear to me that my
+mental model of how autopkgtest works is very different from how it
+does work.  I'll illustrate by borrowing my [previous
+example](https://reproducible.alioth.debian.org/blog/posts/people/ceridwen/reprotest_week5/).
+I know that schroot has the following behavior:
 
 > The default behaviour is as follows (all directory paths are inside
 > the chroot).  A login shell is run in the current working directory.
@@ -37,27 +37,50 @@ the right way to do it is to build an abstract syntax tree
 representation of the shell script and then convert it to code.
 
 Whether I need more complicated shell scripts depends on my approach
-to handling state in the containers.  First, I need to figure out what
-state-handling needs to be embedded *in* the shell scripts rather than
-executed from the outside.  For this, I need to know what state
+to handling state in the containers.  I need to know what state
 persists across separate command executions: if I call
 `adt_testbed.Testbed.execute()` twice, what if any changes I make to
-the container will carry forward from the first to the second?  This
-ought to be a documented part of adt_testbed's interface but I can't
-find any documentation to this effect.  Do you know enough about the
-containers to tell me what you would think would carry over?  (This at
-least gives me a starting point for testing the adt_testbed
-code—without documentation, I'm going to have to
-reverse-engineer it.)  For state that carries over from one command to
-another, I can control the state-handling from the Python code,
-similar to how the rebuild.sh script uses pbuilder/cowbuilder hooks to
-manage state.  Any state that doesn't carry over must be controlled
-inside the shell scripts themselves.  There is an additional advantage
-to embedding state handling in the shell scripts: I won't have to
-rewrite autopkgtest's error handling, because the reversion code will
-be executed inside autopkgtest's error handling.  Note that
-autopkgtest has a very different approach, depending on builtin
-reversion capabilities from some of its containers, but I would like
-to avoid doing the same, partly because I'm unsure if some of the
-modifications that I need to make can be reverted by the containers,
-like creating new users or mounting disorderfs.
+the container will carry forward from the first to the second?  There
+are three categories here.  First, some properties of a system aren't
+preserved even from one command execution to the next, like working
+directory and environment variables.  (I thought working directory
+would be preserved, but it's not).  The second is state that persists
+while the testbed is open and is then automatically reverted when it's
+closed, like files copied into temporary directories on the tesbed.
+The third is state that persists across different sessions on the same
+container and must be cleaned up by reprotest.  It's worth noting that
+which state falls into which category may vary by the container in
+question, though for the most part I can either do unnecessary cleanup
+or issue unnecessary commands to handle the differences.  autopkgtest
+itself has a very different approach to cleanup, as it relies almost
+entirely on the builtin reversion capabilities from some of its
+containers.  I would prefer to avoid doing the same, partly because I
+know that some of the modifications I need to make, for instance
+creating new users or mounting disorderfs, can't be reverted by the
+faster, simpler containers like schroot.
+
+From discussions with [Lunar](https://wiki.debian.org/Lunar), I think
+that the variations that correspond to environment variables
+(`captures_environment`, `home`, `locales`, `path`, and `timezone`)
+fall into the first category, but because of the special handling for
+them they don't require sending a separate command.  The shell (`bash
+foo`) accepts a script or command as an argument so it also doesn't
+need a separate command.  Setting the working directory and umask
+require separate commands that have to be joined.  On Linux, `setarch`
+also accepts a command/script as an argument so can be handled like
+the shell, but there's no unified POSIX protocol for mocking `uname`
+so other OSes will require different approaches.  Users, groups, file
+ordering, host, and domain will require cleanup in all containers
+except for (maybe) qemu.  If I want to handle the cleanup in the shell
+scripts themselves, I need conditionals so that for instance the shell
+script only tries to unmount disorderfs if disorderfs was successfully
+mounted.  This approach would simplify the error handling problems
+I've had before, where when a build crashes cleanup code run from
+Python doesn't get run until after the testbed stop accepting
+commands.
+
+Lunar suggested the [Plumbum](https://plumbum.readthedocs.io/) library
+to me, and I think I can use it to avoid writing my own shell AST
+library.  It has a
+[method](https://plumbum.readthedocs.io/en/latest/api/commands.html#plumbum.commands.base.BaseCommand.formulate)
+that converts the Python representation of a shell script into a string that can be passed to `Testbed.command()`.  Integrating Plumbum to generate the necessary scripts is where I'm going in the next week.

-- 
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/reproducible/blog.git

