[Build-common-hackers] rewrite plans -- Python implementation

[Tom Huckstep]

On Tue, May 25, 2004 at 07:08:51AM +0000, W. Borgert wrote:
> I just want to show that there are "high-level" scripting
> options other than perl.  Using make and sh is OK for me,
> but if it's not sufficient better use Python than perl.

Agreed.  Python is great.  As such, here is my implementation of a "PyDBS"
framework (requires python >= 2.2):

http://www.srcf.ucam.org/~teh30/debian/pydbs/

Grab PyDBS-libs.tar.gz (and if you want to actually run it, some test
source, polipo-debian-source.tar.gz).


Instructions for use
---------------------

Untar the PyDBS-libs.tar.gz somewhere and set PYTHONPATH=.../PyDBS-libs.
Untar the polipo-debian-source.tar.gz somewhere.
cd polipo-debian-source

Prepare the build: python debian/pydbs.py prepare
Build the binary: python debian/pydbs.py build
Install the binary: fakeroot python debian/pydbs.py install
Build the package: fakeroot python debian/pydbs.py package
Clean up: python debian/pydbs.py clean


Some explanation
----------------

The file debian/pydbs.py corresponds to debian/rules.  You can see that,
like the CDBS rules, pydbs.py is small and simple.  (To make it smaller we
can get rid of the PyDBS.run() at the end, when we integrate with
debian/rules).

"import"ing PyDBS, PyDBS_Make and PyDBS_Debhelper sets up configuration
variables (PyDBS.config, PyDBS_Make.config etc.) which are used to
personalise the behaviour of these modules.  The "import" also inserts the
appropriate tasks (make tasks, debhelper tasks etc.) into a queue of jobs to
be run at appropriate times.

  The back end
  ------------

The back end is a simple task queueing system.  A Task is basically a set of
jobs.  Different jobs in a Task are run in five different "phases": prepare,
build, install, package, clean.

Tasks are inserted into a TaskQueue and given a "priority" for each of the
phases.  "priority" is a bit of a misnomer; what it actually means is an
integer specifying in what order the TaskQueue should run the Tasks.

For example, you might have a Make task, and a Patch task.

prepare: You want to patch the source with Patch *before* running "make
         configure" with Make (say)
build:   You want to build the source with Make
install: You want to install the source with Make
package: [You want to run some packaging commands, ignore for this example]
clean:   You want to run "make clean" with Make *before* unpatching the source
         with Patch (for example, your patches touch the Makefile)

Then you would create a queue, first

insert Make into it, with priorities 70, 50, 50, 30 in the prepare, build,
install, and clean phases (for example)

then

insert Patch into it, with priorities 50, 70 in the prepare and clean
phases.

Now in the "prepare" phase, Patch has "priority" 50, so gets run before
"make configure" which has priority 70.

In the "clean" phase, Make gets run first, because its priority is 30
compared to Patch's 70.

In pratice, "import"ing the suitable PyDBS_ modules will set up a queue for
you.  If you like, you can still fiddle with the existing priorities, and
add your own Tasks with priorities you specify.

(Lower "priorities" get run first which may be unintuitive.  I'm going to
change the name "priority" to something clearer)


Conclusions
-----------

Python is a nice system for developing this kind of stuff, and allows
encapsulation and complicated logic in a way that make does not.

I can write Python, but I don't know a lot about Debian packaging.  For
example, I need someone to explain to me what debhelper should be doing when
building multiple packages, because I can't work it out from debhelper.mk
alone.

The Task/TaskQueue idiom needs a bit of fiddling to get the design right.
Anyway, explore PyDBS and let me know if you like it as an infrastructure. 
I'd love to hear what you think.

Cheers,

Tom