Packaging Zope 2.12

Wed Aug 18 13:47:30 UTC 2010

On 7/27/10 9:20 AM, Jonas Meurer wrote:
> Hey Michael,
>
> On 26/07/2010 Michael Mulich wrote:
>> On 7/20/10 6:14 PM, Jonas Meurer wrote:
>>> On 14/07/2010 Michael Mulich wrote:
>>> I for sure can help you with packaging issues, but unfortunately I don't
>>> know much about buildout and all that distutils stuff. Maybe we can work
>>> together in order to package zope2.12, that would be awesome.
>>
>> Sounds like a plan.
>
> Indeed. Let's try it ...
>
>>> Do you know the pkg-zope SVN repository? It already has a zope2.12
>>> directory, but I guess that most of debian/rules (the makefile used to
>>> build the package) needs to be rewritten. You can find the repos at
>>> svn://svn.debian.org/svn/pkg-zope/zope2.12
>>>
>> Thanks for the link. I found the repository the other day. I decided
>> to fork the zope2.12 package for now, but would be more than willing
>> to work on the package in the debian repo if someone can give me
>> access.
>
> Sure, but first you need an account on alioth. if you don't have one
> yet, you can register at http://alioth.debian.org/
>
Got it... my user is pumazi-guest.

>>> If I got it right, the main difference between zope2.11 and zope2.12
>>> build system is, that zope2.12 no longer contains the required zope
>>> dependencies (libraries developed for zope). Instead it uses some
>>> buildout receipe to fetch these dependencies as python eggs, and builds
>>> something like a full python environment, including all required python
>>> dependencies.
>>>
>> Right. My latest stab has been to create a buildout recipe that will
>> isolate these dependencies.[0] I've tried to solve the dependency
>> issue without using buildout, but I'm tired of fighting that battle,
>> so I've embraced the buildout. I intend to use this isolation recipe
>> to populate a python environment (or virtual environment) rather
>> than trying to build Zope inside the that environment.
>>
>> The isolation recipe has three deliverables that are relevant to the
>> packaging of zope:
>>
>>     1) The distribution packages and any dependencies are placed in a
>> directory.
>>     2) A .pth file, which is basically a list of paths where these
>> packages are located, is created; and placed by default along side
>> the packages.
>>     3) The scripts that get generated from these distributions are
>> placed into a separate directory.
>>
>> I created a document that somewhat illustrates the new build process
>> for the zope2.12 package.[1] Rather than repeat all that here, I'll
>> just supply the link at the end of this email under number [1]. But
>> for completeness, the gist of it is that we can use the .pth file to
>> pull all the zope dependencies into a single python environment.
>
> I took a quick look at it, see below.
>
>>> So one issue with packaging is how to make the dependencies available to
>>> the build process. Fetching them from internet is not an option at all.
>>> Unfortunately the dependencies break even for minor version changes (in
>>> other words: they don't ensure backwards compability), thus packaging
>>> all the zope libs as seperate packages is not an option either. Thus the
>>> only option is to distribute all required eggs within the zope package
>>> tarball, which is a bad solution security-wise.
>>
>> Can you elaborate on why that would be "bad solution security-wise".
>
> because some of the dependencies are already packaged as seperate
> packages within debian. these packages are updated regularely for new
> upstream versions. as zope2.12 requires particular version of these
> dependencies in order to build, using the debian packages as build
> dependencies is no option. on the other hand, shipping older versions of
> the dependencies within the zope2.12 package creates a security problem:
>
> code is duplicated. both the zope2.12 debian package and the zope
> modules packages do contain the same software. if a security issue
> requires a package update by the security team, the security team needs
> to know that zope2.12 contains a copy of this code.
>
I don't see a way around this. I'd much rather see a working packaged 
version of zope2.12 rather than bog down with duplication details at 
this time. Factoring out the duplication should be a near future goal 
IMHO. I'm definitely not going to concern myself with the duplication 
just yet.

>>> Another issue is how to build the zope/python environment in a FHS
>>> compliant way: placing the different parts of zope (libraries, logs,
>>> configs) at places similar to legacy zope versions: with relative paths,
>>> using the system python interpreter instead of a local copy, etc.
>>>
>> This is the part I'm still fuzzy on. I'm not very familiar with the
>> legacy zope setup. I currently use the zope2.10 package, and I'm
>> only somewhat somewhat familiar with that.
>
> you can see the structure of zope2.10 package with 'dpkg -L zope2.10'.
>
> basicly this is:
>
> /usr/lib/zope2.10:
> 	/bin:		the zope binaries such as repozo.py
> 	/skel:		the skeleton files for new zope instances
> 	/lib/python:	all shipped python and zope modules
>
A good portion of this should be taken care of by 1.a in my debian 
packaging notes.[1] The skel directory is not something I have addressed 
yet. I was under the assumption that mkzopeinstance script that comes 
with zope2 would be used to create an instance. Would using this script 
instead of skel be an option?

> /usr/share/doc/zope2.10: documentation stuff
>
It's sphinx docs, so this shouldn't be a problem.

> /usr/lib/zope2.11/bin/python: a symlink to the system python version
>
Would using virtualenv to create the environment (1.a of my notes[1]), 
which does not create a symlink back to the system python cause 
problems? I'm not exactly sure what the process is that virtualenv uses 
to create the environment; so maybe there is a better way to create the 
environment without depending on virtualenv to do it. Also, since 
virtualenv clones the python binary (I think?), this would likely cause 
problems during an update, no?

> and some empty directories which are used by instances:
>
> /var/log/zope2.10/<i_name>:		<i_name>  log files
> /etc/zope2.10/<i_name>:			<i_name>  config files
> /var/lib/zope2.10/instance/<i_name>:	all other<i_name>  files
>
> /usr/share/zope/Products: Zope Products installed as Debian packages
>
> this structure should stay as is if possible.
>
Yeah, this structure is sound, no sense in changing it unless problems 
arise.

>>> And last but not least instances need to be seperated from the global
>>> zope2.12 files in a similar way as done up to zope2.11. continuing
>>> support for the debian 'dzhandle' utility for managing zope instances,
>>> products and extensions should be the goal.
>>>
>> I'll have to think about this a bit more. I decided early on in this
>> process that I was going to abandon Products support, because 1)
>> Plone has decided to remove the products section from it's default
>> buildout 2) the unified installer has done this as well. Therefore,
>> I think it would be easier for us to drop the Products stuff. But I
>> would like to add support for adding distributes (aka eggs) much
>> like products are added currently.
>
> I'd like to keep the Products support, and if I got it right that will
> not add any extra work for packaging. Zope2.12 already contains support
> for Products, Debian tools (dzhandle) just needs to manage symlinks from
> the instance's Products folder to actual place of Products.
>
Ok. I don't imagine much of this will change from its current 
implementation. I'm not opposed to continuing support for Products, I 
just thought it would be a good time to abandon them.

I would however express the need to include distribution support in 
dzhandle. For example, installing collective.contentleadimage with 
dzhandle. Something like, "dzhandle add-dist plone-site 
collective.contentleadimage". This functionality would work similar to 
add-product, in that it would not go fetch the package but instead 
simply grab it from a designated location like it does for Products. 
Additionally it would create a zcml slug for the package if requested as 
a flag in the command. This is not something I plan to get working

>>> greetings,
>>>   jonas
>> Thanks for the support Jonas!
>>
>> I'd be grateful if a few people could look over the document [1] and
>> let me know if there are any glaring misunderstandings and/or
>> missing pieces. Thanks.
>>
>> [0] https://weblion.psu.edu/trac/weblion/browser/weblion/buildout.recipe.isolation?rev=12091
>> [1] https://weblion.psu.edu/trac/weblion/browser/users/mrm41/zope2.12-debian_packaging_notes.rst?rev=12092
>
> this looks very promising. but am I right, that you want to install the
> python environment to /usr/lib/zope2.12 at the build process? that is
> not possible. The whole build process needs to be done inside the
> package directory. In most cases, Debian packages build the package into
> ./debian/<pkg-name>/. If zope2.11 is built from /home/user/zope2.11/ as
> example, the package will be compiled and built into
> /home/usr/zope2.11/debian/zope2.11/, and this directory contains the
> whole filesystem structure of the package:
> /home/usr/zope2.11/debian/zope2.11/usr/lib/zope2.11 ...
> /home/usr/zope2.11/debian/zope2.11/usr/share/doc/zope2.11 ...
> and so on.
>
Right. The build locations were implied. I put the final locations in 
the notes, because I thought it might make it easier to understand.

> thus the build process needs to take care of paths in the built package.
> this is done by configure/build flags for software built with autotools.
> Unfortunately, I don't know how to do this with buildout.
>
> If you have any further questions, don't hesitate to ask.
>
> greetings,
>   jonas

-Michael Mulich (pumazi)