[Pkg-xfce-devel] Bug#654468: Bug#645191: update on waf binary data

[Carsten Hey]

I think we should drop ftpmaster from CC in further mails.

* Yves-Alexis Perez [2012-03-17 09:36 +0100]:
> On sam., 2012-03-17 at 02:45 +0100, Carsten Hey wrote:
> > waf scripts are not cleanly divided into python and data, but instead
> > the python part contains also two two byte sequences (found using brute
> > force whilst building the waf script).  My original plan was to ship two
> > scripts debian/waf-unpack and debian/waf-repack to provide an easy way
> > to edit the waf sources and document this in README.source. Due to the
> > above mentioned mix of header information and python source code this
> > can not be done in a clean way, so there so there is nothing to review
> > for ftpmaster.
>
> I don't completely understand this. What are those two bytes sequences
> you bruteforced? Afaict you don't call waf in your snippets (and I
> specifically asked you about that).

The bug report mentioned that the waf script contains an embedded
.tar.bz2.  A short look at the script revealed that there is only one
line where it could be embedded and that it replaces a two byte sequence
with \n and an other one with \r.  Finding out that the line's first
character and the line's last do not belong to the .tar.bz2 wasn't that
hard.  All described can be done easily using sed and similar tools
- and the reverse can also easily be done using sed and similar tools.
This is what I did, I put the described in simple commands.  The idea
was to be able to extract the data from the waf script, modify it and
then embed it again into the waf script - just using simple standard
tools.

The above was based on the assumption that these two two byte sequences
are always the same because they can not occur in bzip2 compressed data
for whatever reason.  This assumption was wrong, waf creates the
.tar.bz2 and then tries all possible two byte sequences until if finds
some that do not occur in the .tar.bz2 and then sets the variables C1
and C2 in the python waf template to these values.  If we would do this
in sed and co. we would need to parse python, which doesn't seem to be
an option.

> > http://bugs.debian.org/660193 (search for the string waf) contains
> > snippets, based on what Tolimar pointed to in his mail, you just need to
> > paste into the midori package and some additional notes.  The remaining
> > part is IMHO to document this in README.source.  One thing I forgot to
> > mention in my mail to #660193 is that the reason to remove the blob from
> > the used waf script is to ensure that the unpacked waf source is used.
>
> Well, in midori diff, I repack and ensure the new one and the old ones
> are the same to be sure I don't do anything bad. Now indeed it could be
> split in two parts, one run by maintainer, which would then hack in the
> waf sources themeselves, and one at build time, which would pick the
> extracted sources and make a new waf script.

My point of all this was to provide an easy way to change the source
code, but this can't be accomplished.  You provided a way to extract the
source to be able to review it, but not to change it.

> And changing the way waf is used at built time is not supported and
> might fail in bad ways too, so it's not really helpful to do things
> *against* what advised by waf upstream.

Users might not be advised to use an extracted source, but "#devs use
$WAFDIR" (this is a comment in the waf script shipped in midori), so
using the extracted wafadmin directory isn't unsupported at all.
$WAFDIR is the directory in which the extracted wafadmin directory is
found.

> And I still consider the decision bad because the source *is* there and
> is tunable, even though it's not the easiest way in the world. But
> upstream(s) made a choice here, we can disagree (and I do) but at the
> end of the day, unless you want to fork, there's not much you can do.

Use an trivial build system for trivial projects and ship waf unpacked
for non-trivial packages is what we can do (midori is clearly
non-trivial).  Having an easy command to unpack and repack waf scripts
would have been great, but this is not possible unless we would adapt
the values of C1 and C2 in the waf script (and thus parsing python),
which would lead to an ugly hack.


Regards
Carsten