[Pbuilder-maint] A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

[Junichi Uekawa]

Hi,

I've blogged about it, but I'll re-post to the list for easier commenting.

http://www.netfort.gr.jp/~dancer/diary/daily/2006-Mar-4.html.en#2006-Mar-4-23:12:11

While I was reviewing Debian Weekly News translations, I noticed there
was autopkgtest. Nice to see something that emerging to be
available. I've already written quite a few tests for pbuilder
testsuite, so it's unwelcome in terms of having a incompatible test
interface. Browsing autopkgtest, it's non-obvious how to write the
tests, or how to use it, so some documentation is probably
required. pbuilder testsuite has been deployed on quite a few of my
packages, so I have some insights. It's pbuilder, so it's slow, but if
it's included in the automated workflow (kick off build --
automatically-test -- upload), it's not too bad. The basic tests that
can be done are console based tests like running ldd, and giving it
some input to get some output. The tests are in source-code
debian/pbuilder-test/, and are just shell scripts run through
run-parts. The following is a rough list of things that I've done:

    * LADSPA plugins (.so files): run analyseplugin to see that it can
      be detected: see mcp-plugins

    * library package: create and build and run a test program from
      source-code and see that it does compile and run: see ecasound
      package, dancer-xml, dlisp, libdshconfig

    * emacs lisp: run the lisp code non-interactively using -eval, and
      grep for output, see yc-el.

I've not yet done much GUI testing, since I am yet to find a usable GUI automatic testing tool. 


> > >     * Let's modify pbuilder to run test-build tests and (if
> > >       possible) also the generic tool and test-install tests. 
> > >       These belong, I think, better into pbuilder then piuparts, 
> > >       but it might be that piuparts should run them also.
> > 
> > pbuilder hook is available for those who want to test this framework.
> 
> I've been working with this, with a very simplistic set-up of running
> multiple shell scripts. There are things that you know beforehand (if
> you know the gory details of pbuilder as it is documented).
> 
> 1. your source code is available in /tmp/buildd/package-version/,
> which you can probably match with the wildcard '/tmp/buildd/*/debian/../'
> 
> 2. your testsuite resides in /tmp/buildd/*/debian/pbuilder-testsuite/*, 
> which will be executed with run-parts.
> 
> 3. previous version of your package will be installed by B92test-pkg
> by default, from your default apt sources, and if it fails, test won't progress.
> 
> 4. within the chroot, B92test-pkg by default does the following:
> 	your previous version is installed from apt sources
> 	your previous version is removed
> 	your new package is installed with dpkg -i
> this sequence should be improved, and should probably allow failures
> 
> 
> 
> Things that I really want to have after testing a few packages.
> 
> 
> 1. support for common operations, maybe a shell library is in order?
>    For example, I usually want to compile/link/execute a test code to
>    test a -dev package. That requires some amount of idiom in plain
>    shell code, and I shouldn't have to copy-and-paste code all over
>    the place.
> 
> 2. support for providing source data. Most applications eat data and
>    output data. I really want some way to provide data and to say
>    what's the expected output.
> 
> 3. support for X. Some of my packages are command-line console tools,
>    but many are actually graphical apps. It would be a plus to have
>    some kind of interactive/noninteractive X-based testing.


regards,
	junichi
-- 
dancer@{debian.org,netfort.gr.jp}   Debian Project