Bug#823979: autodep8: Support for ELPA testsuites

[Antonio Terceiro]

On Wed, May 11, 2016 at 08:08:48AM -0700, Sean Whitton wrote:
> Dear Antonio,
> 
> (CCing David to check my understanding on something below)
> 
> On Wed, May 11, 2016 at 10:57:28AM -0300, Antonio Terceiro wrote:
> > Even without having a clue about ELPA, or emacs in general, I can guess
> > with a certain degree of confidence that those tests will be testing the
> > source tree, not the installed packages. The purpose of
> > autopkgtest/DEP-8 is testing the installed binary packages, not the
> > source tree.
> 
> I had forgotten that that was the purpose of DEP-8, so thanks for
> reminding me.
> 
> > Is it possible to run the tests against the installed packages, making
> > sure that code from the source tree is not used (except of course the
> > test code itself)?
> 
> I do not think that there is a general way to ensure that an ELPA
> testsuite uses the installed files rather than the source tree.
> 
> Most ELPA test suites can be run without any helper code: dh_elpa_test
> just loads and executes all the test definitions it can find.  In that
> case we could add some code to ensure it used the installed package when
> it saw --autopkgtest.  However, a lot of ELPA test suites use upstream's
> helper code to do setup before the tests are run.  This code generally
> loads the package out of the source tree.  There is no general way to
> stop this from happening: each package would need its own patch to the
> upstream helper code.  And that makes it impossible for autodep8 to run
> ELPA test suites.

Well, that happens. The situation in Ruby is the following.

By default Ruby source packages have their library code in lib/, and C
extensions in ext/ (mostly). When passed the --autopkgtest switch,
gem2deb-test-runner will first move those away, then run the tests,
then restore them do their original locations. This make sure the
package can't load code from the source package, even if it tries.

The vast majority of upstream test suites rely on the testing
frameworks setting the $LOAD_PATH (Ruby's list of directories to load
code from) to include the source-local directories, and just use
`require "foo"`; these work out of the box.

Some upstream test suites do explicitly load stuff from the source
package; these will pass during builds, and fail under autopkgtest. We
will usually patch the test suite to not rely on a relative location,
and load libraries assuming they are installed system-wide (which should
also work during the build because gem2deb-test-runner does set the
$LOAD_PATH correctly during builds); usualy that means changing
something like `require test_directory + '/../lib/foo'` to something
like `require "foo"`
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20160511/29770c1f/attachment.sig>