Bug#823979: autodep8: Support for ELPA testsuites

[Sean Whitton]

Dear Antonio,

(CCing David to check my understanding on something below)

On Wed, May 11, 2016 at 10:57:28AM -0300, Antonio Terceiro wrote:
> Even without having a clue about ELPA, or emacs in general, I can guess
> with a certain degree of confidence that those tests will be testing the
> source tree, not the installed packages. The purpose of
> autopkgtest/DEP-8 is testing the installed binary packages, not the
> source tree.

I had forgotten that that was the purpose of DEP-8, so thanks for
reminding me.

> Is it possible to run the tests against the installed packages, making
> sure that code from the source tree is not used (except of course the
> test code itself)?

I do not think that there is a general way to ensure that an ELPA
testsuite uses the installed files rather than the source tree.

Most ELPA test suites can be run without any helper code: dh_elpa_test
just loads and executes all the test definitions it can find.  In that
case we could add some code to ensure it used the installed package when
it saw --autopkgtest.  However, a lot of ELPA test suites use upstream's
helper code to do setup before the tests are run.  This code generally
loads the package out of the source tree.  There is no general way to
stop this from happening: each package would need its own patch to the
upstream helper code.  And that makes it impossible for autodep8 to run
ELPA test suites.

If I'm right about this, I guess we have to close this as wontfix.
(David: do you think I'm right about this?)

> If it's possible, it would be useful if dh_elpa_test could get input
> from outside of debian/rules so that tests can be run be used
> independent of a build (e.g. dh_elpa_test --autopkgtest). For an
> example on how that can be done, you can look at the "FILES" section
> of dh_ruby(1) (package gem2deb), in special the description of
> debian/ruby-test-files.yaml, debian/ruby-tests.rake,
> debian/ruby-tests.rb, and gem2deb-test-runner(1)
> 
> you could for example make dh_elpa_test read environment variables form
> debian/dh_test_elpa.conf or something similar, and make it also have a
> --autopkgtest switch that makes sure the tests will not load code from
> the source tree, then use that switch here in autodep8:
> 
> Test-Command: dh_test_elpa --autopkgtest

If we can resolve the above issue, that's a good idea, although rather
than introduce a new config file I'm inclined to try to parse the
information out of debian/rules.

-- 
Sean Whitton
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20160511/74b7a83f/attachment.sig>