[debhelper-devel] dh_auto_test - Integration with CTest/CDash

[Niels Thykier]

Kyle Edwards:
> On 04/10/2018 03:27 PM, Niels Thykier wrote:
>> How would you normally do this without the Debian packaging stack?  Is
>> it something like:
>>
>>    cmake ...
>>    make/ninja build
>>    make test/ninja test
>>    cdash ... ?
>>
>> Note that cmake is not a very strong suit for me so rather assume I know
>> less rather than more.
> 
> Sort of. The dashboard script does all of those steps, so all you have
> to do is set up a cronjob that calls ctest -S /.../my_dashboard.cmake.
> The dashboard script looks vaguely like this:
> 
> ctest_start()
> ctest_configure()
> ctest_build()
> ctest_test()
> ctest_submit()
> 
> ...with a bunch of extra options added to it. However, I have spoken to
> our CDash expert, who told me I can write a simpler dashboard script
> which omits the ctest_configure() and ctest_build() steps, since those
> have already been completed by the Debian packaging stack. So it looks
> like my strategy would be to have the Debian packaging stack call the
> CDash dashboard script in the dh_auto_test rule.

Ok, sounds good.

> But that brings me to the next problem...
> 
>> If it does not contain passwords or sensitive material, then it could be
>> copied into the debian dir before the source package was built and have
>> the build use the package from debian/.  But given the nature of what it
>> is doing, I have a feeling that you will shoot down that assumption.
> 
> The dashboard scripts typically do not contain passwords - Kitware's own
> VTK CDash server is open to the public, so anyone can anonymously submit
> test results without authentication. However, the scripts DO contain
> site-specific configuration such as hostnames, and I would rather not
> pollute the Debian packaging with this information. I would like to do
> something like:
> 
> $ VTK_DASHBOARD_SCRIPT=path/to/dashboard/script.cmake gbp buildpackage
> --git-builder=sbuild
> 

You will have to "pollute" /something/ if you want site specific
information.  Either it will be the debian package (incl. d/rules), or
the build environment (e.g. the chroot or/and via a bunch of environment
variables).

They are all "unclean" but they have different trade-offs.  Generally,
though I cannot recommend fiddling with defaults or PATH in chroots -
you will eventually forgot those changes and have a hard time reproduce
a report because of this non-standard setup.

> [...]> Has anyone else done something similar to this? Uploading test results
> somewhere from dh_auto_test? Or perhaps on our end we could change CMake
> to provide better support for testing inside Debhelper?
> 
> Kyle
> 
> [...]

The best that I can think of is the dak git repo, where they use the
debian/rules to (optionally) export test (and coverage?) results in XML
for jenkins (to a file that jenkins extracted before deleting the
chroot).  However, it is a poor example because the dak git repo never
intended to build debs (that will be installed).


I am not sure how to get the file into the chroot (maybe sbuild supports
bind mounts like pbuilder/cowbuilder?).  But besides that - a custom
environment variable (with an "obviously, this will affect by build to
do non-standard things"-name) plus some if-statements in d/rules is the
best I can think of at the top of my head.

Thanks,
~Niels