[buildd-tools-devel] Bug#731940: buildd.debian.org: consider stderr as well as stdout before timeout

[Johannes Schauer]

Hi,

On Wed, 11 Dec 2013 14:07:14 +0100 Nicolas Boulenguez <nicolas at debian.org> wrote:
> Some (admittedly faulty) build tools produce messages on stderr instead of
> stdout.

I don't think it can be called faulty to produce messages on stderr. I'd not
want, for example a progress meter on stdout because that would mean that the
actual program output on stdout that I might want to pipe to another process
gets destroyed.

So I'd argue that status messages should especially sent to stderr and not
stdout. And it would then follow that for sbuild to check whether the build is
still alive it should especially also consider stderr.

> However, sbuild only considers stdout to detect timeout and kill inactive
> builds.
> 
> This results in hard to diagnose failures, as the log (rightfully) mixes
> stout and stderr.

This is weird because in the man page of sbuild.conf it says in the section of
STALLED_PKG_TIMEOUT:

>> Time (in minutes) of inactivity after which a build is terminated. Activity
>> is measured by output to the log file.

stdout and stderr are mixed in the log file so the above makes me believe that
stderr would be considered as well as stdout for the timeout.

> For an example, the gnat-gps package has been randomly failing to build from
> scratch for years. Various solutions have been tried for years, causing this
> big package to be reuploaded and rebuild for years.  (of course, gnat-gps is
> now corrected)

In botch, which builds very slowly on architectures without support for native
OCaml, I use the following Makefile wrapper around the slow parts to make the
build succeed:

sh -c 'starttime=`date +%s`; \
	while sleep 300; do \
		now=`date +%s`; \
		diff=$$(((now-starttime)/60)); \
		echo "running for $$diff minutes and still alive"; \
	done' & PID=$$!; \
do_some_stuff_that_takes_ages_on_the_wrong_architectures; \
kill $$PID

I'd love to drop this bit because if the build is *really* stalled then it will
wrongfully never quit.

> Please avoid this waste of human and computer time with something like
> 
> line 1500 of lib/Sbuild/BUILD.pm.
> -my $pipe = $self->get('Session')->pipe_command($command);
> +my $pipe = $self->get('Session')->pipe_command(
> +        { COMMAND => [$command],
> +          STREAMERR => \*STDOUT});

That patch is not correct anymore and will only result in a failure when trying
to run sbuild with it. The following works:

--- a/lib/Sbuild/Build.pm
+++ b/lib/Sbuild/Build.pm
@@ -1595,7 +1595,8 @@ sub build {
        USER => $self->get_conf('BUILD_USER'),
        SETSID => 1,
        PRIORITY => 0,
-       DIR => $bdir
+       DIR => $bdir,
+       STREAMERR => \*STDOUT,
     };
 
     my $pipe = $self->get('Session')->pipe_command($command);


Please consider applying it to sbuild.

Thanks!

cheers, josch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20150716/f2c525ad/attachment.sig>