[buildd-tools-devel] Parallelising buildd

[Roger Leigh]

Hi Phil,

We've talked about this before, and I've been taking a look at the
code.  I think there could be a really easy solution to this, by
making buildd work in a multi-process fashion rather than as a
single process.

The key wait point in the mainloop is the do_build call, where we
are forced to wait until the sbuild run completes, which could be
a long time.  If we could run multiple sbuilds in parallel (which
does work well!), the problem would be solved.

This could be done by setting a parallel=n variable in the buildd
config, and before the mainloop starts creating an array of length
n.  We could modify the do_build function to fork, store the child
pid in the first free slot in the array and then return.  The child
process can then fork again to run sbuild, and wait for sbuild to
complete, and once it completes it can do the normal wanna-build
updates etc.  Meanwhile, do build returned back to the mainloop
and we can repeat the process until all the slots are full.  All
we need to do is add a waitpid(pid, &status, WNOHANG) equivalent
to check for child terminations, and then clear the relevant slots
each iteration through the mainloop.

We would have a process tree like this:

     ├─buildd─┬─buildd [do_build child 1]───sbuild
     │        ├─buildd [do_build child 2]───sbuild
     ┊        ┊
     │        └─buildd [do_build child n]───sbuild

do_build would effectively be:

if (fork()) {
  wait and then exit
} else {
  do_build as currently implemented
}


Any thoughts?  Have I missed anything obvious that precludes running
buildd like this?

One consideration would be that handling shutdown will have to
iterate though all pids, not just $main::sbuild_pid.  $main::sbuild_pid
would need replacing with @main::sbuild_pids or equivalent.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110223/01d3e293/attachment.pgp>