[buildd-tools-devel] Parallelising buildd

[Roger Leigh]

On Thu, Feb 24, 2011 at 01:25:18AM +0100, Philipp Kern wrote:
> Hi,
> 
> On Wed, Feb 23, 2011 at 11:42:47PM +0000, Roger Leigh wrote:
> > We would have a process tree like this:
> > 
> >      ├─buildd─┬─buildd [do_build child 1]───sbuild
> >      │        ├─buildd [do_build child 2]───sbuild
> >      ┊        ┊
> >      │        └─buildd [do_build child n]───sbuild
> 
> What's the advantage to forking, doing some bookkeeping and directly exec'ing
> sbuild?  I would like to have a single buildd process around, not multiple
> ones.  Killing the parent process should distribute the signal to its builder
> childs properly.

We could do this either way.  The advantage to the separate process is
that we can handle the wanna-build interaction post-build in exactly
the same way as the current do_build, without needing to integrate that
into the mainloop.  i.e. to keep code changes to a minimum.  We also
have the option of using Sbuild::Build directly, rather than execing
the sbuild executable.  Doing this would give buildd direct access to
the build metadata, without the need for implementing communications
protocols between sbuild and buildd.  (I've already written the
RFC822-style code, so we have either option here.)

> Apart from that, if parallel=n is correctly weighted against the free space in
> the LVM VG it should work and would be helpful on some buildds.  And maybe it
> should behave similar to the old "spawn a second buildd instance if the
> backlog is greater than N", i.e. increase parallel sbuild count if you can't
> keep up.  It should also enable us to pass other build options, to
> reduce parallelity in package builds when multiple sbuilds are spawned.
> Otherwise the machine will either schedule or swap itself to death.

Agreed.  We could include code to evaluate disc space, load average,
free memory/swap and take these factors into account when scheduling a
new build.  Given the very different disc/cpu/memory requirements of
different packages, it might be worthwhile to get sbuild to record the
resource usage of a build in more detail so we can take the resource
usage of previous builds into account.  Our current measurement of
disc usage is poor--just the final build tree size, not maximum usage.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110224/c1180015/attachment.pgp>