[Buildd-tools-devel] sbuild --auto-give-back / sbuild<>wanna-build interaction

[Roger Leigh]

Florian Lohoff <flo at rfc822.org> writes:

> On Tue, Jan 08, 2008 at 12:24:30PM +0100, Michael Banck wrote:
>> On Tue, Jan 08, 2008 at 11:38:44AM +0100, Florian Lohoff wrote:
>> > is there a reason why sbuild has a "wanna-build" dependency? As i
>> > understand sbuild in case its been run from buildd gives back packages 
>> > when they fail building due to problems of the build environment.
>> 
>> That is usually when sbuild doesn't even get to installing the
>> build-depends, or unpacking the source package fails, IME.
>>
>
> I was just asking what the point is to split the error handling between 
> sbuild and buildd. On one end buildd takes a package from wanna-build
> and sbuild hands them back if something goes wrong. Coming from the KISS
> end of life i'd rather make it simple by letting sbuild just deal with
> building and instead of trying error handling in wanna-build return
> sensible error codes that buildd knows enough to give back the
> package.

Agreed.

> This would fail in case sbuild gets handed more than one package 
> as then the error code would be useless. This is why i ask on the
> purpose of the package batching to sbuild.

Historical reasons before our time.  When we finish integrating buildd
and wanna-build, this is one bad design decision I would like to
correct.

My current thoughts on this are:

- sbuild.git now contains buildd and wanna-build.  These need merging
  with the sbuild code to integrate better with Sbuild.pm and other
  common code.  This includes writing proper manual pages for all the
  utilities and renaming them to fit a common scheme.  It also
  involved moving some files around to be LSB compliant (mostly done).
  I envisage moving most of the common functionality of all programs
  into a whole collection of perl modules under the Buildd:: or
  Sbuild:: heirarchy.

- move most sbuild functionality into a perl module.  sbuild and
  buildd can both use this.  This will mean buildd won't need to call
  the "sbuild" program--it will just use the sbuild module.

- move most wanna-build functionality into a perl module.
  wanna-build/buildd/[sbuild] can then use this.  We can even write a
  better replacement tool while continuing to provide wanna-build for
  backward compatibility!

- Fix up Debian packaging to build buildd and wanna-build packages,
  plus packages for perl modules.

I'm sure there's a bunch of other cleanups we haven't considered yet.

>> > My guess would be that a return code from sbuild wouldn't work because
>> > buildd hands out multiple packages at once to sbuild? Is this a real
>> > performance win on any platform? Compared to build time this shouldnt
>> > matter should it?
>> 
>> At least in the distant past, all buildds hammering buildd.debian.org
>> via SSH was a serious performance issue, so taking the packages to build
>> by batch was sensible.  These days, the buildds re-use a SSH connection
>> so this might not be an issue anymore.
>
> I rather meant the buildd -> sbuild package batching not the one buildd
> <-> wanna-build. This can be limited by max_build in the config file
> but is the sbuild overhead really that huge that it makes sense?

I don't think the overhead is that great myself.  Certainly the
increased cleanliness would outweigh any overhead.

> Not hammering at wanna-build makes sense. I was too wondering
> why listing a wanna-build database with 3 packages takes ages
> on an R4k 250Mhz I2.

That's because it is a GDBM database using the horrific MLDBM frontend
(multi-level-hash in flat file).  The performance is awful, because
it's basically munging what should be several relations in a proper
relational database into a single flat file.

I would like to make the wanna-build data storage backend
configurable, with at least sqlite and postgres as options in addition
to the existing GDBM format.  The would allow us to use several
tables, including storage of the build statistics and fast and easy
queries with proper joins.  It can even store multiple distributions
and architectures without much penalty.

I originally proposed this to Ryan Murray well over five years ago,
but he strongly disagreed with this.  His rationale was that
wanna-build/buildd/sbuild should have minimal dependencies in order to
allow bootstrapping new architectures--which is fair enough.  However,
there's no reason that the more advanced backends couldn't be
/optional/ so as to still allow initial bootstrapping with a less
functional backend.  There's no reason that the rare need to bootstrap
should hold back the development more powerful, efficient, and
user-friendly tools.  Given the small number of build-deps for sqlite
(adds quilt and tcl8.4 only in total), we could probably drop MLDBM
and provide an upgrade path to sqlite via a GDBM dump and sqlite
reload.

If this is done properly, the wanna-build program interface can remain
entirely compatible with older implementations.  I'd like to put all
the database interaction in a module, with the wanna-build,
sbuild-stats and buildd use of the database all accessing it via the
module--they don't need to be aware of the backend in use.

>> BTW, this list is very probably not read by the wanna-build/buildd
>> author, it's for the end-user version of sbuild in unstable (though
>> there is some effort underway to provide wanna-build/buildd as well)
>
> I was sending simple patches to Roger lately for various aspects of
> wanna-build/buildd/sbuild and he told me to subscribe here.
>
> Making the whole autobuilding infrastructure usable for everyone
> with little effort would be a huge win. I started building
> the trunk and sid kernels in a loop on mips for example.
> I have more machines of different architectures i could simple
> let heat air or do something more sensible.

This sounds interesting.


If you are interested in working on any of the above, I'm sure your
contributions will be most welcome.  While you can of course clone the
git repo to work for the time being, we can add you to the
buildd-tools Alioth project, giving you write access to the master GIT
repo, so you can push changes into it directly.  I'd like to review
the first few patches first, though--it's currently a fairly hairly
codebase.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url : http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20080108/2f8f8423/attachment.pgp