[buildd-tools-devel] Bug#778571: Bug#778571: [Reproducible-builds] Bug#790868: sbuild: Please allow sbuild to use a deterministic build path to build packages

[Roger Leigh]

On 14/07/2015 19:28, Johannes Schauer wrote:
> Hi,
>
> Quoting Benjamin Drung (2015-07-14 21:19:51)
>> The idea to bind mount the directory is to avoid build conflicts. When you
>> build the same package twice in parallel, one chroot would bind mount
>> /build/$package-XXXXXX/$package-$version to
>> /usr/src/debian/$package/$full-version and the other
>> /build/$package-YYYYYY/$package-$version to
>> /usr/src/debian/$package/$full-version. Does this make sense or do I have a
>> flaw in my logic?
>
> maybe I have a flaw in mine :) So let me inquire about this last thing you
> wrote.
>
> Do you mean to bind mount /build/$package-XXXXXX/$package-$version from the
> host running sbuild to /usr/src/debian/$package/$full-version in the schroot
> where the build is done? If yes, then why would I want the /build directory on
> my host system? That would violate the FHS. Secondly, who would be responsible
> to set up /build/$package-XXXXXX and /build/$package-YYYYYY on my host system?
> Which permissions would that require?
>
> So maybe you mean it the other way round, to bind mount
> /usr/src/debian/$package/$full-version from the host running sbuild to
> /build/$package-XXXXXX/$package-$version in the schroot where the build is
> done? If yes, then what is XXXXXX? If it is random, then how would it make the
> build path reproducible? If it is not random, why does the XXXXXX part exist?

/build should never, ever, exist on the host system.  It exists only in 
the build chroot.  The reason?  It needs to be somewhere, and it needs 
to be short (long path lengths cause problems with unix sockets).  There 
aren't really any FHS considerations in our private environment, and 
even if there were the shortness is paramount.

IIRC, The -XXXXXX is historically not to protect against parallel builds 
inside the build chroot.  It's to protect against clashes on the host if 
you do one or more of:
- multiple builds of the same package+version (parallel or sequential)
- multiple builds of the same package+version for different 
architectures (parallel or sequential)
- preservation of the build directory for debugging without later builds 
trashing it

We used to encode all this information in the filename, but there were 
complaints about it (and it also had path length problems).

To have a reproducible build, you only need a stable name inside the 
build chroot.  You can call it whatever you like on the host.  As an 
example:

/var/lib/sbuild/build/foobar-1.2.3-1-powerpc-201507142014-XXXXXX -> 
/build/foobar-1.2.3-1

The only hard part is who has responsibility for the bind mount since 
this requires root privs.  Current sbuild uses schroot to bind 
/var/lib/sbuild/build -> /build.  But you want it to be more precise 
than that.  And a symlink is no good either, since you share the 
directory and this would clash potentially.  I think schroot 1.7 with 
user-modifiable keys would help here.  You could then invoke schroot 
internally with -o 
sbuild-hostpackagedir=/var/lib/sbuild/build/foobar-1.2.3-1-powerpc-201507142014-XXXXXX 
sbuild-packagedir=/build/foobar-1.2.3-1 and have a custom setup script 
handle the binding using SBUILD_HOSTPACKAGEDIR and SBUILD_PACKAGEDIR.

We would need to release a 1.7 schroot for that.  I haven't worked on it 
in recent months.  Last time I looked it was perfectly functional but 
had more features I wanted to add before making a 1.8 release, but those 
can be pushed to a later version.  It most likely needs any missing 
commits picking from schroot-1.6 and more testing.  And a new uploader 
since I no longer have privileges for that.

https://github.com/codelibre-net/schroot/commits/schroot-1.6
https://github.com/codelibre-net/schroot/commits/master


Regards,
Roger