[buildd-tools-devel] sbuild just hangs

[Roger Leigh]

On 31/07/2015 05:58, Johannes Schauer wrote:
> Hi,
>
> Quoting Roger Leigh (2015-07-31 01:06:56)
>>> Secondly, whenever I saw somebody use the -d argument, they would just use
>>> it as a way to select the chroot to build in. But in addition to that it
>>> will also override the value set in debian/changelog, thus possibly leading
>>> to wrong uploads as the argument to -d is put in the resulting .changes
>>> file. The sbuild man page doesn't even mention this effect of the -d option
>>> which is probably why it's used so much. The help text in the configuration
>>> file explains better what $distribution does but the sbuild wiki even goes
>>> so far to suggest putting $distribution='sid' in the configuration file
>>> which means that unless the user is really aware of the implications of -d,
>>> they will always create .changes file with sid as the distribution no
>>> matter what they put in debian/changelog.  Is this "feature" of putting a
>>> different value in the resulting .changes file than the maintainer put in
>>> their debian/changelog useful for anything?
>>
>> Yes.  In the general case for autobuilding, we don't have the
>> distribution until after the build starts--it comes from the changelog
>> not the dsc, so we don't have the information until after we're already
>> running in the chroot.  This could be rearchitected, but it would be
>> quite painful and come with a significant cost since you'd need to copy
>> the downloaded source into the build chroot; and when all the
>> information you have is foo_version you can't even download it without
>> the apt package list from inside the chroot, so you have to make an
>> up-front decision about the chroot to use before you even get the
>> source.  This even applies to local sources--if you build from a .dsc,
>> you have to unpack it to get this information, and that could be very
>> expensive for large source packages.
>
> Okay, I understand that this would be hard to do when using sbuild so that it
> `apt-get source` the source package to build itself but I'd argue that this
> really only is done by autobuilders which will only have one proper
> configuration they will never change anyways.
>
> As a normal developer, you are interested in building the source you already
> have downloaded, either in its still packed or already unpacked state. But even
> in its packed state, is it not easy to acquire the distribution in the very
> beginning, just by doing something along the lines of:
>
> 	tar --to-stdout -xf package_version.debian.tar.xz debian/changelog
>
> And at that point you could then state in the documentation that the only
> situation in which you might want to use the -d option is if sbuild is
> instructed to retrieve the source itself.

This could certainly work, though it's not a complete solution--what 
about .diff.gz?  Are there additional variants we need to cope with?  Is 
it liable to break with future packaging format changes?

The point I want to make here is that while a quick hack to extract it 
is *possible*, it needs to be *absolutely robust* in every case.  And 
since that likely requires unpacking with dpkg-source it would become 
very expensive.  Please don't add hacky circumventions of the standard 
tools for processing the package formats; this will inevitably lead to 
problems, and would be bad practice.  Some of the improvements I made to 
sbuild were to remove such problematic hacks and use the standard 
tools--they always come back to bite.

> And in that case, could one not deprecate sbuild's rewriting of the
> distribution given in debian/changelog with the -d option? Instead, how about
> throwing an error if the distribution given in debian/changelog ends up being
> different from the chroot? This disparity would not happen often anymore since
> calling `apt-get source` from inside a sid chroot probably gets you a sid
> source and if it doesn't then it's an error.

Possibly.  I think you would need to investigate the history and 
rationale behind that rewriting before changing it.

> You also didn't answer my question from above: in which scenario would anybody
> using sbuild want that the distribution in the output .changes be different
> from the one in debian/changelog?

I'm not sure of any good reasons off the top of my head, but it's likely 
that there are reasons for it.

Just a word of caution: sbuild is old, and has a long history.  I have 
attempted to clean it up somewhat over the years, and have removed a lot 
of obsolete cruft.  But it's often hard to tell for a given feature 
whether it's actually useful or if it's obsolete without looking back in 
the history.  But there are a lot of people using sbuild for all sorts 
of varied purposes, and more often than not people *do* use many of the 
more esoteric features and will shout if they are broken or removed, so 
I would advise against removing things just because the reason for their 
use isn't immediately obvious; definitely check thoroughly first and 
then think again before deleting it.  Likewise, please be very cautious 
about changing the behaviour or options in use--there are a lot of 
people using sbuild who would find their workflows break.

>>> But if there is no real world use for putting a different distribution
>>> value in the generated .changes file than in the input debian/changelog,
>>> then how about deprecating this functionality?
>>
>> If there's a simple way to do this without any major downsides, I'm sure
>> it would be an improvement.  But as I described above, getting the input
>> debian/changelog is the difficult part.  I would have implemented this
>> as the default years ago if it was a simple change.
>
> So you are saying that *not* changing the distribution value of
> debian/changelog to something entirely different in the output .changes would
> not break anything? This is a different question than which chroot to use.

Not sure I follow.

>>> Thirdly, I think that the default sbuild configuration should do the right
>>> thing when building Debian packages. This means, that even if the target
>>> release in debian/changelog is UNRELEASED, then sbuild should default to
>>> build for "sid" (or another default value settable in the sbuildrc). If the
>>> distribution in debian/changelog is sid, then sbuild should also consider
>>> the chroot called "unstable" and the other way round.
>>
>> I once implemented exactly this functionality, and I was castigated for
>> it when people uploaded packages unintentionally built against the wrong
>> distribution, so it was backed out rather rapidly.  The intention was
>> good, and it even warns you about it in colour that there's a mismatch,
>> but it's inevitable that people will fail to see it and also fail to
>> check the .changes.
>
> Again, the problem seems not about chroot selection but about sbuilds behaviour
> to change the distribution in the output .changes file. Why would sbuild not
> just keep UNRELEASED in that field if that was the value in debian/changelog?

It probably comes down to the behaviour of always manually specifying 
the distribution.  You would need to investigate the historical reasons 
for doing this before changing anything.

>> I think you could make a case for UNRELEASED defaulting to unstable.  For
>> everything else it's risky ground.  I don't think using sid is generally
>> advised?  Better to use unstable everywhere?
>
> I don't know about any resource that claims that one is preferred over the
> other.

dch has always used unstable AFAICT, if that's any guide.  I've never 
seen anyone using "sid" in the last 18 years, so I would suggest that 
this is not typical usage by any means.

>>>> Note that you can add "aliases=unstable-amd64-sbuild" or equivalent for
>>>> your distribution/arch to your schroot configuration, which will make it
>>>> select the appropriate distribution when you use "-d unstable".
>>>
>>> Fifthly, I think this should be the default behaviour for the sid/unstable
>>> pair.  I'm just unsure whether schroot should set this default or whether
>>> sbuild should translate sid to unstable and the other way round?
>>
>> It's not the job of schroot to impose any policy here.  It knows nothing
>> about sbuild.  On the sbuild side, sbuild-createchroot is the place
>> where such mappings could be added as aliases at initial debootstrap of
>> the chroot.  But as a general point, I would question the use of "sid"
>> in place of "unstable".  Just add the "unstable", "unstable-amd64" or
>> "unstable-amd64-sbuild" alias(es) and it will automatically select the
>> right chroot.  However, automatically adding the unstable-amd64-sbuild
>> alias if you debootstrap a sid chroot is absolutely fine--though it
>> might need a check for the name already being in use.
>
> Okay, but is sbuild not supposed to potentially also work with other backends
> than schroot in the future? Would you also say that if other backends are added
> later then it should be them to do the mapping between unstable/sid?

The backends just provide access to the virtualised build environments. 
  In the case of the sudo backend, it's just a directory of symlinks. 
They are all simple--it's not their place to provide or impose any sort 
of policy.  You specify a distribution to build for, and you get that. 
No more or less.

I think this is a bit of a problem in search of a solution.  It could be 
made slightly nicer by automatically adding additional aliases with 
sbuild-createchroot so you can use sid or unstable, and for stable and 
testing we could add -proposed-updates as well, etc.  But for codenames, 
it's not realistic to maintain a mapping of them in the tool, so the 
logical names should be preferred.  Note that sbuild is used in 
downstream distributions and by people outside Debian doing custom 
repos, so it needs to be kept generic.  Historically it hardcoded all 
this and I ripped it out due to the breakage and inflexibility it caused.


Regards,
Roger