[PKG-Openstack-devel] ongoing OpenStack packaging in Debian

[Allison Randal]

On 02/21/2017 07:31 PM, Thomas Goirand wrote:
> On 02/21/2017 07:58 PM, Allison Randal wrote:
> 
>> The "deb-" prefixed repositories are what OpenStack Infra calls "managed
>> projects", meaning that they automatically sync changes from the
>> mainline OpenStack repositories. Which simplifies things for us.
> 
> You should also keep in mind the generalization. The packages don't all
> have git.openstack.org as upstream. Sometimes, it's github, or others,
> for 3rd party libs.

Nod, I was focused on the OpenStack project source code when I met with
Paul, but it is worth documenting that general Python dependencies are
different.

>> When you submit a patch to a "deb-" repo in Gerrit, the package gets
>> rebuilt in OpenStack CI/CD in the check pipeline and the gate pipeline.
>> The package is only built (on an untrusted worker for safety), it
>> doesn't run any tests or lintian checks.
> 
> Not correct. There's lintian checks, and the build is failed if the
> package doesn't pass the lintian checks.

Great, I'll make sure that's noted in the docs.

>> The gate build uploads the built packages to:
>>
>> http://tarballs.openstack.org/packaging-deb/
> 
> Technically, the gate build, but it's the POST job that builds and
> upload to tarballs.o.o.

Thanks for the detail, I'll note it in the docs.

>> In subdirectories named for the repository built, and the git hash of
>> the specific build, for example:
>>
>> http://tarballs.openstack.org/packaging-deb/deb-nova/uploads/2fb1c3bc96a52721b6f05bbdfaf0306b8705ac2e/
>>
>> These packages are signed by the Infra key
> 
> Nop, they aren't signed with the infra key. They are signed with a key
> which is discarded after build, so the signing part is something that
> would needs to be rethought. That is, if we care about it. In fact, I
> don't think we need to care about package signatures: the signature of
> packages is never used for any check, only the *repository* key is.

Nod, I'll make the documentation more precise. The important point is
that it's not signed with a key that could be used to upload packages to
Debian, so these are a kind of "disposable" packages.

>> and then loaded into a
>> Debian mirror used by OpenStack Infra for testing. This mirror has two
>> archive areas for each OpenStack release, one for the packages
>> maintained by OpenStack PKG, and the other for custom backports of
>> dependencies for those packages. For Newton these are:
>>
>> http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton/
>> http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton-backports/
>>
>> (It also hosts a full unmodified mirror of Jessie, for use by the build
>> slaves at http://mirror.dfw.rax.openstack.org/debian/dists/)
> 
> Which unfortunately can't be used by openstack pkg (it'd be useful for
> example when building the chroot), because that Jessie mirror is made
> with reprepro, and therefore isn't signed. :(

Nod.

>> Paul needs to do some manual work for us here, because the directories
>> for Ocata packages and backports have to be created before we can start
>> publishing packages.
> 
> Correct. This was discussed in Barcelona. Basically, it would just be a
> reprepro command to type, to import all Newton packages into the Ocata
> repository. Thanks to the pool structure, this only adds indices files,
> we don't need to actually copy all packages to a new place.

In theory, this is now automated when the first package makes it through
the gate to be published to the jessie-ocata mirror. (More details at
the end of this message.)

>> The CI/CD process is not running integration tests on packages, so ther
>> are no cross-package dependencies. But, from what I understand, Thomas
>> was manually timing his changes to Gerrit to control for dependencies.
> 
> I'm not sure what you're saying here.
> 
> For running tempest tests and checking everything works, I was running
> them on a separate hardware using a Debian live system. All of it is
> included in the source packages (see openstack-meta-packages). I don't
> think it would be too hard to move it all to a VM in infra, though I
> doubt it'd be useful to run the tempest tests on each build (this would
> be too resource intensive and would take too long to run).

I asked Paul a question about timing uploads of dependencies. For
example, whether I would need to +A all the updated dependencies for
nova before we +A the nova package. So, this was his answer to my
question, not a more general comment.

>> Also, sometimes, this whole process can be quite slow. The patch builds
>> 3 times before it lands, which can take a long time if the gate is under
>> heavy usage. It might only take 10 minutes for a change to get into the
>> gate, but take an entire day to get the package published.
> 
> I *never* took a day.
> 
> The quickest build I could observe was about 7 minutes, on which maybe 6
> are spent creating the build environment (ie: installing sbuild,
> setting-up the chroot, etc.). There's room for improvement, and I'm sure
> we could get it as quick as 1 or 2 minutes for the simplest packages. I
> didn't have time to focus on this, as I was in a hurry to deliver Newton
> on time, which I did.

He was talking about delays in the gate itself, that package publishing
was sometimes delayed because it was a lower priority job than others.
IIRC, he mentioned that you sometimes complained about how slow package
publishing was.

> Then, there's the fact that packages are built once when someone sends a
> change request, then once for the gating, and once more for the POST
> job. Only the files from the 3rd build are used to publish the actual
> package. I've been told by the infra team that, with zuul 3, we would be
> able to keep the files from the gating and use them for the POST job,
> which would reduce the number of build to 2 only. IMO, that's the only
> improvement we should attempt.

Nod, he mentioned that.

>> That's all for now. I'm working on getting a first few Ocata packages
>> through, and will let you know how that goes.
> 
> Once Paul has enabled the Ocata repo, the first patch that should land
> is for openstack-pkg-tools, so that it uses the correct debian/ocata
> branch. So branch openstack-pkg-tools to debian/ocata, and work on it
> there. A simple "grep -r newton *" will IMO be enough to get you going.
> Then the new openstack-pkg-tools will be pushed to the newly created
> ocata repo, and you should be good to go for working on other packages.
> 
> If you need me for more explanation, let me know, I'll be very happy to
> help if I can.

The patch to enable the Ocata repos has merged.

The patch to update openstack-pkg-tools is pending. The first version
failed, because it expected jessie-ocata-backports to already exist. I
chatted with Paul, and he suggested doing the update to
openstack-pkg-tools in two stages: first update should keep using
jessie-newton-backports (for apt/sources.list.d), and once that passes
the gate, the package build will trigger the creation of jessie-ocata
and jessie-ocata-backports. The second update to openstack-pkg-tools
will then switch to using jessie-ocata-backports (for
apt/source.list.d). So, once this change is +A and merged:

https://review.openstack.org/437750

We can see if the mirrors for jessie-ocata and jessie-ocata-backports
were created properly.

Thanks for the corrections and clarifications,
Allison