Bug#871806: uscan: (dpkg, git-buildpackage) accept/mangle/store signed git tags in cases where upstream does not publish detached sigs on tarballs

[Daniel Kahn Gillmor]

On Fri 2017-08-11 15:09:41 -0400, Osamu Aoki wrote:
> I was just talking about similar things on
> debian-policy at lists.debian.org and Bug#811565.

thanks for the pointer!  811565 seems to be about repos without tags,
and i'm interested in repos signed tags, so i think the use cases are
slightly different, though the tools might be the same.

> Also git tag and git HEAD packaging support is currently in progress.

very interesting, though i confess to being a bit sad about not being
able to unpack the sourcecode of a package without having git itself
installed.  can you give me pointers to that?

> On Fri, Aug 11, 2017 at 02:15:28PM -0400, Daniel Kahn Gillmor wrote:
>>     git archive --format=tar --prefix=${projname}-${version} ${tagname} | gzip -9n
>
> This is already a part of uscan.  It needs a bit more refinement.

cool, it'd be even better if this was a standardized, set command so
that at most the packager supplied a few constrained parameters and that
was it.

>>  * make a shallow clone of the git archive at the tag, including the
>>    tag. (i've confirmed that a signed git tag in a shallow repo does
>>    validate correctly).
>> 
>>      git clone --bare --depth 1 -b ${tagname} \
>>             file://path/to/upstream/${projname}.git ${projname}-${version}.git
>
> The quesion is, we need to make local full clone if git is served dumb
> http server.  (Github is smart :-)  Shallow clone or git archive does
> not work if git server is dumb.

In my experience, if the goal is to create a minimal "shallow clone
snapshot" of the git archive, it's quite often the case that i as the
packager already have a full clone of the upstream repo i'm working on.

So if i'm working in a local copy /home/dkg/src/foo/foo and i've just
seen upstream's tag v3.44, i might do the following to create a "shallow
clone snapshot":

      git clone --depth=1 --bare -b v3.44 file:///home/dkg/src/foo/foo foo-3.44.git

This is the equivalent of an orig.tar.gz (i think you can use the above
git archive command against it), but it has the advantage of potentially
also including the signed tag.

>>  * write a simple tool to verify an orig.tar.gz against a signing key
>>    and a gtsig, by extracting the "shallow clone" of the git repository,
>>    verifying git tag -v, using git-archive, and then comparing the
>>    results.
>
> Is this simple?  Please show me working example as shell log.

I did some experimenting with the "hddemux" package, which is a package
with a very small source tarball, which i'm upstream on, and which is
hosted on collab-maint and on gitlab and on 0xacab.

I made the .gtsig of hddemux 0.3 with:

    git clone https://gitlab.com/dkg/hddemux
    git clone --depth=1 --bare -b hddemux-0.3 file://$(pwd)/hddemux hddemux-0.3.git
    tar cz hddemux-0.3.git > hddemux-0.3.gtsig
    rm -rf hddemux hddemux-0.3.git

I was thinking of something like this:

     tar xz < hddemux-0.3.gtsig 
     ( cd hddemux-0.3.git && \
       git tag -v hddemux-0.3 && \
       git archive --format=tar --prefix=hddemux-0.3/ hddemux-0.3 | sha256sum )
     rm -rf hddemux-0.3.git/

and testing whether (a) the tag verified, and (b) the the output of
sha256sum matches against:

     zcat < hddemux_0.3.orig.tar.gz | sha256sum -

(above, i'm testing the digest of the uncompressed tarball.  testing the
compressed version would also be pretty easy)

>> Some of the outstanding concerns:
>> 
>>  * what if there is non-DFSG-free data in the upstream git repo?  We
>>    want to make sure we avoid shipping it to our mirrors.  that's why i
>>    was leaning toward the "shallow clone", but if there are other
>>    techniques, i'd be curious to hear them.
>
> gtsig is from upstream on git repo.  So check it there, I think. (Not on
> tar.)
>
> If we do non-DFSG-free tarball generation, maintainer need to sign
> the tarball.

my goal here is to be able to verify (and store and track) the upstream
maintainer's signatures.

Many upstream maintainers can be convinced to make the HEAD of their
repo DFSG-free -- they understand that some weird jpg some previous
developer committed a few years ago isn't really kosher.

but that doesn't mean that their history is DFSG-free.  My concern was
about avoiding shipping non-DFSG-free history, when the tag we want to
ship is itself DFSG-free.

>>  * the .gtsig will be quite large -- roughly the same size as the
>
> I am not quite sure about what you mean here.

the gtsig contains a snapshot of all the files in the git repo.  it's
not a free-floating signature, it's equivalent to a tarball with an
embedded signature and a bunch of dangling pointers to previous git
history.

The attached hddemux-0.3.gtsig is 42000 octets.  hddemux_0.3.orig.tar.gz
is 31329 octets.  Does that make sense?

  --dkg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: hddemux-0.3.gtsig
Type: application/octet-stream
Size: 42000 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/devscripts-devel/attachments/20170811/cb4f691b/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/devscripts-devel/attachments/20170811/cb4f691b/attachment-0001.sig>