Bug#871806: uscan: (dpkg, git-buildpackage) accept/mangle/store signed git tags in cases where upstream does not publish detached sigs on tarballs

[Osamu Aoki]

On Fri, Aug 11, 2017 at 06:59:02PM -0400, Daniel Kahn Gillmor wrote:
> On Fri 2017-08-11 15:09:41 -0400, Osamu Aoki wrote:
> > I was just talking about similar things on
> > debian-policy at lists.debian.org and Bug#811565.
> 
> thanks for the pointer!  811565 seems to be about repos without tags,
> and i'm interested in repos signed tags, so i think the use cases are
> slightly different, though the tools might be the same.

Repo with unsigned tag is already somewhat working.

The trick to read commit date of HEAD has good overlap with reading
signed tag and verifying it against file content hush.
 
> > Also git tag and git HEAD packaging support is currently in progress.
> 
> very interesting, though i confess to being a bit sad about not being
> able to unpack the sourcecode of a package without having git itself
> installed.  can you give me pointers to that?

Its my local activity ... 

> > On Fri, Aug 11, 2017 at 02:15:28PM -0400, Daniel Kahn Gillmor wrote:
> >>     git archive --format=tar --prefix=${projname}-${version} ${tagname} | gzip -9n
> >
> > This is already a part of uscan.  It needs a bit more refinement.
> 
> cool, it'd be even better if this was a standardized, set command so
> that at most the packager supplied a few constrained parameters and that
> was it.
> 
> >>  * make a shallow clone of the git archive at the tag, including the
> >>    tag. (i've confirmed that a signed git tag in a shallow repo does
> >>    validate correctly).
> >> 
> >>      git clone --bare --depth 1 -b ${tagname} \
> >>             file://path/to/upstream/${projname}.git ${projname}-${version}.git
> >
> > The quesion is, we need to make local full clone if git is served dumb
> > http server.  (Github is smart :-)  Shallow clone or git archive does
> > not work if git server is dumb.
> 
> In my experience, if the goal is to create a minimal "shallow clone
> snapshot" of the git archive, it's quite often the case that i as the
> packager already have a full clone of the upstream repo i'm working on.

Yah but tool needs to be general and logic needs to be simple for
maintenance.  uscan is already very complicated...

> So if i'm working in a local copy /home/dkg/src/foo/foo and i've just
> seen upstream's tag v3.44, i might do the following to create a "shallow
> clone snapshot":
> 
>       git clone --depth=1 --bare -b v3.44 file:///home/dkg/src/foo/foo foo-3.44.git
> 
> This is the equivalent of an orig.tar.gz (i think you can use the above
> git archive command against it), but it has the advantage of potentially
> also including the signed tag.
> 
> >>  * write a simple tool to verify an orig.tar.gz against a signing key
> >>    and a gtsig, by extracting the "shallow clone" of the git repository,
> >>    verifying git tag -v, using git-archive, and then comparing the
> >>    results.
> >
> > Is this simple?  Please show me working example as shell log.
> 
> I did some experimenting with the "hddemux" package, which is a package
> with a very small source tarball, which i'm upstream on, and which is
> hosted on collab-maint and on gitlab and on 0xacab.
> 
> I made the .gtsig of hddemux 0.3 with:
> 
>     git clone https://gitlab.com/dkg/hddemux

I see full clone 1st ...

>     git clone --depth=1 --bare -b hddemux-0.3 file://$(pwd)/hddemux hddemux-0.3.git
>     tar cz hddemux-0.3.git > hddemux-0.3.gtsig

You just tar-up everything???? I thought you are going to check
signature of tag against content...

>     rm -rf hddemux hddemux-0.3.git
> 
> I was thinking of something like this:
> 
>      tar xz < hddemux-0.3.gtsig 
>      ( cd hddemux-0.3.git && \
>        git tag -v hddemux-0.3 && \
>        git archive --format=tar --prefix=hddemux-0.3/ hddemux-0.3 | sha256sum )
>      rm -rf hddemux-0.3.git/

Does hush given tar exactly the same as one signed in tag ... I guess I
need to RTFM.

> and testing whether (a) the tag verified, and (b) the the output of
> sha256sum matches against:
> 
>      zcat < hddemux_0.3.orig.tar.gz | sha256sum -
> 
> (above, i'm testing the digest of the uncompressed tarball.  testing the
> compressed version would also be pretty easy)

Compression depends on compression parameter which may be different for
different arch.  Are you sure you tested your thought works???

> >> Some of the outstanding concerns:
> >> 
> >>  * what if there is non-DFSG-free data in the upstream git repo?  We
> >>    want to make sure we avoid shipping it to our mirrors.  that's why i
> >>    was leaning toward the "shallow clone", but if there are other
> >>    techniques, i'd be curious to hear them.
> >
> > gtsig is from upstream on git repo.  So check it there, I think. (Not on
> > tar.)
> >
> > If we do non-DFSG-free tarball generation, maintainer need to sign
> > the tarball.
> 
> my goal here is to be able to verify (and store and track) the upstream
> maintainer's signatures.

I am not familiar with this but I thought I use command like the
following to check signature.

 $ git verify-tag ...

Can you point me to any resource which indicate your way of verifying
tag via tar is the right way...  Please educate me on this.
 
> Many upstream maintainers can be convinced to make the HEAD of their
> repo DFSG-free -- they understand that some weird jpg some previous
> developer committed a few years ago isn't really kosher.

Many ... yes,  All ... no.

uscan is a tool.  Why limit its capability?

> but that doesn't mean that their history is DFSG-free.  My concern was
> about avoiding shipping non-DFSG-free history, when the tag we want to
> ship is itself DFSG-free.

Why send entire history?

> >>  * the .gtsig will be quite large -- roughly the same size as the
> >
> > I am not quite sure about what you mean here.
> 
> the gtsig contains a snapshot of all the files in the git repo.  it's
> not a free-floating signature, it's equivalent to a tarball with an
> embedded signature and a bunch of dangling pointers to previous git
> history.
> 
> The attached hddemux-0.3.gtsig is 42000 octets.  hddemux_0.3.orig.tar.gz
> is 31329 octets.  Does that make sense?

I feel strange .... At least creating such big file to veriy its
current content beat the purpose of making cryptographic signature.

When I heard you talking signature, I thought you are talking to create
a file with a signature on a tagged contents.  But here you are creating
something different.  To be honest, my knowledge is weak on this.  But I
am starting to feel you nor me have concrete idea what needs to be
done.

Osamu