[PKG-Openstack-devel] [MoM] Packaging manila

[Andreas Tille]

Hi,

sorry for the delay - I was a bit busy.

BTW, I was working down the mail step by step and wrote things that I'll
rewrite later.  May be the interesting parts come after the string
"[tag-based-packaging]"

On Thu, Apr 23, 2015 at 12:24:12PM +0430, Malihe Asemani wrote:
> I intend to report in this email what i did these days, on MOM project.

Good.  I'd suggest more frequent and shorter reports.  In other MoM
projects this has lead to more efficient cooperation between the student
and me.
 
> On Mon, Apr 13, 2015 at 2:24 PM, Andreas Tille <andreas at an3as.eu> wrote:
> >
> > Ahhh, fine.  Perhaps I should have read this page more thoroughly.  I
> > did not intended a competing method.  Malihe, in case you decide to
> > recreate the manila repository I'd advise to check out this method which
> > is established in the openstack team.
> >
> Okay. I'll recreate it using this script.
> I installed the openstack-pkg-tools package on my system but could not find
> a command which its name is pkgos-alioth-new-git. Also, I could  not find
> the script on source code. finally, i found it out that there is a link to
> the script on https://openstack.alioth.debian.org/.

I admit I do not see this link myself.  Thomas, could you perhaps put
this more prominently.  Newcomers seem to have trouble finding this
script.  (I can confirm that it is not part of openstack-pkg-tools
package.)
 
> Then, I tried to create a Test rgit reo, using the script. The created git
> repository is not observable on http://git.debian.org/openstack.  Is this
> the issue that Thamos has mentioned it, before?
>    "This should be slightly adapted if you're not add (to handle a
> different username with "-     guest" at the end). "

I guess the web display on git.debian.org is OK since the repository does
not exist at all:

   moszumanska:~$ ls /git/openstack/* | grep manila
   moszumanska:~$
 
> If they are not related to the same issue, would you guide me why may repo
> wont be observable if i use the script? Moreover would you please explain a
> little about what is the mentioned problem by Thomas?

I think it might be a good idea to use the script with "set -x" to see
where it fails and in case of problems do all the steps manually.  If
you send me the link where you downloaded the script I could check
myself but I guess it will be no magic and contains some "git init" etc.
stuff that could be done manually for sure.
 
> The script uploads the local git repository on Alioth.  So I needed to
> create my local Manila, first.
> Therefore, I tried different ways for creating initial commit on Manila
> repo.
> I could choose between 1) last version of Manila on github (tag:
> 2015.1.0rc1), 2) Manila source code on Vivid, and 3) the last uploaded
> version on Manila (which James had mentioned it on the other e-mail).
> I downloaded Manila on github, applied the process which Thomas had
> mentioned it before, and finally, I needed to add debian directory to it.
> I checked the Vivid version and it was older than the version which James
> mentioned it. So, I decided to use 'pull-lp-source manila' from
> ubuntu-dev-tools and use the last version* .* i had a brief check on Bazar to
> find out how can i find ubuntu-Manila is based on which version of
> github-Manila. It was based on 2015.1.0rc1 (the last tag).

Please always keep in mind that I do not know the relation between the
launchpad and the github, but *I* consider it good practice to use the
version that is released by upstream and as far as I can see upstream is
at github.  To fetch the latest version from github you can use a
debian/watch file with the following content:


version=3
opts="uversionmangle=s/([\d.]*)([a-zA-Z])/$1~$2/;s/\.0b/~b/" \
  https://github.com/openstack/manila/releases .*/archive/(\d[\d.-brc]+)\.(?:tar(?:\.gz|\.bz2)?|tgz)


If you do this and go to the main packaging dir (== the dir where you
see the content above if you do `cat debian/watch`) than you can do

 $ uscan --verbose --report
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   opts="uversionmangle=s/([\d.]*)([a-zA-Z])/$1~$2/;s/\.0b/~b/"   https://github.com/openstack/manila/releases .*/archive/(\d[\d.-brc]+)\.(?:tar(?:\.gz|\.bz2)?|tgz)
-- Found the following matching hrefs:
     /openstack/manila/archive/2015.1.0rc1.tar.gz (2015.1.0~rc1)
     /openstack/manila/archive/2015.1.0b3.tar.gz (2015.1.0~b3)
     /openstack/manila/archive/2015.1.0b2.tar.gz (2015.1.0~b2)
     /openstack/manila/archive/2015.1.0b1.tar.gz (2015.1.0~b1)
     /openstack/manila/archive/2014.2.tar.gz (2014.2)
     /openstack/manila/archive/2014.2.rc2.tar.gz (2014.2.~rc2)
     /openstack/manila/archive/2014.2.rc1.tar.gz (2014.2.~rc1)
     /openstack/manila/archive/2014.2.b3.tar.gz (2014.2.~b3)
Newest version on remote site is 2015.1.0~rc1, local version is 2015.1~b2
 => Newer version available from
    https://github.com/openstack/manila/archive/2015.1.0rc1.tar.gz
-- Scan finished

If you do

 $ uscan --verbose --force-download

you get what *I* would start packaging (and I repeat that I'm quite
clueless about OpenStack in general and Manila in special and so I have
no idea whether it is a good idea to use "~b" but "~rc" seems to be even
a release candidate from upstream which might be fit for packaging.

> Now, I dont know as the first commit, I should add a debian dir which is
> added using *dh-make*?

I *personally* hate dh-make since it leaves you alone with a lot of
useless templates you need to delete afterwards.  Since I'd regard this
as a very dumb work and I've seen so many not deleted templates that
just leave noise I always recommend to start with some existing package.
As you might have seen in the Debian Med policy I've created an adapted
package_template but this is not as helpful in your case.

> or i can copy ubuntu-Manila debian directory on my
> repository, and do the first commit?

I'd (strongly) recommend to start from this since this is most probably
the best way to start from.  I'd always consider starting from some
existing packaging work and see how far it leads.  I had a view into
this and have seen several needed changes - but we can do this step by
step.

> if iI copy ubuntu-Manila debian dir,
> is it needed to change some thing like control file, changelog, ...? Or I
> should just do the first commit, and then we will try to adapt it for
> Debain Repositories?

You should have a close look into some other package done by the
openstack team and adapt the "Maintainer" and "Uploaders" field
accordingly.  Please confirm that you were able to follow up to this
point (or not and ask what might remain unclear).  Than we might
continue from here.
 
> >
> > At office time I'm excluded from IRC (and yes, I know how to deal with
> > this - but I will not this).  I think it will work if Malihe will send
> > me some pull request.
> >
> I dont know what do you mean by "sending some pull request to you". would
> you explain what should I do, please?

Well, in the teams I'm usually working I notice any commit since a mail
is sent to an according list.  Since there is no such list in the
openstack team I will miss your commits (since I do not lurk on IRC).
So please send me some mail that you did a commit (perhaps with a short
commit log) and I'll pull from Git and will check your changes.
 
> > Only the packaging branch (ie: debian/kilo) is needed, if you use
> > > tag-based packaging.
> >
> > Can you please give some hint where Malihe can read about tag based
> > packaging?  I'm used to the workflow I posted in my other mail but I
> > think it is most helpful if Malihe adapts to what is used in openstack
> > team.
> >
> I tried to find some thing about tag-based packaging or guess how may it
> works, but I could not find any thing unfortunately.

Unfortunately I also have no idea about this.  Thomas, could you clarify
please what exactly do you mean.  I personally have no idea what to do
once the tarball is downloaded via

    uscan

(see above).  While I'd like to adapt to the openstack workflow I think
if we do not get any enlightenment here we stick to the Git workflow I'm
used to.  I'm afraid we might simply loose our goal to deliver a package
out of sight if we are fighting with different workflows.  As I
understood the Ubuntu folks correctly they are only interested in a
Debian source package.  We will get this with any workflow.  So I have
the following suggestion:  If we do not find out how to do tag based
packaging I will create a Git repository using pristine-tar as many
other teams are successfully using and inject the debian/ dir up to the
point I described above.  I will do this tomorrow morning in case nobody
insists.  Otherwise we will dramatically fail with our attempt to get
the package done in April (which is no problem in principle).  I think
if you learned packaging using the workflow I know adapting to some
other workflow in the end is not hard.

> > BTW, what do you think about my last suggestion to turn the d/watch
> > files to github and take the latest version from there (which is either
> > later than the vivid version if you include (\.0b\d) in the regexp
> > or older if you decide for stable releases)?
> >
> I dont know what are " d/watch files". May I know where can I find some
> documents about it?

Sorry for the stupid shortcut.  Please note:  If I write "d/<something>"
I mean "debian/<something>".  Its a stupid pedagocial mistake of mine to
throw those shortcuts onto newcomers and I'll avoid this in future.  I
think the debian/watch issue was explained above.  Its simply the meta
information where to seek for new upstream versions.


[tag-based-packaging]
> > > git clone git://github.com/openstack/manila.git
> > > cd manila
> > > git tag 2015.1_b3 2015.1.0b3
> > > git reset --hard 2015.1_b3
> > > git checkout -b debian/kilo
> > > git branch -d master
> > > # <add here your debian folder for the packaging>
> > > git add debian
> > > git commit -a -m "Initial debian folder"
> > >
> > > Make sure that, when you add your Debian folder, you get something
> > > like this in debian/gbp.conf:
> > >
> > > [DEFAULT]
> > > debian-branch = debian/kilo
> > > upstream-tag = %(version)s
> > > compression = xz
> > >
> > > [git-buildpackage]
> > > export-dir = ../build-area/
> > >
> > > Then doing "pkgos-bop" should rebuild the package correctly.
> >
> Thank you Thomas for the explanation. I need to study about
> debian/gbp.conf

Hmm, reading this again this is possibly what Thomas means with "tag
based packaging".  Did you went all these single steps?  While I
think the tag is not the one we should target at since uscan reports
some newer version this sounds quite sensible to me.  I'm afraid
you just where not able to push this to git.debian.org.

So please if there are remaining problems:  Do *every* single step and
post the output (may be shortened) to enable me reproducing what you did
and comment ond potential problems.  If you confirm that all above went
smoothly you might just hang on the pkgos-alioth-new-git script which as
I wrote above could be separated into simple steps most probably.  Just
let me know about any single step and any single failure that might
occure.  I'll try to reproduce this here and will take over things that
might fail on your side (for whatever reason).
 
> > OK, Malihe, please follow this advise.  I admit I also need to adapt to
> > this workflow but I'll try.
> 
> I followed it. but I dont know why should I run "git tag 2015.1_b3
> 2015.1.0b3" command.
> Are we re-tagging the last tag on the main project? why? should we do
> re-tagging every time that we pull from the main project on github?
> Also, what is the main policy behind tag names? The last tag of Manila on
> github, is "2015.1.0rc1". So I should re-tag that commit with "2015.1.rc1"
> ? where can I found some explanation about how Openstack team (or
> debian-openstack team) tag their projects?

I guess the retagging is done since in Debian we are using the tilde (~)
to make sure we can easily increase the version number when there is an
official release (without the attached "b3").  You might like to have a
look at the following comparison:

  if dpkg --compare-versions 2015.1~b3 lt 2015.1 ; then echo "2015.1~b3 is lower than 2015.1" ; fi
 
That's the reason why in debian/changelog you find the '~' before the
'b' in the version string and this in turn is the reason why in
debian/watch the '~' is injected in the uversionmangle expression (see
above).  However, (as far as I know) the tilde is no valid character for
Git tags and thus in Git tags it is emulated with an underscore ('_').

On the other hand you are correct that Thomas was not refering to the
latest upstream tag (as we know from the uscan --report above) and thus
I think it is rather

   git tag 2015.1.0_rc1 2015.1.0rc1

what we want to write here.  Please note that I have currently no clone
on my machine - may be the upstream tag is different - adapt if needed.
 
> > > We're not using pristine-tar, because of efficiency: maintaining
> > > tarballs is a very slow process, as they may be very big, and their
> > > number increases with each (beta and RC) release, which isn't nice.
> > > Also, there's now way to tell in which kind of version of tar the
> > > tarball will be generated, and therefore, we may hit the issues
> > > which Joey Hess explained about the non-reliability of pristine-tar
> > > (which made him orphan the package). Therefore, I've been using git
> > > tag based packaging, as per what I just explained above.
> >
> > Thanks for the explanation.
> >
> Thank you Thomas :)
> Andreas, I still like to know about prestine-tar. Maybe, it will be needed
> for some other projects in Debian. I hope we have enough time for learning
> it, too :)

Please refer to Debian Med policy.  Its explained there if you seek for
"pristine-tar".  I need to admit that I do not see the constraints given
above.

> > Malihe, it seems we are now both learning. :-)  Please follow this
> > advise and post any problem you might have.  I'll try to help and Thomas
> > will fix me if I'm wrong. :-)
> >
> hmm... I don't know what exactly do "uscan --verbose --report " and i
> don't know anything about watch files as I mentioned before. So, this part
> is not understandable for me :P

See above.  As I said in the beginning you need to ask in more small
steps and more frequently.  I think this is more efficient than those
large mails.  Please break up your single steps in the future and do
not hesitate to ask any single bit - there is no chance to ask stupid
questions.  Only my answers might be stupid - but I hope they will be
not. ;-)

> Andreas, sometimes I need to ask some questions online (maybe on IRC). Are
> you online on #debian-med? where can I find you?

We need to agree to a meeting.  At daytime I'm not in IRC.  I could join
#debian-openstack this evening at 19:00 GMT+2.

Hope this helps

       Andreas.

-- 
http://fam-tille.de