Bug#734748: uscan with KDE sites

[Osamu Aoki]

Hi,

On Sat, Sep 26, 2015 at 09:53:06AM +0200, Maximiliano Curia wrote:
> ¡Hola Osamu!
> 
> El 2015-09-26 a las 13:28 +0900, Osamu Aoki escribió:
> >Moreover, some KDE packages have multiple watch lines as
> 
> >version=3
> >http://download.kde.org/unstable/plasma/([\d.]+)/kwin-([\d.]+)\.tar\.xz
> >http://download.kde.org/stable/plasma/([\d.]+)/kwin-([\d.]+)\.tar\.xz
> 
> >Yes, version=3 works for multiple lines but this is awkward watch to use
> >with uupdate etc.  I would rather see:
> 
> >version=3
> >http://download.kde.org/(un)?stable/plasma/([\d.]+)/kwin-([\d.]+)\.tar\.xz
> 
> >Since it happens to be unstable >> stable, this work nicely for normal
> >case but does not work for downloading the specific version.
> 
> I dont think this would work in the current implementation of uscan, as it
> would ignore the stable versions even if the stable versions are greater
> than the ones in unstable.
???
> Right now, for example, the latest applications unstable release is
> 15.07.90, while the latest stable release is 15.08.1, but uscan will see
> that there is a pattern to match in (un)?stable list the options and keep
> only the greater one, afaik

As you see the the --dehs output in column Watch, current released code
picks 2 files.
https://qa.debian.org/developer.php?login=debian-qt-kde@lists.debian.org

Let me test under Jessie, from commandline:
$ uscan --version|head -1
This is uscan, from the Debian devscripts package, version 2.15.3
$ uscan
kwin: Newer version (5.3.95) available on remote site:
  http://download.kde.org/unstable/plasma/5.3.95/kwin-5.3.95.tar.xz
  (local version is 5.3.2)
Successfully downloaded updated package kwin-5.3.95.tar.xz
Successfully symlinked ../kwin-5.3.95.tar.xz to ../kwin_5.3.95.orig.tar.xz.
kwin: Newer version (5.4.1) available on remote site:
  http://download.kde.org/stable/plasma/5.4.1/kwin-5.4.1.tar.xz
  (local version is 5.3.2)
Successfully downloaded updated package kwin-5.4.1.tar.xz
Successfully symlinked ../kwin-5.4.1.tar.xz to ../kwin_5.4.1.orig.tar.xz.

Yes, you get 2 separate downloads.

But with the multitar branch local-buiild package:
$ uscan
kwin: Newer version (5.3.95) available on remote site:
  http://download.kde.org/unstable/plasma/5.3.95/kwin-5.3.95.tar.xz
  (local version is 5.3.2)
Successfully downloaded updated package kwin-5.3.95.tar.xz
Successfully symlinked ../kwin-5.3.95.tar.xz to ../kwin_5.3.95.orig.tar.xz.
uscan: too many main upstream tarballs

Hmmm...  this is a bit of regression since version=3 was specified.  But
the new code has check for duplicated entries.

After changinig watch file to one line using (un)?stable with the
multitar branch local-buiild package:
$ uscan
kwin: Newer version (5.3.95) available on remote site:
  http://download.kde.org/unstable/plasma/5.3.95/kwin-5.3.95.tar.xz
  (local version is 5.3.2)
Successfully downloaded updated package kwin-5.3.95.tar.xz
Successfully symlinked ../kwin-5.3.95.tar.xz to ../kwin_5.3.95.orig.tar.xz.

Versort picks the largest version (even for bad version) and
coincidental alphabetic order "stable->testing->unstable" makes unstable
the choice of directory before looking into other directory content.
(That is the current known and well reasoned choice of this program with
history.  Multitar branch dod not touch it.)

> >Just locking /([\d.]+)/ to a version may work for some case but kind of
> >limited.  The better solution is to look for all path and pick up all
> >kwin-([\d.]+)\.tar\.xz URL and pick the highest package version.

This, I know, causes problem of too many data to be downloaded.
 
> Sadly the /([\d.]+)/ doesn't always reflect the upstream version in our case.
> For example, you can find the kdelibs 4.14.12 in:
> http://download.kde.org/stable/applications/15.08.1/src/

I am a bit lost.  What's wrong with watch line like
 http://download.kde.org/stable/applications/([\d.]+)/src/ \
 amor-([\d.]+)\.tar\.xz

> Also, it's not unthinkable to have a project with hundreds of releases,
> crawling the whole site might be a bit of an overkill.
> Asuming the pattern results are sortable in a meaningful way, (that much is
> true in the pkgkde use case, that is, if 15.08.0 were to contain a version
> of kdelibs it would be <= 4.14.12) they could be bisected to have a more
> efficient crawling. But this wont work with the (un)?stable case.
> 
> This might not be a general solution and I'm not sure if it's an indication
> that another layer is needed to indicate if a field is a version, a sortable
> or an all options need to be checked.
> 
> That said, the current git web interface of the kde package offer a nicer
> way to list the versions of a particular packages:
> https://quickgit.kde.org/?p=kdelibs.git&a=tags

Yes it is easier to find newer version.  But this does not give pointer
to the tarball release.  Picking upstream from VCS is another topic by
itself.
 
> I could probably use that to obtain the versions.
> 
> >It may take more time but more robust.
> 
> >Problem is it seems to require a big rewrite.
> 
> Yeah, I've read the code.
> 
> >I could not see the patch on the web page so I am curious what was
> >proposed.
> 
> I didnt know that the patches were deleted after a while in the bts.

Me either ...

> The patch is a bit of a hack that only addresses the case of /([\d.]+)/
> being equal to the version. If not found it returns keeps the previous
> behaviour.
> This should allow most of the kde packages to download a specified version.
> 
> --- /usr/bin/uscan	2013-12-23 17:31:35.000000000 -0300
> +++ uscan.pl	2014-01-09 11:29:49.685459229 -0300
> @@ -1809,6 +1809,15 @@
> 		foreach my $dir (@dirs) { print STDERR "     $$dir[1]\n"; }
> 	    }
> 	    @dirs = Devscripts::Versort::upstream_versort(@dirs);
> +            if ($download_version) {
> +                print STDERR " $download_version\n";
> +                foreach my $dir (@dirs) {
> +                    if ($$dir[0] eq $download_version) {
> +                        print STDERR "Found it: $$dir[2]\n";
> +                        return $$dir[1];
> +                    }
> +                }
> +            }
> 	    my ($newversion, $newdir) = @{$dirs[0]};
> 	    return $newdir;
> 	} else {

This is what I guessed from the to\ne of your message ... Thanks.

I guess to make it possible to handle situation you raised, version
replacement should happen only for (...) part.

Anyway, --download-version or newly planned --download-debversion or
--download-current-version, FTP or Bare HTTP server path selection is
nasty problem.

Osamu