Bug#696492: FTP directory walking has room to improve
Osamu Aoki
osamu at debian.org
Sat Oct 3 14:30:57 UTC 2015
Hi,
I realized problem reported can be avoided if a watch line is
PROBLEM:
ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/(.+)/ubuntu-(.+)-desktop-amd64.iso.torrent
NO PROBLEM:
ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/(\S+)/ubuntu-(\S+)-desktop-amd64.iso.torrent
Yes, just reject spaces :-)
I think this could be avoided if we pre-filter string fead to match (.+).
Here is a interesting log for working one. (under development package
but situation is the same and quite telling what strings it catches
under FTP (without http proxy).
Not all FTP servers list like this but ^- should be skipped and remove all
but last part before the actual string match so even sloppy (.+) does
not cause such a major problem.
$ uscan --debug
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/(\S+)/ubuntu-(\S+)-desktop-amd64.iso.torrent debian
uscan debug: URL ... part of watch line: ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/(\S+)/ubuntu-(\S+)-desktop-amd64.iso.torrent debian
uscan debug: $gpgv_used=0, $gpg_used=0, $download=1, $force_download=0
uscan debug: $options{'pgpmode'}=mangle, $options{'pgpsigurlmangle'}=undef
uscan debug: last orig.tar.* tarball version: 0.0
uscan debug: last orig.tar.* tarball version: 0.0
uscan debug: Last orig.tar.* tarball version (dversionmangled): 0.0
uscan debug: $download_version undefined after dversionmangle
uscan debug: dir=>/sites/releases.ubuntu.com/ dirpattern=>(\S+)
uscan debug: requesting URL ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/
uscan debug: received content:
lrwxrwxrwx 1 ftp ftp 5 Mar 19 2010 10.04 -> lucid
lrwxrwxrwx 1 ftp ftp 5 Feb 16 2012 10.04.4 -> lucid
lrwxrwxrwx 1 ftp ftp 7 Mar 01 2012 12.04 -> precise
lrwxrwxrwx 1 ftp ftp 7 Aug 07 2014 12.04.5 -> precise
lrwxrwxrwx 1 ftp ftp 6 Mar 27 2014 14.04 -> trusty
lrwxrwxrwx 1 ftp ftp 6 Aug 06 19:44 14.04.3 -> trusty
lrwxrwxrwx 1 ftp ftp 6 Sep 26 2014 14.10 -> utopic
lrwxrwxrwx 1 ftp ftp 5 Mar 26 2015 15.04 -> vivid
lrwxrwxrwx 1 ftp ftp 4 Sep 24 23:04 15.10 -> wily
-rw-rw-r-- 1 ftp ftp 22 Feb 01 2006 FOOTER.html
-rw-rw-r-- 1 ftp ftp 2252 Aug 06 20:33 HEADER.html
drwxrwxr-x 2 ftp ftp 4096 Sep 21 2012 cdicons
-rw-rw-r-- 1 ftp ftp 1150 Jun 16 2011 favicon.ico
drwxrwxr-x 3 ftp ftp 4096 Aug 21 2014 include
drwxrwxr-x 2 ftp ftp 4096 Sep 26 2013 lucid
drwxrwxr-x 2 ftp ftp 12288 Jun 03 22:11 precise
lrwxrwxrwx 1 ftp ftp 1 Jul 31 2007 releases -> .
-rw-rw-r-- 1 ftp ftp 49 Oct 29 2009 robots.txt
drwxr-xr-x 2 ftp ftp 4096 Aug 06 20:45 trusty
drwxrwxr-x 2 ftp ftp 4096 Oct 23 2014 utopic
drwxrwxr-x 2 ftp ftp 4096 Sep 17 15:11 vivid
drwxrwxr-x 2 ftp ftp 4096 Sep 24 23:07 wily
[End of received content]
uscan debug: matching pattern (\S+)
-- Found the following matching FTP dirs (newest first):
wily (wily)
vivid (vivid)
utopic (utopic)
trusty (trusty)
robots.txt (robots.txt)
releases (releases)
precise (precise)
lucid (lucid)
include (include)
favicon.ico (favicon.ico)
cdicons (cdicons)
HEADER.html (HEADER.html)
FOOTER.html (FOOTER.html)
15.10 (15.10)
15.04 (15.04)
14.10 (14.10)
14.04.3 (14.04.3)
14.04 (14.04)
12.04.5 (12.04.5)
12.04 (12.04)
10.04.4 (10.04.4)
10.04 (10.04)
uscan debug: newest_dir => 'wily'
uscan debug: requesting URL ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/wily/
uscan debug: received content:
-rw-rw-r-- 1 ftp ftp 27 Sep 24 23:06 FOOTER.html
-rw-rw-r-- 1 ftp ftp 2959 Sep 24 23:06 HEADER.html
-rw-rw-r-- 1 ftp ftp 280 Sep 24 23:07 MD5SUMS
-rw-rw-r-- 1 ftp ftp 300 Sep 24 23:07 MD5SUMS-metalink
-rw-rw-r-- 1 ftp ftp 198 Sep 24 23:07 MD5SUMS-metalink.gpg
-rw-rw-r-- 1 ftp ftp 198 Sep 24 23:07 MD5SUMS.gpg
-rw-rw-r-- 1 ftp ftp 312 Sep 24 23:07 SHA1SUMS
-rw-rw-r-- 1 ftp ftp 198 Sep 24 23:07 SHA1SUMS.gpg
-rw-rw-r-- 1 ftp ftp 408 Sep 24 23:07 SHA256SUMS
-rw-rw-r-- 1 ftp ftp 198 Sep 24 23:07 SHA256SUMS.gpg
lrwxrwxrwx 1 ftp ftp 45 Sep 24 23:04 ubuntu-15.10-beta2-desktop-amd64.iso -> ../.pool/ubuntu-15.10-beta2-desktop-amd64.iso
-rw-rw-r-- 1 ftp ftp 48764 Sep 24 23:04 ubuntu-15.10-beta2-desktop-amd64.iso.torrent
lrwxrwxrwx 1 ftp ftp 51 Sep 24 23:04 ubuntu-15.10-beta2-desktop-amd64.iso.zsync -> ../.pool/ubuntu-15.10-beta2-desktop-amd64.iso.zsync
lrwxrwxrwx 1 ftp ftp 46 Sep 24 23:04 ubuntu-15.10-beta2-desktop-amd64.list -> ../.pool/ubuntu-15.10-beta2-desktop-amd64.list
lrwxrwxrwx 1 ftp ftp 50 Sep 24 23:04 ubuntu-15.10-beta2-desktop-amd64.manifest -> ../.pool/ubuntu-15.10-beta2-desktop-amd64.manifest
-rw-rw-r-- 1 ftp ftp 44976 Sep 24 23:07 ubuntu-15.10-beta2-desktop-amd64.metalink
lrwxrwxrwx 1 ftp ftp 44 Sep 24 23:04 ubuntu-15.10-beta2-desktop-i386.iso -> ../.pool/ubuntu-15.10-beta2-desktop-i386.iso
-rw-rw-r-- 1 ftp ftp 50363 Sep 24 23:05 ubuntu-15.10-beta2-desktop-i386.iso.torrent
lrwxrwxrwx 1 ftp ftp 50 Sep 24 23:05 ubuntu-15.10-beta2-desktop-i386.iso.zsync -> ../.pool/ubuntu-15.10-beta2-desktop-i386.iso.zsync
lrwxrwxrwx 1 ftp ftp 45 Sep 24 23:04 ubuntu-15.10-beta2-desktop-i386.list -> ../.pool/ubuntu-15.10-beta2-desktop-i386.list
lrwxrwxrwx 1 ftp ftp 49 Sep 24 23:04 ubuntu-15.10-beta2-desktop-i386.manifest -> ../.pool/ubuntu-15.10-beta2-desktop-i386.manifest
-rw-rw-r-- 1 ftp ftp 44662 Sep 24 23:07 ubuntu-15.10-beta2-desktop-i386.metalink
lrwxrwxrwx 1 ftp ftp 44 Sep 24 23:06 ubuntu-15.10-beta2-server-amd64.iso -> ../.pool/ubuntu-15.10-beta2-server-amd64.iso
-rw-rw-r-- 1 ftp ftp 27622 Sep 24 23:06 ubuntu-15.10-beta2-server-amd64.iso.torrent
lrwxrwxrwx 1 ftp ftp 50 Sep 24 23:06 ubuntu-15.10-beta2-server-amd64.iso.zsync -> ../.pool/ubuntu-15.10-beta2-server-amd64.iso.zsync
lrwxrwxrwx 1 ftp ftp 46 Sep 24 23:06 ubuntu-15.10-beta2-server-amd64.jigdo -> ../.pool/ubuntu-15.10-beta2-server-amd64.jigdo
lrwxrwxrwx 1 ftp ftp 45 Sep 24 23:06 ubuntu-15.10-beta2-server-amd64.list -> ../.pool/ubuntu-15.10-beta2-server-amd64.list
-rw-rw-r-- 1 ftp ftp 44661 Sep 24 23:07 ubuntu-15.10-beta2-server-amd64.metalink
lrwxrwxrwx 1 ftp ftp 49 Sep 24 23:06 ubuntu-15.10-beta2-server-amd64.template -> ../.pool/ubuntu-15.10-beta2-server-amd64.template
lrwxrwxrwx 1 ftp ftp 43 Sep 24 23:06 ubuntu-15.10-beta2-server-i386.iso -> ../.pool/ubuntu-15.10-beta2-server-i386.iso
-rw-rw-r-- 1 ftp ftp 27301 Sep 24 23:06 ubuntu-15.10-beta2-server-i386.iso.torrent
lrwxrwxrwx 1 ftp ftp 49 Sep 24 23:06 ubuntu-15.10-beta2-server-i386.iso.zsync -> ../.pool/ubuntu-15.10-beta2-server-i386.iso.zsync
lrwxrwxrwx 1 ftp ftp 45 Sep 24 23:06 ubuntu-15.10-beta2-server-i386.jigdo -> ../.pool/ubuntu-15.10-beta2-server-i386.jigdo
lrwxrwxrwx 1 ftp ftp 44 Sep 24 23:06 ubuntu-15.10-beta2-server-i386.list -> ../.pool/ubuntu-15.10-beta2-server-i386.list
-rw-rw-r-- 1 ftp ftp 44347 Sep 24 23:07 ubuntu-15.10-beta2-server-i386.metalink
lrwxrwxrwx 1 ftp ftp 48 Sep 24 23:06 ubuntu-15.10-beta2-server-i386.template -> ../.pool/ubuntu-15.10-beta2-server-i386.template
[End of received content]
uscan debug: matching pattern (?:(?:ftp://ftp.mirrorservice.org)?\/sites\/releases\.ubuntu\.com\/wily\/)?ubuntu-(\S+)-desktop-amd64.iso.torrent
-- Found the following matching files on the web page (newest first):
ubuntu-15.10-beta2-desktop-amd64.iso.torrent (15.10-beta2)
uscan debug: newest upstream tarball version selected for download (uversionmangled): 15.10-beta2
uscan debug: download filename ubuntu-15.10-beta2-desktop-amd64.iso.torrent
uscan debug: new filename selected for download (filenamemangled): ubuntu-15.10-beta2-desktop-amd64.iso.torrent
uscan debug: downloadurlmangled upstream URL ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/wily/ubuntu-15.10-beta2-desktop-amd64.iso.torrent
Newest version on remote site is 15.10-beta2, local version is 0.0
=> Newer version available from
ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/wily/ubuntu-15.10-beta2-desktop-amd64.iso.torrent
-- Downloading updated package ubuntu-15.10-beta2-desktop-amd64.iso.torrent
uscan debug: requesting URL ftp://ftp.mirrorservice.org/sites/releases.ubuntu.com/wily/ubuntu-15.10-beta2-desktop-amd64.iso.torrent
-- Checking for common possible upstream OpenPGP signatures
uscan debug: new orig.tar.gz tarball version (oversionmangled): 15.10-beta2
More information about the devscripts-devel
mailing list