Bug#841499: uscan: support searching in multiple directories for matching files

[Paul Wise]

On Tue, 2016-10-25 at 01:54 +0900, Osamu Aoki wrote:

> If we do not do this, we need to loop over scanning many pages... Not a
> good idea.  Can you think of non-invasive change?

As I said in the original bug report, scan each directory in descending
order of version until at least one file was found. 

In the normal case this change will not change the behaviour of uscan
at all since a file will be matched on the first directory.

Only in watch files where uscan fails to find a file in the first
directory will my proposal change the behaviour.

For the most common case (RCs in the first directory and releases in
the second), uscan will only download one extra page.

For the cases where the file part of the regex does not match any file
in any subdirectory, we can limit it to 5 requests by default, with a
0.5 second delay between them to reduce impact.

> How about scanning https://cmake.org/download/

That is only a workaround for this uscan flaw.

> Most HTTP site has this kind of page.

I've encountered a number of cases over the years on mentors IRC and
other places where this wasn't possible.

The cmake one and most others only show the latest release, which means
that I can't use uscan to download a particular version.

> I think complicating page scanning mechanism

It isn't much of a complication at all really:

On error, if we scanned a directory, go back and scan the next
directory. Possibly with a configurable limit of scanned dirs.

> FTP

FTP has nothing to do with this issue, why do you mention it?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/devscripts-devel/attachments/20161025/9b8cb870/attachment.sig>