[Apt-cacher-ng-users] Avoid client timeout while the proxy (alioth-apt-cacher-ng: message 6 of 20) retrieves a large file

[Wade]

>> I just started using apt-cacher-ng today, and I noticed that sometimes
>> the client will timeout.  Does the proxy retrieve the entire file from
>
> Which version? There was a bug with older versions where the message
> queue could get stuck when it was flooded with requests. But one needed
> a really messy sources.list to get there.

I'm running it on Arch Linux, version 0.6.11.

> A possible workaround for that versions is to set MaxSpareThreadSets to
> something higher like 50 (but watch the memory consumption after a
> while and maybe set it to a lower, still working value).
>
>> the remote server before transferring any of the response back to the
>> client?  Would it be feasible to stream the response back to the
>> client as it is retrieved from the remote server (and cached)?
>
> No, streaming while receiving is an essential feature. A while ago I got another
> report from somebody with similar symptoms but the person was not very
> cooperative on debugging, and I could not reproduce it.

I think we're talking about the same thing here with your last
paragraph, so I'll leave the MaxSpareThreadSets issue aside for the
moment.  I see the same behaviour with each and every GET request made
by the package manager, and it is reproducible even with only 1 client
and 1 transaction at a time.

I'm using the direct request method:
http://proxy-host:3142/example.com/path/to/file.tar.xz

What I'm seeing is the following in the case of a cache miss:
1. Client -> Proxy: HTTP GET proxy-host:3142/example.com/path/to/file.tar.xz
2. Proxy -> Server: HTTP GET example.com/path/to/file.tar.xz
3. Proxy <- Server: transfer entire file
4. Client <- Proxy: after 3 is finished, transfer entire file

I've attached a capture to demonstrate the behaviour.  I used tc qdisc
to limit the bandwidth on the server for testing purposes, but you can
see this synchronous transfer behaviour all the same.  The client
'feel' behaviour in the real case is that there is no data transferred
for a long time, and then a sudden burst of traffic at the end
(greater than the WAN bandwidth) as the request is then fully received
by the proxy and transferred to the client at once.

Thanks for the help.
Wade

PS
This is a fantastic project, though, don't get me wrong there.  I love
the cache expiration, which is almost perfect with arch linux.  The
only problem I have on that front is that it downloads the package
indices, but they are renamed from index.db.tar.gz to index.db on
disk, and so not read properly.  I got around this by creating a
symlink in each cache dir from index.db to index.db.tar.gz and then
everything is fantastic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: apt-cacher-ng.zip
Type: application/zip
Size: 12112 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/apt-cacher-ng-users/attachments/20120113/c77d3087/attachment.zip>