[Apt-cacher-ng-users] Avoid client timeout while the proxy (alioth-apt-cacher-ng: message 6 of 20) retrieves a large file

[Eduard Bloch]

Hallo,
* Wade [Fri, Jan 13 2012, 09:43:54AM]:

> > Which version? There was a bug with older versions where the message
> > queue could get stuck when it was flooded with requests. But one needed
> > a really messy sources.list to get there.
> 
> I'm running it on Arch Linux, version 0.6.11.

Interessting.

> moment.  I see the same behaviour with each and every GET request made
> by the package manager, and it is reproducible even with only 1 client
> and 1 transaction at a time.
> 
> I'm using the direct request method:
> http://proxy-host:3142/example.com/path/to/file.tar.xz
> 
> What I'm seeing is the following in the case of a cache miss:
> 1. Client -> Proxy: HTTP GET proxy-host:3142/example.com/path/to/file.tar.xz
> 2. Proxy -> Server: HTTP GET example.com/path/to/file.tar.xz
> 3. Proxy <- Server: transfer entire file
> 4. Client <- Proxy: after 3 is finished, transfer entire file
> 
> I've attached a capture to demonstrate the behaviour.  I used tc qdisc
> to limit the bandwidth on the server for testing purposes, but you can
> see this synchronous transfer behaviour all the same.  The client

Unfortunatelly, that is not what I see in your pcap file. There are
also steps:

2.1: Proxy receives "HTTP/1.1 206 OK" from remote server
2.2: Proxy starts storing data in its cache
2.3: Proxy sends a "HTTP/1.1 2..." to the original client based on
information seen in step 2.1

AFAICS, it takes for the remote server seven (!) seconds to respond, and
then steps 2.2..4 happen almost immediately because the test file is so
small (fits entirely into a single recv() buffer).

> 'feel' behaviour in the real case is that there is no data transferred
> for a long time, and then a sudden burst of traffic at the end
> (greater than the WAN bandwidth) as the request is then fully received
> by the proxy and transferred to the client at once.

To reproduce the behaviour you mentioned, the file should ideally be
larger than 64kb.

BTW, I know from apt's behavior that it can be sometimes confusing.
For example, when access is done through prefixes in particular URLs
(instead of http proxy mode) then apt serializes download requests
instead of starting one queue per server (because the URLs seem to
belong to the same remote machine) and the console output looks unusual.
The performance can slightly suffer because the pipeline depth is
limited per server, though.

> PS
> This is a fantastic project, though, don't get me wrong there.  I love
> the cache expiration, which is almost perfect with arch linux.  The
> only problem I have on that front is that it downloads the package
> indices, but they are renamed from index.db.tar.gz to index.db on
> disk, and so not read properly.  I got around this by creating a
> symlink in each cache dir from index.db to index.db.tar.gz and then
> everything is fantastic.

You mean, it does not pick up the right index files? I think this should
be fixed ASAP. But I am confused... where exactly shall the files
be renamed? AFAICS the file names in the archive are index.db.tar.gz,
and the current code looks for files named like .db.tar (plus optional
compression suffix .gz/.bz2/.xz/.lzma).

Regards,
Eduard.

-- 
<xTs> rpm? Waren das nicht verwunschene debs?