[Apt-cacher-ng-users] slow "apt-get update" via apt-cacher-ng

[Eduard Bloch]

Privet Kirill!

Thank you very much, I think a found the problem. It wasn't easy to spot
and it would be even harder without your verbose logs. This actually
looks like a Heisenbug since I have seen this symptoms from time to time
and never could reproduce it reliably.

Please find the patch attached and report whether it solves the speed
problem.

Problematic event chain goes like this:
 - the connection (or remote servers) have latency, therefore unnessary
   reconnects are a performance killer
 - on 304 responses, the file item management might release the internal
   descriptor too early (that's what the patch is for)
 - this error situation is well handled (actually it's interpreted like
   an client which suddenly disconnected)
 - that makes the download code handle this "error" condition and abort
   the running connection (because it doesn't know what to do with the
   incoming data now)
 - that case again is handled smoothly by reconnecting to the server.
   But as user, you still see the lags if you watch the times closely.

Thanks again!

Eduard.

* Kirill Timofeev [Mon, Aug 04 2014, 04:23:44PM]:
> Hi Eduard,
> 
> I created debug build and did 2 runs of apt-get update with and without
> proxy:
> 
> kirill.timofeev at els-trusty-test-02:~⟫ time sudo apt-get update -o
> Debug::Acquire::Http=true >apt-get-update-noproxy.log 2>&1
> 
> real    0m13.438s
> user    0m5.328s
> sys     0m0.737s
> kirill.timofeev at els-trusty-test-02:~⟫ time sudo apt-get update -o
> Debug::Acquire::Http=true >apt-get-update-withproxy.log 2>&1
> 
> real    0m29.039s
> user    0m4.131s
> sys     0m0.607s
> kirill.timofeev at els-trusty-test-02:~⟫
> 
> Please find logs attached.
> 
> Thanks,
> Kirill.
> 
> On 08/04/2014 10:53 AM, Eduard Bloch wrote:
> >Hallo Kirill,
> >
> >sorry, your last mail was not sent out to the mailing list because of
> >the size limit on the mailing list and I think we should keep it that way.
> >
> >And sorry, I still cannot see any obvious pattern in the log. Maybe you
> >could produce the same using no proxy? Alternatively, you could create a
> >debug build (see manual) and make a more detailed log on the proxy.
> >
> >Regards,
> >Eduard.
> >
> >* Kirill Timofeev [Tue, Jul 29 2014, 02:15:31PM]:
> >>Hi Eduard,
> >>
> >>500 errors were because of misconfigured internal repository
> >>pkg.prod.hulu.com (https://github.com/hulu/urepo). I fixed issue and run
> >>command again:
> >>
> >>root at els-trusty-test-02:~# time apt-get update -o Debug::Acquire::Http=true
> >>>apt-cacher-ng-slow-update.log 2>&1
> >>real    0m40.026s
> >>user    0m4.041s
> >>sys     0m0.603s
> >>root at els-trusty-test-02:~#
> >>
> >>New log is attached, update time unfortunately is still quite long.
> >>
> >>Thanks,
> >>Kirill.
> >>
> >>On 07/29/2014 01:54 PM, Eduard Bloch wrote:
> >>>Hallo,
> >>>* Kirill Timofeev [Mon, Jul 28 2014, 11:43:13PM]:
> >>>>Hi Eduard,
> >>>>
> >>>>thank you very much for quick response. Please find log attached. Here is
> >>>>command I used to obtain this log:
> >>>>
> >>>>root at els-trusty-test-02:~# time apt-get update -o Debug::Acquire::Http=true
> >>>>>apt-cacher-ng-slow-update.log 2>&1
> >>>>real    0m39.796s
> >>>>user    0m4.212s
> >>>>sys     0m0.626s
> >>>>Ign http://us.archive.ubuntu.com trusty InRelease
> >>>>HTTP/1.1 500 Internal Server Error
> >>>>Content-Length: 0
> >>>>Date: Tue Jul 29 06:40:20 2014
> >>>>Server: Debian Apt-Cacher NG/0.7.26
> >>>>X-Original-Source: http://pkg.prod.hulu.com/deb/dists/trusty/Release.gpg
> >>>>Connection: Keep-Alive
> >>>There is a lot of such error messages in your log. However,
> >>>apt-cacher-ng does not generate them! There is only one similar string
> >>>in the source which uses different letter case ("500 Internal server error").
> >>>
> >>>This leads me to the assumption that you must have something inbetween
> >>>(another proxy that rewrites the responses incorrectly?) which goes nuts.
> >>>Maybe a transparent proxy which you don't know anything about?
> >>>
> >>>If that assumption is correct, you might play around with PipelineDepth
> >>>option until you find a value which this proxy is willing/capable to
> >>>handle.
> >>>
> >>>Regards,
> >>>Eduard.
> >>GET http://us.archive.ubuntu.com/ubuntu/dists/trusty/InRelease HTTP/1.1
> >>Host: us.archive.ubuntu.com
> >>Cache-Control: max-age=0
> >>Accept: text/*
> >>User-Agent: Debian APT-HTTP/1.3 (1.0.1ubuntu2)
> >
> >(attachment stripped)
> >
> 



-- 
Computer können nicht denken.
Nur Menschen denken, daß Computer denken.
		-- Michael Uhlemann
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Really-make-sure-that-a-fileitem-owner-object-doesn-.patch
Type: text/x-diff
Size: 1235 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/apt-cacher-ng-users/attachments/20140805/5b55615b/attachment.patch>