[Apt-cacher-ng-users] Synchronizing / keeping up to date with multiple repos in offline mode

[Phil (list)]

Hi Eduard,

On Fri, 2016-09-23 at 12:36 +0200, Eduard Bloch wrote:
> Hallo,
> * Phil (list) [Fri, Sep 16 2016, 04:41:15PM]:
> > 
> > I was wondering if anyone could point me in the right direction on
> > how
> > to do the following: I currently have apt-cacher-ng running with
> > multiple repos on multiple architectures (i.e. debian 8/testing for
> > i386/amd64 etc) and everything is working quite well as an online
> > repo.
> >  What I would like to do is run it in offline mode and only
> > periodically update the repositories.  So my question is how to do
> > this
> > in as transparent a way as possible to the client machines?  The
> > most
> > common recommendation I have found for offline repository
> > management is
> > to use apt-offline but can that work (if so, how?) in conjunction
> > with
> 
> IMHO apt-offline is just a fat wrapper script for apt-get commands
> with
> an URL downloader built-in (one could use wget/curl/... and some
> renaming tricks instead).
> 
> See https://help.ubuntu.com/community/AptGet/Offline/PrintUris for
> the way it basically works, and there are more HOWTOs, like
> http://askubuntu.com/questions/974/how-can-i-install-software-or-pack
> ages-without-internet-offline .
> I vaguelly remember having seen first such instructions more than a
> decade ago.

Thanks for the pointers, I'll have a look.

> 
> > 
> > apt-cacher-ng?  Is there any way to update apt-cacher based on the
> > index/package files it already has cached or do I need to update
> > per
> > client/arch (via apt-offline) on the client side and then somehow
> > merge
> > in the updates to the cache server?  Has anyone done a post / wiki
> > on
> > this use case?
> 
> Your description lacks some details. Which host is in offline mode
> and
> when? If apt-cacher-ng machine itself is going online for short time,

Exactly: the acng system (but not the actual clients) will be
periodically online, everything else would be connected to an offline
internal network.

> then the case is clear. Build your download list and just download
> packages through it (probably throwing data away). Maybe with
> apt-offline or like this:
> 
> apt-get --print-uris --force-yes -q dist-upgrade | cut -f2 -d"'" >
> dl.list
> http_proxy=http://acnghost:acngport wget -i dl.list -O /dev/null
> 
> The tricky part here is how to deal with index data. apt-offline (or
> the mentioned howtos) use simple tricks here, injecting previously
> downloaded files into apt cache on the target machine, renaming to
> mimick the apt's internal cache file name convention.
> Which enables apt to run package set calculations for
> update/install/remove/... , and in the next step the same trick is
> used
> to inject package files.
> 
> For apt-cacher-ng the same trick does not work, because it uses
> different cache format and stores extra meta data in separate files.
> Although there is the special "import" function which does similar
> things, it works differently - it uses file contents to build
> fingerprints, and it needs (fresh) index files (at least the higher
> level ones) present in the cache or updated once to match the
> fingerprints. So it won't help in case of "pure offline" mode.
> 
> I think the most simple and most convenient solution for this usecase
> would be if I introduce a new feature to acngtool. Maybe something
> like
> this:
> 
> acngtool dl4cache LISTFILE storagefolder
> 
> Then you call it with a LISTFILE built with the command above (or
> from
> apt-offline) on a networked machine and it produces a folder which
> can
> be transfered to apt-cacher-ng machine and "installed" with a "cp -a"
> command (i.e. simply copy its contents into the cache directory,
> considering the file structure).
> 
> Does this sound good enough?
> 

I think so... but I'm not understanding the issue you're describing re:
the cache format (as long as all of the requests run through acng, why
wouldn't it have everything needed in its cache that it built?  I don't
doubt you, I just don't understand yet) Below I've described what I had
in mind so hopefully you can tell me if/where I've gone off track.

> Networked machine could even be win32, although I am not doing
> regular
> build tests with the Cygwin port... it's about time to check that
> anyhow.
> 
> The problem with this approach would be the need to extract some
> configuration information from the cache machine and bring it over to
> the downloading machine. This probably needs a couple of tricks but
> should be feasible.

What I was thinking is that between acng and the clients, everything
needed seems to exist to do the following:

1) During 'normal' operation, the acng server runs in offline mode on
the internal network and is serving valid, but somewhat dated packages
which the clients should be fine with (i.e. the offline packages and
package indexes should be in sync so the clients don't realize they're
offline from an apt standpoint)

2) While still connected to the internal (offline) network,
periodically pull whatever data is needed from each of the clients
(whether via apt-offline or some other means) and save it off to the
acng system.

3) Connect the acng system to the Internet in online mode and using the
copied data from step 2, use apt-get on the acng system to run through
a series of 'fake' updates (i.e. send the requests through acng to pull
the updated index data and packages so to acng it would look like a
client was running an update when in reality it is all happening on the
acng system to populate the cache and discarding things on the client
side).

4) Reconnect to the internal network running in offline mode and now
the clients could apt-get update to get their updated package index
files which would then allow them to upgrade transparently (i.e. the
clients never know that the acng system actually isn't online since
they would be seeing updated indexes and packages... which is now a
fresher, but still likely somewhat out of date snapshot)

5) Steps 1-4 seem like they'd handle the 'static' (already installed)
packages.  So to add new packages, could I create apt-offline 'batches'
which could behave like yet another client configuration to cache the
needed packages? Then when back in offline mode on the internal
network, just run 'apt-get install X' from the client machines to
install them via acng?

Of course not knowing what I don't know about the mechanics of this, it
merely seems logical/doable... but it may be a flawed plan.
 Essentially, I'm trying to find a middle ground that gets me the best
features of an online acng environment and an offline mirror.

> 
> Regards,
> Eduard.
> 

Thanks,
Phil