r291 - branches/rewrite/doc
Sergio Talens-Oliag
partial-mirror-devel@lists.alioth.debian.org
Wed, 10 Nov 2004 17:26:40 -0700
Author: sto
Date: Wed Nov 10 17:26:39 2004
New Revision: 291
Modified:
branches/rewrite/doc/IDEAS
Log:
IDEAS reorganized to be a reStructuredText document. Included notes about backend snapshots and shared pools.
Modified: branches/rewrite/doc/IDEAS
==============================================================================
--- branches/rewrite/doc/IDEAS (original)
+++ branches/rewrite/doc/IDEAS Wed Nov 10 17:26:39 2004
@@ -1,90 +1,157 @@
-This page is to coordinate
-[http://packages.debian.org/unstable/net/debpartial-mirror
-departial-mirror] development.
+=========================
+ DEBPARTIAL-MIRROR IDEAS
+=========================
-debpartial-mirror is a tool to create a local partial mirror of
+This page is to coordinate `debpartial-mirror`_ development.
+
+.. _`debpartial-mirror`: http://packages.debian.org/unstable/net/debpartial-mirror
+
+Introduction
+============
+
+``debpartial-mirror`` is a tool to create a local partial mirror of
apt-repository.
-The "partial-mirror" can be used both as a source of packages to build
-Custom Debian Distributions cd-images, or as a local APT
-repository. Packages into the partial mirror can be downloaded from
+The *partial-mirror* can be used both as a source of packages to build
+**Custom Debian Distributions** archives, cd-images or local APT
+repositories.
+
+The packages included into the partial mirror can be downloaded from
remote APT repository or local directories.
The list of packages that are included into the partial-mirror is
specified by means of a configuration file: it is also possible to
filter from a package list.
-The partial mirror is organized as follows: it is a collection of
-Backends. A Backend is a repository with packages from one source (a
-local or a remote repository):
-
-/partial_mirror/
- sarge/
- sid/
- debian-np/
- local-np/
- br/
- it/
- ....
+Mirror organization
+-------------------
+
+The partial mirror is organized as a collection of Backends. A Backend is a
+repository with packages from one source (a local or a remote repository):
+
+ /partial_mirror/
+ sarge/
+ sid/
+ debian-np/
+ local-np/
+ br/
+ it/
+ ....
In turn each backend, has a fixed structure:
-/partial_mirror/backend/
- dists
- pool
+ /partial_mirror/backend/
+ dists/
+ pool/
+
+the ``dists/`` directory contains the ``Packages.gz``, ``Sources.gz`` and
+``Release`` files for the Backend, while the ``pool/`` directory contais an
+apt pool-style tree of packages.
+
+Merged backends
+---------------
+
+Once each Backend is filled, it is possible to create several *Merged
+Backends*.
+
+A *Merged Backend* is a Backend built using a list of Backends from the local
+mirror that contains a ``dists/`` dir that merges the contents of the
+different ``dists/`` subdirectories of the selected Backends and a ``pool/``
+subdirectory built using symbolic (or hard) links to the packages included on
+the merged Backends.
+
+In addition, when the Merged Backend is created, it is possible to filter
+which packages to include from the selected Backends.
+
+Backend snapshots
+-----------------
-the "dists" directories contains Packages.gz, Sources.gz and Release
-files for the Backend, while the "pool" dir contains an apt pool-style
-tree of packages.
+Each backend has a list of binary and source packages inside the ``dists/``
+subdirectory and a copy of the corresponding files into the ``pool/`` dir.
-Once each Backend is filled, it is possible to create several Merged
-Backends: A Merged Backend is a Backend realized using a list of Backends
-in local mirror. The Merged Backend, contains a "dists" dir that merge
-the contents of dists/* of each selected Backend, and a "pool" dir with
-symbolic links to the packages contained in the pool dir of selected
-Backend.
+The standard way of updating each backend would be to download the new
+Packages and Sources lists, remove the old files from the pool and download
+the new ones.
-In addition when the Merged Backend is created, it is possible to filter
-between the packages in the selected Backend.
+A very interesting option could be to support multiple *snapshots* of the same
+backend, the idea would be to keep diferent versions of the ``dists/``
+subdirectory using a file structure like the following:
-Usage:
-------
+ /partial_mirror/backend/
+ snapshots/YYYYMMDD/dists/
+ snapshots/YYYYMMDD+1/dists/
+ ...
+ dists/
+ pool/
-(Note: This is different from the actual implementartion..)
-(Note: the following could be used as trace for the man-page)
+With this schema each ``dists/`` subdirectory contains the corresponding
+lists of source and binary packages and the ``pool/`` subdir contains the
+source and package files of all included snapshots.
+Of course the system has to allow the selective removal of one snapshot (i.e.,
+remove the 20041109 snapshot), that is, it has to support the removal of the
+source and package files that belong to the selected snapshot and are not
+included on any of the rest of *snapshots*.
-SYNOPSIS:
+Shared pool
+-----------
- debpartial-mirror [-c=file]
- {update [backend, [....]] | merge [backend, [....]] |
- | check | clean [backend, [....]]}
+Extending the idea of the *snapshots* presented before we could use a shared
+``pool/`` to store the source and package files included on each *backend*.
+
+This could reduce the download time and the pool size; if a file is included
+into two *backends* it is only installed once.
+
+The problem with this model is similar to the one described earlier: to remove
+unwanted files from previous *backends* without breaking all the rest we have
+to calculate the differences as described when talking about removing a
+*snapshot*.
+
+
+Program description
+===================
+
+.. Note: This is different from the actual implementartion, the following
+ could be used as trace for the man-page.
+
+SYNOPSIS
+........
+
+::
+
+ debpartial-mirror [-c=file]
+ {update [backend, [....]] | merge [backend, [....]] |
+ | check | clean [backend, [....]]}
DESCRIPTION
+...........
-debpartial-mirror will accept some command in the way apt-get does, and
+``debpartial-mirror`` will accept some command in the way apt-get does, and
read conf information from a configuration file (default
-/etc/debpartial-mirror/debpartial-mirror.conf) that in turn can include
+``/etc/debpartial-mirror/debpartial-mirror.conf``) that in turn can include
other conf files.
- update
+update
- this command let debpartial-mirror to update the backends that
- compose the partial mirror. If no backend is specified, all
- backends will be update. Else, only the listed backends will be
- updated.
-
- merge
-
- this command will create one (or more) merge backend. The list
- of backends to be used to merge is specified in the main
- conf-file.
+ this command let debpartial-mirror to update the backends that
+ compose the partial mirror. If no backend is specified, all
+ backends will be update. Else, only the listed backends will be
+ updated.
+
+merge
+
+ this command will create one (or more) merge backend. The list
+ of backends to be used to merge is specified in the main
+ conf-file.
After update and/or merge, debpartial-mirror will clean up the mess
removing old and uneeded files from it's structure.
-
-A description of Backend class:
--------------------------------
+
+Notes about the source code
+===========================
+
+The backend class
+-----------------
Properties:
need_update (boolean)
@@ -101,11 +168,12 @@
__create_package_list -> Generate a package list reading from
main conf-file.
-A description of Dists class:
------------------------------
+The dists class
+---------------
Properties:
got_files
+
Methods:
update -> Update the Dists files (Packages, Release and Sources)
_get -> Get each file