Alternative format for the configuration file
Free Ekanayaka
free@agnula.org
Wed, 28 Jul 2004 17:00:10 +0200
|--==> "NLB" == Nathaniel L Budin <natb@brandeis.edu> writes:
NLB> On Wed, 28 Jul 2004, Free Ekanayaka wrote:
>>IMHO this further filtering is a confusing. I would move all the
>>filtering statements in the merge section, and use the remote backend
>>as unfiltered APT sources, that define the set of all the available
>>packages.
NLB> This would remove important features as well as rendering one of the
NLB> biggest use cases of this tool unusable: partial mirroring (hence the
NLB> name).
I think I did not explain the details, hence the misunderstanding.
When I talk about APT sources, defining the set of all available
package, I do not mean that you have to download the whole associated
pools! just the relevant Packages.gz and Sources.gz
I'll try to elaborate my idea again.
The aim would be not only to partially mirror an APT repository, but
even to build custom repositories by merging chunks of existing ones
(e.g. debian-installer form sid, base system from sarge, and xfree86
packages form experimental). I think we all agree on this (BTW the
appropriate name for such a tool perhaps would be debcustom-mirror).
Practically speaking what I'd do would simple be:
1) download the Packages.gz and Sources.gz file from whatever APT
repository I want to use
2) generate my custom Packages.gz and Source.gz according to the
rules (filters/include/exclude/resolv_dep_using) defined in
the config file
3) populate the pool/ fetching the relevant packages, and possibly
deleting the old ones
Of course everything has to be done architecture-wise and
component-wise, that means if a package was listed in
some_apt_url/dists/sarge/non-free/binary-i386/Packages
then it must go in
my_rep_root/dists/my_custom_distro/non-free/binary-i386/Packages
But this goes without saying..
Said this, my class-layout/implementation proposal is (don't care
about class names, we might change them to better ones):
AptSource - basically implements 1), caching the files in, say,
/var/cache/debpartial-mirror/<server>/dists/..
CustomDist - basically implements 2) and 3)
Both classes would use PackageList. AptSource would build a
PackageList object out of each downloaded Packages.gz and Source.gz,
while CustomDist would build its PackageList objects in this way:
0) We have empty PackageList objects (on for each
architecture/component couple)
1) Start to fill the empty PackageList objects by scanning entry by
entry the PackageList objects of AptSource defined for this
custom repository.
The entries are selected by applying filter/include/exclude
rules to the defined for this custom repository. [0]
2) Scan the PackageList objects generated in 1) adding the
dependences, respecting the priorities defined in the
configuration file [1]
Finally we populate the pool/ according to the the PackageList objects
generated by the CustomDist object(s).
I think such approach grants you a great degree of freedom when
filtering/merging repositories together.
>>Let me add that in this case the configuration file is not a simple
>>flat list of variables, that defines paths, options, switches etc, but
>>it's rather similar to a tiny programming language, used to build up
>>your CDD using wide APT pools as raw bricks.
>>
>>[rest of explanation snipped]
NLB> OK, Free, let me square with you. If you want me to sit down and throw
NLB> away the last 3 weeks or so of work on the new configuration format,
NLB> you're going to have to give me a better reason than "this other format is
NLB> prettier". The current (newly rewritten) config format isn't the best
NLB> possible one, I agree, but it's very flexible, very easy to read, and does
NLB> everything we need out of it.
NLB> If you were to show me a grave problem with the current format, then I'd
NLB> consider throwing it away and doing something different. As it is, I
NLB> don't see a good reason to rewrite this code. If YOU want to sit down and
NLB> rewrite it, rather than just telling me to do it, that's great, and that's
NLB> what Free Software is all about, and we'll talk about it when you've got
NLB> patches ready that pass the test suite. :-D
NLB> As it is, I'd rather get to work on the actual functionality of this
NLB> program, instead of quibbling about details of the (IMO, rather
NLB> superficial) config format it uses.
I perfectly understand your point, and I do agree with it. However if like
the alternate format too, I can find some time to write the changes to the
Config.py module myself.
Cheers,
Free Ekanayaka
[0] Note that the order in which the various AptSource objects are
scanned does matter, so that if package P is present both in the
AptSource objects A and B, then you can give precendence to the one
in A or B. For example this would be useful for including
backports, e.g a configuration like:
AptSources
{
Debian
{
Location "http://ftp.debian.it/debian";
Distributions { stable; }
Components { main; contrib; non-free; };
};
Backports
{
Location "http://www.backports.org/";
Distributions { stable; }
Components { package1; package2; package3; }
};
};
CustomDist
{
MyDistro
{
MyAdminPackages
{
AptSources { Backports::stable; Debian::stable; };
Filters { Section { admin; }; };
}
StableBase
{
AptSources { Debian::stable; };
Filters
{
Section { base; };
Priority { required; important; };
};
};
}
}
would result in a custom distro using the base system from stable
and the admin packages both from stable and backports, prefering
the second over the first.
[1] For example you may want to take some all the packages in section
sound from Sid, but give precendence to the packages from Sarge
to satisfy the dependences, as long as versions are ok:
ResolveDepsUsing { Debian::sarge; Debian::sid; };