r331 - / python-zc.catalog python-zc.catalog/branches
python-zc.catalog/branches/upstream
python-zc.catalog/branches/upstream/current
python-zc.catalog/branches/upstream/current/src
python-zc.catalog/branches/upstream/current/src/zc
python-zc.catalog/branches/upstream/current/src/zc/catalog
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info
Brian Sutherland
jinty-guest at costa.debian.org
Wed Oct 25 09:18:26 UTC 2006
Author: jinty-guest
Date: 2006-10-25 09:18:24 +0000 (Wed, 25 Oct 2006)
New Revision: 331
Added:
python-zc.catalog/
python-zc.catalog/branches/
python-zc.catalog/branches/upstream/
python-zc.catalog/branches/upstream/current/
python-zc.catalog/branches/upstream/current/Makefile
python-zc.catalog/branches/upstream/current/PKG-INFO
python-zc.catalog/branches/upstream/current/ZopePublicLicense.txt
python-zc.catalog/branches/upstream/current/setup.py
python-zc.catalog/branches/upstream/current/src/
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/PKG-INFO
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/SOURCES.txt
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/namespace_packages.txt
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/not-zip-safe
python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/top_level.txt
python-zc.catalog/branches/upstream/current/src/zc/
python-zc.catalog/branches/upstream/current/src/zc/__init__.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/
python-zc.catalog/branches/upstream/current/src/zc/catalog/__init__.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/catalogindex.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/configure.zcml
python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.txt
python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.txt
python-zc.catalog/branches/upstream/current/src/zc/catalog/i18n.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/index.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/interfaces.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/normalizedindex.txt
python-zc.catalog/branches/upstream/current/src/zc/catalog/setindex.txt
python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.txt
python-zc.catalog/branches/upstream/current/src/zc/catalog/tests.py
python-zc.catalog/branches/upstream/current/src/zc/catalog/valueindex.txt
python-zc.catalog/branches/upstream/current/zc.catalog-configure.zcml
python-zc.catalog/tags/
Log:
[svn-inject] Installing original source of python-zc.catalog
Added: python-zc.catalog/branches/upstream/current/Makefile
===================================================================
--- python-zc.catalog/branches/upstream/current/Makefile (rev 0)
+++ python-zc.catalog/branches/upstream/current/Makefile 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,19 @@
+PYVERS=2.4
+
+all:
+ # nothing for now
+
+clean: $(PYVERS:%=clean-python%)
+ rm -rf dist
+ rm -rf build
+ rm -rf src/*.egg-info
+
+clean-python%:
+ python$* setup.py clean
+
+.PHONY: dist
+dist: clean $(PYVERS:%=build-python%-egg)
+ python setup.py sdist
+
+build-python%-egg:
+ python$* setup.py bdist_egg
Property changes on: python-zc.catalog/branches/upstream/current/Makefile
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/PKG-INFO
===================================================================
--- python-zc.catalog/branches/upstream/current/PKG-INFO (rev 0)
+++ python-zc.catalog/branches/upstream/current/PKG-INFO 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,14 @@
+Metadata-Version: 1.0
+Name: zc.catalog
+Version: 0.1.1
+Summary: zc.catalog contains a number of extensions to the Zope 3 catalog,
+such as some new indexes, improved globbing and stemming support,
+and an alternative catalog implementation.
+
+Home-page: UNKNOWN
+Author: Zope Project
+Author-email: zope3-dev at zope.org
+License: ZPL
+Description: UNKNOWN
+Keywords: zope zope3
+Platform: UNKNOWN
Added: python-zc.catalog/branches/upstream/current/ZopePublicLicense.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/ZopePublicLicense.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/ZopePublicLicense.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,54 @@
+Zope Public License (ZPL) Version 2.1
+-------------------------------------
+
+A copyright notice accompanies this license document that
+identifies the copyright holders.
+
+This license has been certified as open source. It has also
+been designated as GPL compatible by the Free Software
+Foundation (FSF).
+
+Redistribution and use in source and binary forms, with or
+without modification, are permitted provided that the
+following conditions are met:
+
+1. Redistributions in source code must retain the
+ accompanying copyright notice, this list of conditions,
+ and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the accompanying
+ copyright notice, this list of conditions, and the
+ following disclaimer in the documentation and/or other
+ materials provided with the distribution.
+
+3. Names of the copyright holders must not be used to
+ endorse or promote products derived from this software
+ without prior written permission from the copyright
+ holders.
+
+4. The right to distribute this software or to use it for
+ any purpose does not give you the right to use
+ Servicemarks (sm) or Trademarks (tm) of the copyright
+ holders. Use of them is covered by separate agreement
+ with the copyright holders.
+
+5. If any files are modified, you must cause the modified
+ files to carry prominent notices stating that you changed
+ the files and the date of any change.
+
+Disclaimer
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS''
+ AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT
+ NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+ AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN
+ NO EVENT SHALL THE COPYRIGHT HOLDERS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ DAMAGE.
Property changes on: python-zc.catalog/branches/upstream/current/ZopePublicLicense.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/setup.py
===================================================================
--- python-zc.catalog/branches/upstream/current/setup.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/setup.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,25 @@
+from setuptools import setup, find_packages
+
+setup(
+ name="zc.catalog",
+ version="0.1.1",
+ packages=find_packages('src', exclude=["*.tests", "*.ftests"]),
+
+ package_dir= {'':'src'},
+
+ namespace_packages=['zc'],
+ package_data = {
+ '': ['*.txt', '*.zcml'],
+ },
+
+ zip_safe=False,
+ author='Zope Project',
+ author_email='zope3-dev at zope.org',
+ description="""\
+zc.catalog contains a number of extensions to the Zope 3 catalog,
+such as some new indexes, improved globbing and stemming support,
+and an alternative catalog implementation.
+""",
+ license='ZPL',
+ keywords="zope zope3",
+ )
Property changes on: python-zc.catalog/branches/upstream/current/setup.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/__init__.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/__init__.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/__init__.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,5 @@
+# this is a namespace package
+try:
+ __import__('pkg_resources').declare_namespace(__name__)
+except ImportError:
+ pass
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/__init__.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/__init__.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/__init__.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/__init__.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1 @@
+#
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/__init__.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/catalogindex.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/catalogindex.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/catalogindex.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,67 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Indexes appropriate for zope.app.catalog
+
+$Id: catalogindex.py 2918 2005-07-19 22:12:38Z jim $
+"""
+import zope.interface
+
+import zope.app.container.contained
+import zope.app.catalog.attribute
+
+import zc.catalog.index
+import zc.catalog.interfaces
+
+class ValueIndex(zope.app.catalog.attribute.AttributeIndex,
+ zc.catalog.index.ValueIndex,
+ zope.app.container.contained.Contained):
+
+ zope.interface.implements(zc.catalog.interfaces.ICatalogValueIndex)
+
+class SetIndex(zope.app.catalog.attribute.AttributeIndex,
+ zc.catalog.index.SetIndex,
+ zope.app.container.contained.Contained):
+
+ zope.interface.implements(zc.catalog.interfaces.ICatalogSetIndex)
+
+class NormalizationWrapper(
+ zope.app.catalog.attribute.AttributeIndex,
+ zc.catalog.index.NormalizationWrapper,
+ zope.app.container.contained.Contained):
+
+ pass
+
+ at zope.interface.implementer(
+ zope.interface.implementedBy(NormalizationWrapper),
+ zc.catalog.interfaces.IValueIndex)
+def DateTimeValueIndex(
+ field_name=None, interface=None, field_callable=False,
+ resolution=2): # hour; good for per-day searches
+ ix = NormalizationWrapper(
+ field_name, interface, field_callable, zc.catalog.index.ValueIndex(),
+ zc.catalog.index.DateTimeNormalizer(resolution), False)
+ zope.interface.directlyProvides(ix, zc.catalog.interfaces.IValueIndex)
+ return ix
+
+ at zope.interface.implementer(
+ zope.interface.implementedBy(NormalizationWrapper),
+ zc.catalog.interfaces.ISetIndex)
+def DateTimeSetIndex(
+ field_name=None, interface=None, field_callable=False,
+ resolution=2): # hour; good for per-day searches
+ ix = NormalizationWrapper(
+ field_name, interface, field_callable, zc.catalog.index.SetIndex(),
+ zc.catalog.index.DateTimeNormalizer(resolution), False)
+ zope.interface.directlyProvides(ix, zc.catalog.interfaces.ISetIndex)
+ return ix
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/catalogindex.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/configure.zcml
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/configure.zcml (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/configure.zcml 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,70 @@
+<configure
+ xmlns="http://namespaces.zope.org/zope"
+ xmlns:browser="http://namespaces.zope.org/browser"
+ i18n_domain="zope"
+ >
+
+ <localUtility class=".extentcatalog.Catalog">
+ <factory
+ id="zc.catalog.extentcatalog"
+ />
+ <require
+ interface="zope.app.catalog.interfaces.ICatalogQuery"
+ permission="zope.Public"
+ />
+ <require
+ interface="zope.app.catalog.interfaces.ICatalogEdit"
+ permission="zope.ManageServices"
+ />
+ <require
+ interface="zope.app.container.interfaces.IContainer"
+ permission="zope.ManageServices"
+ />
+ </localUtility>
+
+ <class class=".extentcatalog.FilterExtent">
+ <require
+ attributes="addable __iter__ __or__ __ror__ union __and__ __rand__
+ intersection __sub__ difference __rsub__ rdifference
+ __nonzero__ __contains__"
+ permission="zope.View"/>
+ <require
+ attributes="add remove discard clear"
+ permission="zope.ManageServices"
+ />
+ </class>
+
+ <class class=".catalogindex.ValueIndex">
+ <require
+ permission="zope.ManageServices"
+ interface="zope.app.catalog.interfaces.IAttributeIndex
+ zope.index.interfaces.IStatistics"
+ set_schema="zope.app.catalog.interfaces.IAttributeIndex"
+ />
+ </class>
+
+ <class class=".catalogindex.SetIndex">
+ <require
+ permission="zope.ManageServices"
+ interface="zope.app.catalog.interfaces.IAttributeIndex
+ zope.index.interfaces.IStatistics"
+ set_schema="zope.app.catalog.interfaces.IAttributeIndex"
+ />
+ </class>
+
+ <class class=".catalogindex.NormalizationWrapper">
+ <require
+ permission="zope.ManageServices"
+ interface="zope.app.catalog.interfaces.IAttributeIndex
+ zope.index.interfaces.IStatistics"
+ set_schema="zope.app.catalog.interfaces.IAttributeIndex"
+ />
+ </class>
+
+ <class class="BTrees.Length.Length">
+ <require
+ permission="zope.ManageServices"
+ attributes="__call__" />
+ </class>
+
+</configure>
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/configure.zcml
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,203 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""extent catalog
+
+$Id: extentcatalog.py 3296 2005-09-09 19:29:20Z benji $
+"""
+
+from BTrees import IFBTree
+import persistent
+import persistent.dict
+from zope import interface, component
+from zope.app.catalog import catalog
+from zope.app.intid.interfaces import IIntIds
+from zope.app import zapi
+import zope.app.component.hooks
+
+from zc.catalog import interfaces
+
+class FilterExtent(persistent.Persistent):
+ interface.implements(interfaces.IFilterExtent)
+ __parent__ = None
+
+ def __init__(self, filter):
+ self.set = IFBTree.IFTreeSet()
+ self.filter = filter
+
+ def addable(self, uid, obj):
+ return self.filter(self, uid, obj)
+
+ def clear(self):
+ self.set.clear()
+
+ def __or__(self, other):
+ "extent | set"
+ return self.union(other)
+
+ __ror__ = __or__
+
+ def union(self, other, self_weight=1, other_weight=1):
+ return IFBTree.weightedUnion(
+ self.set, other, self_weight, other_weight)[1]
+
+ def __and__(self, other):
+ "extent & set"
+ return self.intersection(other)
+
+ __rand__ = __and__
+
+ def intersection(self, other, self_weight=1, other_weight=1):
+ return IFBTree.weightedIntersection(
+ self.set, other, self_weight, other_weight)[1]
+
+ def __sub__(self, other):
+ "extent - set"
+ return self.difference(other)
+
+ def difference(self, other):
+ return IFBTree.difference(self.set, other)
+
+ def __rsub__(self, other):
+ "set - extent"
+ return self.rdifference(other)
+
+ def rdifference(self, other):
+ return IFBTree.difference(other, self.set)
+
+ def __iter__(self):
+ return iter(self.set)
+
+ def __nonzero__(self):
+ return bool(self.set)
+
+ def __contains__(self, uid):
+ return self.set.has_key(uid)
+
+ def add(self, uid, obj):
+ if not self.addable(uid, obj):
+ raise ValueError
+ else:
+ self.set.insert(uid)
+
+ def remove(self, uid):
+ self.set.remove(uid)
+
+ def discard(self, uid):
+ try:
+ self.set.remove(uid)
+ except KeyError:
+ pass
+
+UNINDEX = object()
+
+class Catalog(catalog.Catalog):
+ interface.implements(interfaces.IExtentCatalog)
+
+ def __init__(self, extent):
+ if extent.__parent__ is not None:
+ raise ValueError("extent's __parent__ must be None")
+ super(Catalog, self).__init__()
+ self.extent = extent
+ extent.__parent__ = self # inform extent of catalog
+ self.queue = persistent.dict.PersistentDict()
+
+ def clear(self):
+ self.extent.clear()
+ super(Catalog, self).clear()
+
+ def _register(self):
+ """Try to register to process queue later.
+
+ Return whether we succeeded.
+ """
+ connection = self._p_jar
+ if connection is None:
+ return False
+
+ try:
+ tm = connection._txn_mgr
+ except AttributeError:
+ tm = connection.transaction_manager
+
+ tm.get().addBeforeCommitHook(self._process)
+ return True
+
+ def _process(self):
+ current_site = old_site = zope.app.component.hooks.getSite()
+ try:
+ for docid, (obj, site) in self.queue.items():
+ if current_site is not site:
+ zope.app.component.hooks.setSite(site)
+ current_site = site
+ if obj is not UNINDEX:
+ super(Catalog, self).index_doc(docid, obj)
+ elif docid in self.extent:
+ super(Catalog, self).unindex_doc(docid)
+ self.extent.remove(docid)
+ self.queue.clear()
+ finally:
+ zope.app.component.hooks.setSite(old_site)
+
+ def index_doc(self, docid, texts):
+ """Register the data in indexes of this catalog.
+ """
+
+ registered = True
+ if not self.queue:
+ registered = self._register()
+
+ try:
+ self.extent.add(docid, texts)
+ except ValueError:
+ #self.unindex_doc(docid)
+ self.queue[docid] = (UNINDEX, zope.app.component.hooks.getSite())
+ else:
+ #super(Catalog, self).index_doc(docid, texts)
+ self.queue[docid] = (texts, zope.app.component.hooks.getSite())
+
+ if not registered:
+ self._process()
+
+
+ def unindex_doc(self, docid):
+ """Unregister the data from indexes of this catalog."""
+
+ registered = True
+ if not self.queue:
+ registered = self._register()
+
+ self.queue[docid] = (UNINDEX, zope.app.component.hooks.getSite())
+
+ if not registered:
+ self._process()
+
+ def updateIndex(self, index):
+ if index.__parent__ is not self:
+ # not an index in us. Let the superclass handle it.
+ super(Catalog, self).updateIndex(index)
+ else:
+ uidutil = zapi.getUtility(IIntIds)
+ for uid in uidutil:
+ obj = uidutil.getObject(uid)
+ try:
+ self.extent.add(uid, obj)
+ except ValueError:
+ self.unindex_doc(uid)
+ else:
+ index.index_doc(uid, obj)
+
+ def updateIndexes(self):
+ uidutil = zapi.getUtility(IIntIds)
+ for uid in uidutil:
+ self.index_doc(uid, uidutil.getObject(uid))
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,209 @@
+An extent catalog is very similar to a normal catalog except that it only
+indexes items addable to its extent. The extent is both a filter and a
+set that may be merged with other result sets.
+
+To show the extent catalog at work, we need an intid utility, an index,
+some items to index, and a filter that determines what the extent accepts.
+
+ >>> from zc.catalog import interfaces, extentcatalog
+ >>> from zope import interface, component
+ >>> from zope.interface import verify
+ >>> import zope.app.intid.interfaces
+ >>> class DummyIntId(object):
+ ... interface.implements(zope.app.intid.interfaces.IIntIds)
+ ... MARKER = '__dummy_int_id__'
+ ... def __init__(self):
+ ... self.counter = 0
+ ... self.data = {}
+ ... def register(self, obj):
+ ... intid = getattr(obj, self.MARKER, None)
+ ... if intid is None:
+ ... setattr(obj, self.MARKER, self.counter)
+ ... self.data[self.counter] = obj
+ ... intid = self.counter
+ ... self.counter += 1
+ ... return intid
+ ... def getObject(self, intid):
+ ... return self.data[intid]
+ ... def __iter__(self):
+ ... return iter(self.data)
+ ...
+ >>> intid = DummyIntId()
+ >>> component.provideUtility(
+ ... intid, zope.app.intid.interfaces.IIntIds)
+ >>> import sets
+ >>> from zope.app.container.interfaces import IContained
+ >>> class DummyIndex(object):
+ ... interface.implements(IContained)
+ ... __parent__ = __name__ = None
+ ... def __init__(self):
+ ... self.uids = sets.Set()
+ ... def unindex_doc(self, uid):
+ ... self.uids.discard(uid)
+ ... def index_doc(self, uid, obj):
+ ... self.uids.add(uid)
+ ... def clear(self):
+ ... self.uids.clear()
+ ...
+ >>> class DummyContent(object):
+ ... pass
+ ...
+ >>> content = {}
+ >>> for i in range(100):
+ ... c = DummyContent()
+ ... content[intid.register(c)] = c
+ ...
+ >>> def filter(extent, uid, ob):
+ ... assert interfaces.IExtent.providedBy(extent)
+ ... assert getattr(ob, DummyIntId.MARKER) == uid
+ ... # This is an extent of objects with odd-numbered uids without a
+ ... # True ignore attribute
+ ... return uid % 2 and not getattr(ob, 'ignore', False)
+ >>> extent = extentcatalog.FilterExtent(filter)
+ >>> verify.verifyObject(interfaces.IFilterExtent, extent)
+ True
+ >>> catalog = extentcatalog.Catalog(extent)
+ >>> verify.verifyObject(interfaces.IExtentCatalog, catalog)
+ True
+ >>> index = DummyIndex()
+ >>> catalog['index'] = index
+
+Now we have a catalog set up with an index and an extent, and some content to
+index. If we ask the catalog to index all of the content, only the ones that
+match the filter will be in the extent and in the index.
+
+ >>> for c in content.values():
+ ... catalog.index_doc(intid.register(c), c)
+ ...
+ >>> matches = list(sorted(
+ ... [id for id, ob in content.items() if filter(extent, id, ob)]))
+ >>> list(sorted(extent)) == list(sorted(index.uids)) == matches
+ True
+
+If a content object is indexed that used to match the filter but no longer
+does, it should be removed from the extent and indexes.
+
+ >>> 5 in catalog.extent
+ True
+ >>> content[5].ignore = True
+ >>> catalog.index_doc(5, content[5])
+ >>> 5 in catalog.extent
+ False
+ >>> matches.remove(5)
+ >>> list(sorted(extent)) == list(sorted(index.uids)) == matches
+ True
+
+Unindexing an object that is in the catalog should simply remove it from the
+catalog and index as usual.
+
+ >>> 99 in catalog.extent
+ True
+ >>> 99 in catalog['index'].uids
+ True
+ >>> catalog.unindex_doc(99)
+ >>> 99 in catalog.extent
+ False
+ >>> 99 in catalog['index'].uids
+ False
+ >>> matches.remove(99)
+ >>> list(sorted(extent)) == list(sorted(index.uids)) == matches
+ True
+
+And similarly, unindexing an object that is not in the catalog should be a
+no-op.
+
+ >>> 0 in catalog.extent
+ False
+ >>> catalog.unindex_doc(0)
+ >>> 0 in catalog.extent
+ False
+ >>> list(sorted(extent)) == list(sorted(index.uids)) == matches
+ True
+
+Clearing the catalog clears both the extent and the contained indexes.
+
+ >>> catalog.clear()
+ >>> list(catalog.extent) == list(catalog['index'].uids) == []
+ True
+
+Updating all indexes and an individual index both also update the extent.
+
+ >>> catalog.updateIndexes()
+ >>> matches.append(99)
+ >>> list(sorted(extent)) == list(sorted(index.uids)) == matches
+ True
+ >>> index2 = DummyIndex()
+ >>> catalog['index2'] = index2
+ >>> index.uids.remove(1) # to confirm that only index 2 is touched
+ >>> catalog.updateIndex(index2)
+ >>> list(sorted(extent)) == list(sorted(index2.uids)) == matches
+ True
+ >>> 1 in index.uids
+ False
+ >>> 1 in index2.uids
+ True
+ >>> index.uids.add(1) # normalize things again.
+
+If you update a single index and an object is no longer a member of the extent,
+it is removed from all indexes.
+
+ >>> 1 in catalog.extent
+ True
+ >>> 1 in index.uids
+ True
+ >>> 1 in index2.uids
+ True
+ >>> content[1].ignore = True
+ >>> catalog.updateIndex(index2)
+ >>> 1 in catalog.extent
+ False
+ >>> 1 in index.uids
+ False
+ >>> 1 in index2.uids
+ False
+ >>> matches.remove(1)
+ >>> matches == list(sorted(catalog.extent))
+ True
+
+The extent itself provides a number of merging features to allow its values to
+be merged with other BTrees.IFBTree data structures. These include
+intersection, union, difference, and reverse difference. Given an extent
+named 'extent' and another IFBTree data structure named 'data', intersections
+can be spelled "extent & data" or "data & extent"; unions can be spelled
+"extent | data" or "data | extent"; differences can be spelled "extent - data";
+and reverse differences can be spelled "data - extent". Unions and
+intersections are weighted.
+
+ >>> from BTrees import IFBTree
+ >>> alt_set = IFBTree.IFTreeSet()
+ >>> alt_set.update(range(0, 166, 33)) # return value is unimportant here
+ 6
+ >>> list(sorted(alt_set))
+ [0, 33, 66, 99, 132, 165]
+ >>> list(sorted(catalog.extent & alt_set))
+ [33, 99]
+ >>> list(sorted(alt_set & catalog.extent))
+ [33, 99]
+ >>> list(sorted(catalog.extent.intersection(alt_set)))
+ [33, 99]
+ >>> union_matches = sets.Set(matches)
+ >>> union_matches.union_update(alt_set)
+ >>> union_matches = list(sorted(union_matches))
+ >>> list(sorted(alt_set | catalog.extent)) == union_matches
+ True
+ >>> list(sorted(catalog.extent | alt_set)) == union_matches
+ True
+ >>> list(sorted(catalog.extent.union(alt_set))) == union_matches
+ True
+ >>> list(sorted(alt_set - catalog.extent))
+ [0, 66, 132, 165]
+ >>> list(sorted(catalog.extent.rdifference(alt_set)))
+ [0, 66, 132, 165]
+ >>> matches.remove(33)
+ >>> matches.remove(99)
+ >>> list(sorted(catalog.extent - alt_set)) == matches
+ True
+ >>> list(sorted(catalog.extent.difference(alt_set))) == matches
+ True
+ >>> from zope.app.testing import ztapi
+ >>> ztapi.unprovideUtility(zope.app.intid.interfaces.IIntIds)
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/extentcatalog.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,40 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""glob all terms at the end.
+
+$Id: globber.py 2918 2005-07-19 22:12:38Z jim $
+"""
+from zope.index.text import queryparser, parsetree
+
+reconstitute = {}
+reconstitute["NOT"] = lambda nd: "not %s" % (
+ reconstitute[nd.getValue().nodeType()](nd.getValue()),)
+reconstitute["AND"] = lambda nd: "(%s)" % (" and ".join(expand(nd)),)
+reconstitute["OR"] = lambda nd: "(%s)" % (" or ".join(expand(nd)),)
+reconstitute["ATOM"] = lambda nd: '%s*' % (nd.getValue())
+reconstitute["PHRASE"] = lambda nd: '"%s"' % (
+ ' '.join((v + '*') for v in nd.getValue()),)
+reconstitute["GLOB"] = lambda nd: nd.getValue()
+
+expand = lambda nd: [reconstitute[n.nodeType()](n) for n in nd.getValue()]
+
+def glob(query, lexicon): # lexicon is index.lexicon
+ try:
+ tree = queryparser.QueryParser(lexicon).parseQuery(query)
+ except parsetree.ParseError:
+ return None
+ if tree is not None:
+ return reconstitute[tree.nodeType()](tree)
+ else:
+ return None
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,12 @@
+The globber takes a query and makes any term that isn't already a glob into
+something that ends in a star. It was originally envisioned as a *very* low-
+rent stemming hack. The author now questions its value, and hopes that the new
+stemming pipeline option can be used instead. Nonetheless, here is an example
+of it at work.
+
+ >>> from zope.index.text import textindex
+ >>> index = textindex.TextIndex()
+ >>> lex = index.lexicon
+ >>> from zc.catalog import globber
+ >>> globber.glob('foo bar and baz or (b?ng not boo)', lex)
+ '(((foo* and bar*) and baz*) or (b?ng and not boo*))'
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/globber.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/i18n.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/i18n.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/i18n.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,31 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""I18N support for tasks.
+
+This defines a `MessageFactory` for the I18N domain for the catalog
+package. This is normally used with this import::
+
+ from i18n import MessageFactory as _
+
+The factory is then used normally. Two examples::
+
+ text = _('some internationalized text')
+ text = _('helpful-descriptive-message-id', 'default text')
+"""
+__docformat__ = "reStructuredText"
+
+
+from zope import i18nmessageid
+
+MessageFactory = _ = i18nmessageid.MessageFactory("zc.catalog")
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/i18n.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/index.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/index.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/index.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,446 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""indexes, as might be found in zope.index
+
+$Id: index.py 2918 2005-07-19 22:12:38Z jim $
+"""
+import datetime
+import pytz.reference
+import persistent
+from BTrees import IFBTree, OOBTree, IOBTree, Length
+
+from zope import component, interface
+import zope.interface.common.idatetime
+import zope.index.interfaces
+import zope.security.management
+from zope.publisher.interfaces import IRequest
+
+import zc.catalog.interfaces
+from zc.catalog.i18n import _
+
+class AbstractIndex(persistent.Persistent):
+
+ interface.implements(zope.index.interfaces.IInjection,
+ zope.index.interfaces.IIndexSearch,
+ zope.index.interfaces.IStatistics,
+ zc.catalog.interfaces.IIndexValues,
+ )
+
+ def __init__(self):
+ self.clear()
+
+ def clear(self):
+ self.values_to_documents = OOBTree.OOBTree()
+ self.documents_to_values = IOBTree.IOBTree()
+ self.documentCount = Length.Length(0)
+ self.wordCount = Length.Length(0)
+
+ def minValue(self, min=None):
+ if min is None:
+ return self.values_to_documents.minKey()
+ else:
+ return self.values_to_documents.minKey(min)
+
+ def maxValue(self, max=None):
+ if max is None:
+ return self.values_to_documents.maxKey()
+ else:
+ return self.values_to_documents.maxKey(max)
+
+ def values(self, min=None, max=None, excludemin=False, excludemax=False,
+ doc_id=None):
+ if doc_id is None:
+ return iter(self.values_to_documents.keys(
+ min, max, excludemin, excludemax))
+ else:
+ values = self.documents_to_values.get(doc_id)
+ if values is None:
+ return ()
+ else:
+ return iter(values.keys(min, max, excludemin, excludemax))
+
+ def containsValue(self, value):
+ return bool(self.values_to_documents.has_key(value))
+
+ def ids(self):
+ return self.documents_to_values.keys()
+
+def parseQuery(query):
+ if isinstance(query, dict):
+ if len(query) > 1:
+ raise ValueError(
+ 'may only pass one of key, value pair')
+ elif not query:
+ return None, None
+ query_type, query = query.items()[0]
+ query_type = query_type.lower()
+ else:
+ raise ValueError('may only pass a dict to apply')
+ return query_type, query
+
+class ValueIndex(AbstractIndex):
+
+ interface.implements(zc.catalog.interfaces.IValueIndex)
+
+ def _add_value(self, doc_id, added):
+ values_to_documents = self.values_to_documents
+ docs = values_to_documents.get(added)
+ if docs is None:
+ values_to_documents[added] = IFBTree.IFTreeSet((doc_id,))
+ self.wordCount.change(1)
+ else:
+ docs.insert(doc_id)
+
+ def index_doc(self, doc_id, value):
+ if value is None:
+ self.unindex_doc(doc_id)
+ else:
+ values_to_documents = self.values_to_documents
+ documents_to_values = self.documents_to_values
+ old = documents_to_values.get(doc_id)
+ documents_to_values[doc_id] = value
+ if old is None:
+ self.documentCount.change(1)
+ elif old != value:
+ docs = values_to_documents.get(old)
+ docs.remove(doc_id)
+ if not docs:
+ del values_to_documents[old]
+ self.wordCount.change(-1)
+ self._add_value(doc_id, value)
+
+ def unindex_doc(self, doc_id):
+ documents_to_values = self.documents_to_values
+ value = documents_to_values.get(doc_id)
+ if value is not None:
+ values_to_documents = self.values_to_documents
+ self.documentCount.change(-1)
+ del documents_to_values[doc_id]
+ docs = values_to_documents.get(value)
+ docs.remove(doc_id)
+ if not docs:
+ del values_to_documents[value]
+ self.wordCount.change(-1)
+
+ def apply(self, query): # any_of, any, between, none,
+ values_to_documents = self.values_to_documents
+ query_type, query = parseQuery(query)
+ if query_type is None:
+ res = None
+ elif query_type == 'any_of':
+ res = IFBTree.multiunion(
+ [s for s in (values_to_documents.get(v) for v in query)
+ if s is not None])
+ elif query_type == 'any':
+ if query is None:
+ res = IFBTree.IFSet(self.ids())
+ else:
+ assert zc.catalog.interfaces.IExtent.providedBy(query)
+ res = query & IFBTree.IFSet(self.ids())
+ elif query_type == 'between':
+ res = IFBTree.multiunion(
+ [s for s in (values_to_documents.get(v) for v in
+ values_to_documents.keys(*query))
+ if s is not None])
+ elif query_type == 'none':
+ assert zc.catalog.interfaces.IExtent.providedBy(query)
+ res = query - IFBTree.IFSet(self.ids())
+ else:
+ raise ValueError(
+ "unknown query type", query_type)
+ return res
+
+ def values(self, min=None, max=None, excludemin=False, excludemax=False,
+ doc_id=None):
+ if doc_id is None:
+ return iter(self.values_to_documents.keys(
+ min, max, excludemin, excludemax))
+ else:
+ value = self.documents_to_values.get(doc_id)
+ if (value is None or
+ min is not None and (
+ value < min or excludemin and value == min) or
+ max is not None and (
+ value > max or excludemax and value == max)):
+ return ()
+ else:
+ return (value,)
+
+class SetIndex(AbstractIndex):
+
+ interface.implements(zc.catalog.interfaces.ISetIndex)
+
+ def _add_values(self, doc_id, added):
+ values_to_documents = self.values_to_documents
+ for v in added:
+ docs = values_to_documents.get(v)
+ if docs is None:
+ values_to_documents[v] = IFBTree.IFTreeSet((doc_id,))
+ self.wordCount.change(1)
+ else:
+ docs.insert(doc_id)
+
+ def index_doc(self, doc_id, value):
+ new = OOBTree.OOTreeSet(v for v in value if v is not None)
+ if not new:
+ self.unindex_doc(doc_id)
+ else:
+ values_to_documents = self.values_to_documents
+ documents_to_values = self.documents_to_values
+ old = documents_to_values.get(doc_id)
+ if old is None:
+ documents_to_values[doc_id] = new
+ self.documentCount.change(1)
+ self._add_values(doc_id, new)
+ else:
+ removed = OOBTree.difference(old, new)
+ added = OOBTree.difference(new, old)
+ for v in removed:
+ old.remove(v)
+ docs = values_to_documents.get(v)
+ docs.remove(doc_id)
+ if not docs:
+ del values_to_documents[v]
+ self.wordCount.change(-1)
+ old.update(added)
+ self._add_values(doc_id, added)
+
+ def unindex_doc(self, doc_id):
+ documents_to_values = self.documents_to_values
+ values = documents_to_values.get(doc_id)
+ if values is not None:
+ values_to_documents = self.values_to_documents
+ self.documentCount.change(-1)
+ del documents_to_values[doc_id]
+ for v in values:
+ docs = values_to_documents.get(v)
+ docs.remove(doc_id)
+ if not docs:
+ del values_to_documents[v]
+ self.wordCount.change(-1)
+
+ def apply(self, query): # any_of, any, between, none, all_of
+ values_to_documents = self.values_to_documents
+ query_type, query = parseQuery(query)
+ if query_type is None:
+ res = None
+ elif query_type == 'any_of':
+ res = IFBTree.IFBucket()
+ for v in query:
+ _, res = IFBTree.weightedUnion(
+ res, values_to_documents.get(v))
+ elif query_type == 'any':
+ if query is None:
+ res = IFBTree.IFSet(self.ids())
+ else:
+ assert zc.catalog.interfaces.IExtent.providedBy(query)
+ res = query & IFBTree.IFSet(self.ids())
+ elif query_type == 'all_of':
+ res = None
+ values = iter(query)
+ try:
+ res = values_to_documents.get(values.next())
+ except StopIteration:
+ res = IFBTree.IFTreeSet()
+ while res:
+ try:
+ v = values.next()
+ except StopIteration:
+ break
+ res = IFBTree.intersection(res, values_to_documents.get(v))
+ elif query_type == 'between':
+ res = IFBTree.IFBucket()
+ for v in values_to_documents.keys(*query):
+ _, res = IFBTree.weightedUnion(res, values_to_documents.get(v))
+ elif query_type == 'none':
+ assert zc.catalog.interfaces.IExtent.providedBy(query)
+ res = query - IFBTree.IFSet(self.ids())
+ else:
+ raise ValueError(
+ "unknown query type", query_type)
+ return res
+
+class NormalizationWrapper(persistent.Persistent):
+
+ interface.implements(zc.catalog.interfaces.INormalizationWrapper)
+
+ index = normalizer = None
+ collection_index = False
+
+ def documentCount(self):
+ return self.index.documentCount()
+
+ def wordCount(self):
+ return self.index.wordCount()
+
+ def __init__(self, index, normalizer, collection_index=False):
+ self.index = index
+ self.normalizer = normalizer
+ self.collection_index = collection_index
+
+ def index_doc(self, doc_id, value):
+ if self.collection_index:
+ self.index.index_doc(
+ doc_id, (self.normalizer.value(v) for v in value))
+ else:
+ self.index.index_doc(doc_id, self.normalizer.value(value))
+
+ def unindex_doc(self, doc_id):
+ self.index.unindex_doc(doc_id)
+
+ def apply(self, query):
+ query_type, query = parseQuery(query)
+ if query_type == 'any_of':
+ res = set()
+ for v in query:
+ res.update(self.normalizer.any(v, self.index))
+ elif query_type == 'all_of':
+ res = [self.normalizer.all(v, self.index) for v in query]
+ elif query_type == 'between':
+ query = tuple(query) # collect iterators
+ len_query = len(query)
+ max_exclude = len_query >= 4 and bool(query[3])
+ min_exclude = len_query >= 3 and bool(query[2])
+ max = len_query >= 2 and query[1] and self.normalizer.maximum(
+ query[1], self.index, max_exclude) or None
+ min = len_query >= 1 and query[0] and self.normalizer.minimum(
+ query[0], self.index, min_exclude) or None
+ res = (min, max, min_exclude, max_exclude)
+ else:
+ res = query
+ return self.index.apply({query_type: res})
+
+ def minValue(self, min=None):
+ if min is not None:
+ min = self.normalizer.minimum(min, self.index)
+ return self.index.minValue(min)
+
+ def maxValue(self, max=None):
+ if max is not None:
+ max = self.normalizer.maximum(max, self.index)
+ return self.index.maxValue(max)
+
+ def values(self, min=None, max=None, excludemin=False, excludemax=False,
+ doc_id=None):
+ if min is not None:
+ min = self.normalizer.minimum(min, self.index)
+ if max is not None:
+ max = self.normalizer.maximum(max, self.index)
+ return self.index.values(min, max, excludemin, excludemax)
+
+ def containsValue(self, value):
+ return self.index.containsValue(value)
+
+ def ids(self):
+ return self.index.ids()
+
+def set_resolution(value, resolution):
+ resolution += 2
+ if resolution < 6:
+ args = []
+ args.extend(value.timetuple()[:resolution+1])
+ args.extend([0]*(6-resolution))
+ args.append(value.tzinfo)
+ value = datetime.datetime(*args)
+ return value
+
+def get_request():
+ i = zope.security.management.queryInteraction()
+ if i is not None:
+ for p in i.participations:
+ if IRequest.providedBy(p):
+ return p
+ return None
+
+def get_tz(default=pytz.reference.Local):
+ request = get_request()
+ if request is None:
+ return default
+ return zope.interface.common.idatetime.ITZInfo(request, default)
+
+def add_tz(value):
+ if type(value) is datetime.datetime:
+ if value.tzinfo is None:
+ value = value.replace(tzinfo=get_tz())
+ return value
+ else:
+ raise ValueError(value)
+
+def day_end(value):
+ return (
+ datetime.datetime.combine(
+ value, datetime.time(tzinfo=get_tz())) +
+ datetime.timedelta(days=1) - # separate for daylight savings
+ datetime.timedelta(microseconds=1))
+
+def day_begin(value):
+ return datetime.datetime.combine(
+ value, datetime.time(tzinfo=get_tz()))
+
+class DateTimeNormalizer(persistent.Persistent):
+
+ interface.implements(zc.catalog.interfaces.IDateTimeNormalizer)
+ def __init__(self, resolution=2):
+ self.resolution = resolution
+ # 0, 1, 2, 3, 4
+ # day, hour, minute, second, microsecond
+
+ def value(self, value):
+ if not isinstance(value, datetime.datetime) or value.tzinfo is None:
+ raise ValueError(
+ _('This index only indexes timezone-aware datetimes.'))
+ return set_resolution(value, self.resolution)
+
+ def any(self, value, index):
+ if type(value) is datetime.date:
+ start = datetime.datetime.combine(
+ value, datetime.time(tzinfo=get_tz()))
+ stop = start + datetime.timedelta(days=1)
+ return index.values(start, stop, False, True)
+ return (add_tz(value),)
+
+ def all(self, value, index):
+ return add_tz(value)
+
+ def minimum(self, value, index, exclude=False):
+ if type(value) is datetime.date:
+ if exclude:
+ return day_end(value)
+ else:
+ return day_begin(value)
+ return add_tz(value)
+
+ def maximum(self, value, index, exclude=False):
+ if type(value) is datetime.date:
+ if exclude:
+ return day_begin(value)
+ else:
+ return day_end(value)
+ return add_tz(value)
+
+ at interface.implementer(
+ zope.interface.implementedBy(NormalizationWrapper),
+ zc.catalog.interfaces.IValueIndex)
+def DateTimeValueIndex(resolution=2): # hour; good for per-day searches
+ ix = NormalizationWrapper(ValueIndex(), DateTimeNormalizer(resolution))
+ interface.directlyProvides(ix, zc.catalog.interfaces.IValueIndex)
+ return ix
+
+ at interface.implementer(
+ zope.interface.implementedBy(NormalizationWrapper),
+ zc.catalog.interfaces.ISetIndex)
+def DateTimeSetIndex(resolution=2): # hour; good for per-day searches
+ ix = NormalizationWrapper(SetIndex(), DateTimeNormalizer(resolution), True)
+ interface.directlyProvides(ix, zc.catalog.interfaces.IValueIndex)
+ return ix
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/index.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/interfaces.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/interfaces.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/interfaces.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,272 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""interfaces for zc.catalog
+
+$Id: interfaces.py 2918 2005-07-19 22:12:38Z jim $
+"""
+
+from zope import interface, schema
+from zope.schema.vocabulary import SimpleVocabulary, SimpleTerm
+import zope.index.interfaces
+import zope.app.catalog.interfaces
+from zc.catalog.i18n import _
+
+class IExtent(interface.Interface):
+
+ __parent__ = interface.Attribute(
+ """The catalog for which this is an extent; must be None before it is
+ set to a catalog""")
+
+ def addable(uid, obj):
+ """returns True or False, indicating whether the obj may be added to
+ the extent"""
+
+ def add(uid, obj):
+ """add uid to extent; raise ValueError if it is not addable.
+
+ If uid is already a member of the extent, calling add is a no-op,
+ except that if the uid and obj are no longer addable to the extent then
+ ValueError is still raised (but without removing the uid)"""
+
+ def remove(uid):
+ """Remove uid from set. Raise KeyError if not a member"""
+
+ def discard(uid):
+ """Remove uid from set. Ignore if not a member"""
+
+ def clear():
+ """Remove all uids from set."""
+
+ def __iter__():
+ """return iterator of uids in set"""
+
+ def __or__(other):
+ "Given BTrees.IFBTree data structure, return weighted union"
+
+ def __ror__(other):
+ "Given BTrees.IFBTree data structure, return weighted union"
+
+ def union(other, self_weight, other_weight):
+ "Given BTrees.IFBTree data structure, return weighted union"
+
+ def __and__(other):
+ "Given BTrees.IFBTree data structure, return weighted intersection"
+
+ def __rand__(other):
+ "Given BTrees.IFBTree data structure, return weighted intersection"
+
+ def intersection(other, self_weight, other_weight):
+ "Given BTrees.IFBTree data structure, return weighted intersection"
+
+ def __sub__(other):
+ "extent - set: given BTrees.IFBTree data structure, return difference"
+
+ def difference(other):
+ "extent - set: given BTrees.IFBTree data structure, return difference"
+
+ def __rsub__(other):
+ "set - extent: given BTrees.IFBTree data structure, return difference"
+
+ def rdifference(other):
+ "set - extent: given BTrees.IFBTree data structure, return difference"
+
+ def __nonzero__():
+ "return boolean indicating if any uids are in set"
+
+ def __contains__(uid):
+ "return boolean indicating if uid is in set"
+
+class IFilterExtent(IExtent):
+
+ filter = interface.Attribute(
+ """A (persistent) callable that is passed the extent, a docid, and the
+ associated obj and should return a boolean True (is member of extent)
+ or False (is not member of extent).""")
+
+class IExtentCatalog(interface.Interface):
+ """A catalog of only items within an extent.
+
+ Interface intended to be used with zope.app.catalog.interfaces.ICatalog"""
+
+ extent = interface.Attribute(
+ """An IExtent of the objects cataloged""")
+
+class IIndexValues(interface.Interface):
+ """An index that allows introspection of the indexed values"""
+
+ def minValue(min=None):
+ """return the minimum value in the index.
+
+ if min is provided, return the minimum value equal to or greater than
+ min.
+
+ Raises ValueError if no min.
+ """
+
+ def maxValue(max=None):
+ """return the maximum value in the index.
+
+ If max is provided, return the maximum value equal to or less than max.
+
+ Raises ValueError if no max.
+ """
+
+ def values(min=None, max=None, excludemin=False, excludemax=False,
+ doc_id=None):
+ """return an iterables of the values in the index.
+
+ if doc_id is provided, returns the values only for that document id.
+
+ If a min is specified, then output is constrained to values greater
+ than or equal to the given min, and, if excludemin is specified and
+ true, is further constrained to values strictly greater than min. A
+ min value of None is ignored. If min is None or not specified, and
+ excludemin is true, the smallest value is excluded.
+
+ If a max is specified, then output is constrained to values less than
+ or equal to the given max, and, if excludemax is specified and
+ true, is further constrained to values strictly less than max. A max
+ value of None is ignored. If max is None or not specified, and
+ excludemax is true, the largest value is excluded.
+ """
+
+ def containsValue(value):
+ """whether the value is used in any of the documents in the index"""
+
+ def ids():
+ """return a BTrees.IFBTree data structure of the document ids in the
+ index--the ones that have values to be indexed. All document ids
+ should produce at least one value given a call of
+ IIndexValues.values(doc_id=id).
+ """
+
+class ISetIndex(interface.Interface):
+
+ def apply(query):
+ """Return None or an IFBTree Set of the doc ids that match the query.
+
+ query is a dict with one of the following keys: any_of, any,
+ all_of, between, and none.
+
+ Any one of the keys may be used; using more than one is not allowed.
+
+ The any_of key should have a value of an iterable of values: the
+ result will be the docids whose values contain any of the given values.
+
+ The all_of key should have a value of an iterable of values: the
+ result will be the docids whose values contain all of the given values.
+
+ The between key should have a value of an iterable of one to four
+ members. The first is the minimum value, or None; the second is the
+ maximum value, or None; the third is boolean, defaulting to False,
+ declaring if the min should be excluded; and the last is also boolean,
+ defaulting to False, declaring if the max should be excluded.
+
+ The any key should take None or an extent. If the key is None, the
+ results will be all docids with any value. If the key is an extent,
+ the results will be the intersection of the extent and all docids with
+ any value.
+
+ The none key should take an extent. It returns the docids in
+ the extent that do not have any values in the index.
+ """
+
+class IValueIndex(interface.Interface):
+
+ def apply(query):
+ """Return None or an IFBTree Set of the doc ids that match the query.
+
+ query is a dict with one of the following keys: any_of, any,
+ between, and none.
+
+ Any one of the keys may be used; using more than one is not allowed.
+
+ The any_of key should have a value of an iterable of values: the
+ result will be the docids whose values contain any of the given values.
+
+ The between key should have a value of an iterable of one to four
+ members. The first is the minimum value, or None; the second is the
+ maximum value, or None; the third is boolean, defaulting to False,
+ declaring if the min should be excluded; and the last is also boolean,
+ defaulting to False, declaring if the max should be excluded.
+
+ The any key should take None or an extent. If the key is None, the
+ results will be all docids with any value. If the key is an extent,
+ the results will be the intersection of the extent and all docids with
+ any value.
+
+ The none key should take an extent. It returns the docids in
+ the extent that do not have any values in the index.
+ """
+
+class ICatalogValueIndex(zope.app.catalog.interfaces.IAttributeIndex,
+ zope.app.catalog.interfaces.ICatalogIndex):
+ """Interface-based catalog value index
+ """
+
+class ICatalogSetIndex(zope.app.catalog.interfaces.IAttributeIndex,
+ zope.app.catalog.interfaces.ICatalogIndex):
+ """Interface-based catalog set index
+ """
+
+class INormalizationWrapper(zope.index.interfaces.IInjection,
+ zope.index.interfaces.IIndexSearch,
+ zope.index.interfaces.IStatistics,
+ IIndexValues):
+ """A wrapper for an index that uses a normalizer to normalize injection
+ and querying."""
+
+ index = interface.Attribute(
+ """an index implementing IInjection, IIndexSearch, IStatistics, and
+ IIndexValues""")
+
+ normalizer = interface.Attribute("a normalizer, implementing INormalizer")
+
+ collection_index = interface.Attribute(
+ """boolean: whether indexed values should be treated as collections
+ (each composite value normalized) or not (original value is
+ normalized)""")
+
+class INormalizer(interface.Interface):
+
+ def value(value):
+ """normalize or check constraints for an input value; raise an error
+ or return the value to be indexed."""
+
+ def any(value, index):
+ """normalize a query value for a "any_of" search; return a sequence of
+ values."""
+
+ def all(value, index):
+ """Normalize a query value for an "all_of" search; return the value
+ for query"""
+
+ def minimum(value, index, exclude=False):
+ """normalize a query value for minimum of a range; return the value for
+ query"""
+
+ def maximum(value, index, exclude=False):
+ """normalize a query value for maximum of a range; return the value for
+ query"""
+
+resolution_vocabulary = SimpleVocabulary([SimpleTerm(i, t, t) for i, t in enumerate(
+ (_('day'), _('hour'), _('minute'), _('second'), _('microsecond')))])
+ # 0 1 2 3 4
+
+class IDateTimeNormalizer(INormalizer):
+ resolution = schema.Choice(
+ vocabulary=resolution_vocabulary,
+ title=_('Resolution'),
+ default=2,
+ required=True)
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/interfaces.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/normalizedindex.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/normalizedindex.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/normalizedindex.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,320 @@
+The index module provides a normalizing wrapper, a DateTime normalizer, and
+a set index and a value index normalized with the DateTime normalizer.
+
+The normalizing wrapper implements a full complement of index interfaces--
+zope.index.interfaces.IInjection, zope.index.interfaces.IIndexSearch,
+zope.index.interfaces.IStatistics, and zc.catalog.interfaces.IIndexValues--
+and delegates all of the behavior to the wrapped index, normalizing values
+using the normalizer before the index sees them.
+
+The normalizing wrapper currently only supports queries offered by
+zc.catalog.interfaces.ISetIndex and zc.catalog.interfaces.IValueIndex.
+
+The normalizer interface requires the following methods, as defined in the
+interface:
+
+ def value(value):
+ """normalize or check constraints for an input value; raise an error
+ or return the value to be indexed."""
+
+ def any(value, index):
+ """normalize a query value for a "any_of" search; return a sequence of
+ values."""
+
+ def all(value, index):
+ """Normalize a query value for an "all_of" search; return the value
+ for query"""
+
+ def minimum(value, index):
+ """normalize a query value for minimum of a range; return the value for
+ query"""
+
+ def maximum(value, index):
+ """normalize a query value for maximum of a range; return the value for
+ query"""
+
+The DateTime normalizer performs the following normalizations and validations.
+Whenever a timezone is needed, it tries to get a request from the current
+interaction and adapt it to zope.interface.common.idatetime.ITZInfo; failing
+that (no request or no adapter) it uses the system local timezone.
+
+- input values must be datetimes with a timezone. They are normalized to the
+ resolution specified when the normalizer is created: a resolution of 0
+ normalizes values to days; a resolution of 1 to hours; 2 to minutes; 3 to
+ seconds; and 4 to microseconds.
+
+- 'any' values may be timezone-aware datetimes, timezone-naive datetimes,
+ or dates. dates are converted to any value from the start to the end of the
+ given date in the found timezone, as described above. timezone-naive
+ datetimes get the found timezone.
+
+- 'all' values may be timezone-aware datetimes or timezone-naive datetimes.
+ timezone-naive datetimes get the found timezone.
+
+- 'minimum' values may be timezone-aware datetimes, timezone-naive datetimes,
+ or dates. dates are converted to the start of the given date in the found
+ timezone, as described above. timezone-naive datetimes get the found
+ timezone.
+
+- 'maximum' values may be timezone-aware datetimes, timezone-naive datetimes,
+ or dates. dates are converted to the end of the given date in the found
+ timezone, as described above. timezone-naive datetimes get the found
+ timezone.
+
+Let's look at the DateTime normalizer first, and then an integration of it
+with the normalizing wrapper and the value and set indexes.
+
+The indexed values are parsed with 'value'.
+
+ >>> from zc.catalog.index import DateTimeNormalizer
+ >>> n = DateTimeNormalizer() # defaults to minutes
+ >>> import datetime
+ >>> import pytz
+ >>> naive_datetime = datetime.datetime(2005, 7, 15, 11, 21, 32, 104)
+ >>> date = naive_datetime.date()
+ >>> aware_datetime = naive_datetime.replace(
+ ... tzinfo=pytz.timezone('US/Eastern'))
+ >>> n.value(naive_datetime)
+ Traceback (most recent call last):
+ ...
+ ValueError: This index only indexes timezone-aware datetimes.
+ >>> n.value(date)
+ Traceback (most recent call last):
+ ...
+ ValueError: This index only indexes timezone-aware datetimes.
+ >>> n.value(aware_datetime) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, tzinfo=<DstTzInfo 'US/Eastern'...>)
+
+If we specify a different resolution, the results are different.
+
+ >>> another = DateTimeNormalizer(1) # hours
+ >>> another.value(aware_datetime) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 0, tzinfo=<DstTzInfo 'US/Eastern'...>)
+
+Note that changing the resolution of an indexed value may create surprising
+results, because queries do not change their resolution. Therefore, if you
+index something with a datetime with a finer resolution that the normalizer's,
+then searching for that datetime will not find the doc_id.
+
+Values in an 'any_of' query are parsed with 'any'. 'any' should return a
+sequence of values. It requires an index, which we will mock up here.
+
+ >>> class DummyIndex(object):
+ ... def values(self, start, stop, exclude_start, exclude_stop):
+ ... assert not exclude_start and exclude_stop
+ ... six_hours = datetime.timedelta(hours=6)
+ ... res = []
+ ... dt = start
+ ... while dt < stop:
+ ... res.append(dt)
+ ... dt += six_hours
+ ... return res
+ ...
+ >>> index = DummyIndex()
+ >>> tuple(n.any(naive_datetime, index)) # doctest: +ELLIPSIS
+ (datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Local...>),)
+ >>> tuple(n.any(aware_datetime, index)) # doctest: +ELLIPSIS
+ (datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Eastern...>),)
+ >>> tuple(n.any(date, index)) # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
+ (datetime.datetime(2005, 7, 15, 0, 0, tzinfo=<...Local...>),
+ datetime.datetime(2005, 7, 15, 6, 0, tzinfo=<...Local...>),
+ datetime.datetime(2005, 7, 15, 12, 0, tzinfo=<...Local...>),
+ datetime.datetime(2005, 7, 15, 18, 0, tzinfo=<...Local...>))
+
+Values in an 'all_of' query are parsed with 'all'.
+
+ >>> n.all(naive_datetime, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Local...>)
+ >>> n.all(aware_datetime, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Eastern...>)
+ >>> n.all(date, index) # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError: ...
+
+Minimum values in a 'between' query as well as those in other methods are
+parsed with 'minimum'. They also take an optional exclude boolean, which
+indicates whether the minimum is to be excluded. For datetimes, it only
+makes a difference if you pass in a date.
+
+ >>> n.minimum(naive_datetime, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Local...>)
+ >>> n.minimum(naive_datetime, index, exclude=True) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Local...>)
+
+ >>> n.minimum(aware_datetime, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Eastern...>)
+ >>> n.minimum(aware_datetime, index, True) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Eastern...>)
+
+ >>> n.minimum(date, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 0, 0, tzinfo=<...Local...>)
+ >>> n.minimum(date, index, True) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 23, 59, 59, 999999, tzinfo=<...Local...>)
+
+Maximum values in a 'between' query as well as those in other methods are
+parsed with 'maximum'. They also take an optional exclude boolean, which
+indicates whether the maximum is to be excluded. For datetimes, it only
+makes a difference if you pass in a date.
+
+ >>> n.maximum(naive_datetime, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Local...>)
+ >>> n.maximum(naive_datetime, index, exclude=True) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Local...>)
+
+ >>> n.maximum(aware_datetime, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Eastern...>)
+ >>> n.maximum(aware_datetime, index, True) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, 32, 104, tzinfo=<...Eastern...>)
+
+ >>> n.maximum(date, index) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 23, 59, 59, 999999, tzinfo=<...Local...>)
+ >>> n.maximum(date, index, True) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 0, 0, tzinfo=<...Local...>)
+
+Now let's examine these normalizers in the context of a real index.
+
+ >>> from zc.catalog.index import DateTimeValueIndex, DateTimeSetIndex
+ >>> setindex = DateTimeSetIndex() # minutes resolution
+ >>> data = [] # generate some data
+ >>> def date_gen(
+ ... start=aware_datetime,
+ ... count=12,
+ ... period=datetime.timedelta(hours=10)):
+ ... dt = start
+ ... ix = 0
+ ... while ix < count:
+ ... yield dt
+ ... dt += period
+ ... ix += 1
+ ...
+ >>> gen = date_gen()
+ >>> count = 0
+ >>> while True:
+ ... try:
+ ... next = [gen.next() for i in range(6)]
+ ... except StopIteration:
+ ... break
+ ... data.append((count, next[0:1]))
+ ... count += 1
+ ... data.append((count, next[1:3]))
+ ... count += 1
+ ... data.append((count, next[3:6]))
+ ... count += 1
+ ...
+ >>> print data # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ [(0,
+ [datetime.datetime(2005, 7, 15, 11, 21, 32, 104, ...<...Eastern...>)]),
+ (1,
+ [datetime.datetime(2005, 7, 15, 21, 21, 32, 104, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 16, 7, 21, 32, 104, ...<...Eastern...>)]),
+ (2,
+ [datetime.datetime(2005, 7, 16, 17, 21, 32, 104, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 17, 3, 21, 32, 104, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 17, 13, 21, 32, 104, ...<...Eastern...>)]),
+ (3,
+ [datetime.datetime(2005, 7, 17, 23, 21, 32, 104, ...<...Eastern...>)]),
+ (4,
+ [datetime.datetime(2005, 7, 18, 9, 21, 32, 104, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 18, 19, 21, 32, 104, ...<...Eastern...>)]),
+ (5,
+ [datetime.datetime(2005, 7, 19, 5, 21, 32, 104, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 19, 15, 21, 32, 104, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 20, 1, 21, 32, 104, ...<...Eastern...>)])]
+ >>> data_dict = dict(data)
+ >>> for doc_id, value in data:
+ ... setindex.index_doc(doc_id, value)
+ ...
+ >>> list(setindex.ids())
+ [0, 1, 2, 3, 4, 5]
+ >>> set(setindex.values()) == set(
+ ... setindex.normalizer.value(v) for v in date_gen())
+ True
+
+For the searches, we will actually use a request and interaction, with an
+adapter that returns the Eastern timezone. This makes the examples less
+dependent on the machine that they use.
+
+ >>> import zope.security.management
+ >>> import zope.publisher.browser
+ >>> import zope.interface.common.idatetime
+ >>> import zope.publisher.interfaces
+ >>> request = zope.publisher.browser.TestRequest()
+ >>> zope.security.management.newInteraction(request)
+ >>> from zope import interface, component
+ >>> @interface.implementer(zope.interface.common.idatetime.ITZInfo)
+ ... @component.adapter(zope.publisher.interfaces.IRequest)
+ ... def tzinfo(req):
+ ... return pytz.timezone('US/Eastern')
+ ...
+ >>> component.provideAdapter(tzinfo)
+ >>> n.all(naive_datetime, index).tzinfo is pytz.timezone('US/Eastern')
+ True
+
+ >>> set(setindex.apply({'any_of': (datetime.date(2005, 7, 17),
+ ... datetime.date(2005, 7, 20),
+ ... datetime.date(2005, 12, 31))})) == set(
+ ... (2, 3, 5))
+ True
+
+Note that this search is using the normalized values.
+
+ >>> set(setindex.apply({'all_of': (
+ ... datetime.datetime(
+ ... 2005, 7, 16, 7, 21, tzinfo=pytz.timezone('US/Eastern')),
+ ... datetime.datetime(
+ ... 2005, 7, 15, 21, 21, tzinfo=pytz.timezone('US/Eastern')),)})
+ ... ) == set((1,))
+ True
+ >>> list(setindex.apply({'any': None}))
+ [0, 1, 2, 3, 4, 5]
+ >>> set(setindex.apply({'between': (
+ ... datetime.datetime(2005, 4, 1, 12), datetime.datetime(2006, 5, 1))})
+ ... ) == set((0, 1, 2, 3, 4, 5))
+ True
+ >>> set(setindex.apply({'between': (
+ ... datetime.datetime(2005, 4, 1, 12), datetime.datetime(2006, 5, 1),
+ ... True, True)})
+ ... ) == set((0, 1, 2, 3, 4, 5))
+ True
+
+'between' searches should deal with dates well.
+
+ >>> set(setindex.apply({'between': (
+ ... datetime.date(2005, 7, 16), datetime.date(2005, 7, 17))})
+ ... ) == set((1, 2, 3))
+ True
+ >>> len(setindex.apply({'between': (
+ ... datetime.date(2005, 7, 16), datetime.date(2005, 7, 17))})
+ ... ) == len(setindex.apply({'between': (
+ ... datetime.date(2005, 7, 15), datetime.date(2005, 7, 18),
+ ... True, True)})
+ ... )
+ True
+
+Removing docs works as usual.
+
+ >>> setindex.unindex_doc(1)
+ >>> list(setindex.ids())
+ [0, 2, 3, 4, 5]
+
+Value, Minvalue and Maxvalue can take timezone-less datetimes and dates.
+
+ >>> setindex.minValue() # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 15, 11, 21, ...<...Eastern...>)
+ >>> setindex.minValue(datetime.date(2005, 7, 17)) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 17, 3, 21, ...<...Eastern...>)
+
+ >>> setindex.maxValue() # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 20, 1, 21, ...<...Eastern...>)
+ >>> setindex.maxValue(datetime.date(2005, 7, 17)) # doctest: +ELLIPSIS
+ datetime.datetime(2005, 7, 17, 23, 21, ...<...Eastern...>)
+
+ >>> list(setindex.values(
+ ... datetime.date(2005, 7, 17), datetime.date(2005, 7, 17)))
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ [datetime.datetime(2005, 7, 17, 3, 21, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 17, 13, 21, ...<...Eastern...>),
+ datetime.datetime(2005, 7, 17, 23, 21, ...<...Eastern...>)]
+
+ >>> zope.security.management.endInteraction() # TODO put in tests tearDown
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/normalizedindex.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/setindex.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/setindex.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/setindex.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,231 @@
+The setindex is an index similar to, but more general than a traditional
+keyword index. The values indexed are expected to be iterables; the index
+allows searches for documents that contain any of a set of values; all of a set
+of values; or between a set of values.
+
+Additionally, the index supports an interface that allows examination of the
+indexed values.
+
+It is as policy-free as possible, and is intended to be the engine for indexes
+with more policy, as well as being useful itself.
+
+On creation, the index has no wordCount, no documentCount, and is, as
+expected, fairly empty.
+
+ >>> from zc.catalog.index import SetIndex
+ >>> index = SetIndex()
+ >>> index.documentCount()
+ 0
+ >>> index.wordCount()
+ 0
+ >>> index.maxValue() # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> index.minValue() # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> list(index.values())
+ []
+ >>> len(index.apply({'any_of': (5,)}))
+ 0
+
+The index supports indexing any value. All values within a given index must
+sort consistently across Python versions. In our example, we hope that strings
+and integers will sort consistently; this may not be a reasonable hope.
+
+ >>> data = {1: ['a', 1],
+ ... 2: ['b', 'a', 3, 4, 7],
+ ... 3: [1],
+ ... 4: [1, 4, 'c'],
+ ... 5: [7],
+ ... 6: [5, 6, 7],
+ ... 7: ['c'],
+ ... 8: [1, 6],
+ ... 9: ['a', 'c', 2, 3, 4, 6,],
+ ... }
+ >>> for k, v in data.items():
+ ... index.index_doc(k, v)
+ ...
+
+After indexing, the statistics and values match the newly entered content.
+
+ >>> list(index.values())
+ [1, 2, 3, 4, 5, 6, 7, 'a', 'b', 'c']
+ >>> index.documentCount()
+ 9
+ >>> index.wordCount()
+ 10
+ >>> index.maxValue()
+ 'c'
+ >>> index.minValue()
+ 1
+ >>> list(index.ids())
+ [1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+The index supports five types of query. The first is 'any_of'. It
+takes an iterable of values, and returns an iterable of document ids that
+contain any of the values. The results are weighted.
+
+ >>> list(index.apply({'any_of':('b', 1, 5)}))
+ [1, 2, 3, 4, 6, 8]
+ >>> list(index.apply({'any_of': ('b', 1, 5)}))
+ [1, 2, 3, 4, 6, 8]
+ >>> list(index.apply({'any_of':(42,)}))
+ []
+ >>> index.apply({'any_of': ('a', 3, 7)})
+ BTrees._IFBTree.IFBucket([(1, 1.0), (2, 3.0), (5, 1.0), (6, 1.0), (9, 2.0)])
+
+Another query is 'any'. If the key is None, all indexed document ids with any
+values are returned. If the key is an extent, the intersection of the extent
+and all document ids with any values is returned.
+
+ >>> list(index.apply({'any': None}))
+ [1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+ >>> from zc.catalog.extentcatalog import FilterExtent
+ >>> extent = FilterExtent(lambda extent, uid, obj: True)
+ >>> for i in range(15):
+ ... extent.add(i, i)
+ ...
+ >>> list(index.apply({'any': extent}))
+ [1, 2, 3, 4, 5, 6, 7, 8, 9]
+ >>> limited_extent = FilterExtent(lambda extent, uid, obj: True)
+ >>> for i in range(5):
+ ... limited_extent.add(i, i)
+ ...
+ >>> list(index.apply({'any': limited_extent}))
+ [1, 2, 3, 4]
+
+The 'contains_all' argument also takes an iterable of values, but returns an
+iterable of document ids that contains all of the values. The results are not
+weighted.
+
+ >>> list(index.apply({'all_of': ('a',)}))
+ [1, 2, 9]
+ >>> list(index.apply({'all_of': (3, 4)}))
+ [2, 9]
+
+The 'between' argument takes from 1 to four values. The first is the
+minimum, and defaults to None, indicating no minimum; the second is the
+maximum, and defaults to None, indicating no maximum; the next is a boolean for
+whether the minimum value should be excluded, and defaults to False; and the
+last is a boolean for whether the maximum value should be excluded, and also
+defaults to False. The results are weighted.
+
+ >>> list(index.apply({'between': (1, 7)}))
+ [1, 2, 3, 4, 5, 6, 8, 9]
+ >>> list(index.apply({'between': ('b', None)}))
+ [2, 4, 7, 9]
+ >>> list(index.apply({'between': ('b',)}))
+ [2, 4, 7, 9]
+ >>> list(index.apply({'between': (1, 7, True, True)}))
+ [2, 4, 6, 8, 9]
+ >>> index.apply({'between': (2, 6)})
+ BTrees._IFBTree.IFBucket([(2, 2.0), (4, 1.0), (6, 2.0), (8, 1.0), (9, 4.0)])
+
+The 'none' argument takes an extent and returns the ids in the extent
+that are not indexed; it is intended to be used to return docids that have
+no (or empty) values.
+
+ >>> list(index.apply({'none': extent}))
+ [0, 10, 11, 12, 13, 14]
+
+Trying to use more than one of these at a time generates an error.
+
+ >>> index.apply({'all_of': (5,), 'any_of': (3,)})
+ ... # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+
+Using none of them simply returns None.
+
+ >>> index.apply({}) # returns None
+
+Invalid query names cause ValueErrors.
+
+ >>> index.apply({'foo':()})
+ ... # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+
+When you unindex a document, the searches and statistics should be updated.
+
+ >>> index.unindex_doc(6)
+ >>> len(index.apply({'any_of': (5,)}))
+ 0
+ >>> index.documentCount()
+ 8
+ >>> index.wordCount()
+ 9
+ >>> list(index.values())
+ [1, 2, 3, 4, 6, 7, 'a', 'b', 'c']
+ >>> list(index.ids())
+ [1, 2, 3, 4, 5, 7, 8, 9]
+
+Reindexing a document that has new additional values also is reflected in
+subsequent searches and statistic checks.
+
+ >>> data[8].extend([5, 'c'])
+ >>> index.index_doc(8, data[8])
+ >>> index.documentCount()
+ 8
+ >>> index.wordCount()
+ 10
+ >>> list(index.apply({'any_of': (5,)}))
+ [8]
+ >>> list(index.apply({'any_of': ('c',)}))
+ [4, 7, 8, 9]
+
+The same is true for reindexing a document with both additions and removals.
+
+ >>> 2 in set(index.apply({'any_of': (7,)}))
+ True
+ >>> 2 in set(index.apply({'any_of': (2,)}))
+ False
+ >>> data[2].pop()
+ 7
+ >>> data[2].append(2)
+ >>> index.index_doc(2, data[2])
+ >>> 2 in set(index.apply({'any_of': (7,)}))
+ False
+ >>> 2 in set(index.apply({'any_of': (2,)}))
+ True
+
+Reindexing a document that no longer has any values causes it to be removed
+from the statistics.
+
+ >>> del data[2][:]
+ >>> index.index_doc(2, data[2])
+ >>> index.documentCount()
+ 7
+ >>> index.wordCount()
+ 9
+ >>> list(index.ids())
+ [1, 3, 4, 5, 7, 8, 9]
+
+This affects both ways of determining the ids that are and are not in the index
+(that do and do not have values).
+
+ >>> list(index.apply({'any': None}))
+ [1, 3, 4, 5, 7, 8, 9]
+ >>> list(index.apply({'none': extent}))
+ [0, 2, 6, 10, 11, 12, 13, 14]
+
+The values method can be used to examine the indexed values for a given
+document id.
+
+ >>> set(index.values(doc_id=8)) == set([1, 5, 6, 'c'])
+ True
+
+And the containsValue method provides a way of determining membership in the
+values.
+
+ >>> index.containsValue(5)
+ True
+ >>> index.containsValue(20)
+ False
+
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/setindex.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,73 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""A stemmer based on the textindexng stemmer, itself based on snowball.
+
+$Id: stemmer.py 2918 2005-07-19 22:12:38Z jim $
+"""
+import re
+
+try:
+ import txngstemmer
+except ImportError:
+ txngstemmer = None
+ class Broken:
+ def stem(self, l):
+ return l
+ broken = Broken()
+
+# as of this writing, trying to persist a txngstemmer.Stemmer makes the python
+# process end, only printing a "Bus error" message before quitting. Don't do
+# that. July 16 2005
+
+class Stemmer(object):
+
+ def __init__(self, language='english'):
+ self.language = language
+
+ @property
+ def stemmer(self):
+ if txngstemmer is None:
+ return broken
+ return txngstemmer.Stemmer(self.language)
+
+ rxGlob = re.compile(r"[*?]") # See globToWordIds() in
+ # zope/index/text/lexicon.py
+
+ def process(self, lst):
+ stemmer = self.stemmer
+ result = []
+ for s in lst:
+ try:
+ s = unicode(s)
+ except UnicodeDecodeError:
+ pass
+ else:
+ s = stemmer.stem((s,))[0]
+ result.append(s)
+ return result
+
+ def processGlob(self, lst):
+ stemmer = self.stemmer
+ result = []
+ rxGlob = self.rxGlob
+ for s in lst:
+ if not rxGlob.search(s):
+ try:
+ s = unicode(s)
+ except UnicodeDecodeError:
+ pass
+ else:
+ s = stemmer.stem((s,))[0]
+ result.append(s)
+ return result
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,40 @@
+The stemmer uses Andreas Jung's stemmer code, which is a Python wrapper of
+M. F. Porter's Snowball project (http://snowball.tartarus.org/index.php).
+It is designed to be used as part of a pipeline in a zope/index/text/
+lexicon, after a splitter. This enables getting the relevance ranking
+of the zope/index/text code with the splitting functonality of TextIndexNG 3.x.
+
+It requires that the TextIndexNG extensions--specifically txngstemmer--have
+been compiled and installed in your Python installation. Inclusion of the
+textindexng package is not necessary.
+
+The stemmer must be instantiated with the language for which stemming is
+desired. It defaults to 'english'. For what it is worth, other languages
+supported as of this writing, using the strings that the stemmer expects,
+include the following: 'danish', 'dutch', 'english', 'finnish', 'french',
+'german', 'italian', 'norwegian', 'portuguese', 'russian', 'spanish', and
+'swedish'.
+
+For instance, let's build an index with an english stemmer.
+
+ >>> from zope.index.text import textindex, lexicon
+ >>> import zc.catalog.stemmer
+ >>> lex = lexicon.Lexicon(
+ ... lexicon.Splitter(), lexicon.CaseNormalizer(),
+ ... lexicon.StopWordRemover(), zc.catalog.stemmer.Stemmer('english'))
+ >>> ix = textindex.TextIndex(lex)
+ >>> data = [
+ ... (0, 'consigned consistency consoles the constables'),
+ ... (1, 'knaves kneeled and knocked knees, knowing no knights')]
+ >>> for doc_id, text in data:
+ ... ix.index_doc(doc_id, text)
+ ...
+ >>> list(ix.apply('consoling a constable'))
+ [0]
+ >>> list(ix.apply('knightly kneel'))
+ [1]
+
+Note that query terms with globbing characters are not stemmed.
+
+ >>> list(ix.apply('constables*'))
+ []
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/stemmer.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/tests.py
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/tests.py (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/tests.py 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,33 @@
+#############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""catalog package test runner
+
+$Id: tests.py 2918 2005-07-19 22:12:38Z jim $
+"""
+
+import unittest
+from zope.testing import doctest
+
+def test_suite():
+ return unittest.TestSuite((
+ doctest.DocFileSuite('extentcatalog.txt'),
+ doctest.DocFileSuite('setindex.txt'),
+ doctest.DocFileSuite('valueindex.txt'),
+ doctest.DocFileSuite('normalizedindex.txt'),
+ doctest.DocFileSuite('globber.txt'),
+ doctest.DocFileSuite('stemmer.txt'),
+ ))
+
+if __name__ == '__main__':
+ unittest.main(defaultTest='test_suite')
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/tests.py
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc/catalog/valueindex.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc/catalog/valueindex.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc/catalog/valueindex.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,219 @@
+The valueindex is an index similar to, but more flexible than a standard Zope
+field index. The index allows searches for documents that contain any of a
+set of values; between a set of values; any (non-None) values; and any empty
+values.
+
+Additionally, the index supports an interface that allows examination of the
+indexed values.
+
+It is as policy-free as possible, and is intended to be the engine for indexes
+with more policy, as well as being useful itself.
+
+On creation, the index has no wordCount, no documentCount, and is, as
+expected, fairly empty.
+
+ >>> from zc.catalog.index import ValueIndex
+ >>> index = ValueIndex()
+ >>> index.documentCount()
+ 0
+ >>> index.wordCount()
+ 0
+ >>> index.maxValue() # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> index.minValue() # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> list(index.values())
+ []
+ >>> len(index.apply({'any_of': (5,)}))
+ 0
+
+The index supports indexing any value. All values within a given index must
+sort consistently across Python versions.
+
+ >>> data = {1: 'a',
+ ... 2: 'b',
+ ... 3: 'a',
+ ... 4: 'c',
+ ... 5: 'd',
+ ... 6: 'c',
+ ... 7: 'c',
+ ... 8: 'b',
+ ... 9: 'c',
+ ... }
+ >>> for k, v in data.items():
+ ... index.index_doc(k, v)
+ ...
+
+After indexing, the statistics and values match the newly entered content.
+
+ >>> list(index.values())
+ ['a', 'b', 'c', 'd']
+ >>> index.documentCount()
+ 9
+ >>> index.wordCount()
+ 4
+ >>> index.maxValue()
+ 'd'
+ >>> index.minValue()
+ 'a'
+ >>> list(index.ids())
+ [1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+The index supports four types of query. The first is 'any_of'. It
+takes an iterable of values, and returns an iterable of document ids that
+contain any of the values. The results are not weighted.
+
+ >>> list(index.apply({'any_of':('b', 'c')}))
+ [2, 4, 6, 7, 8, 9]
+ >>> list(index.apply({'any_of': ('b',)}))
+ [2, 8]
+ >>> list(index.apply({'any_of': ('d',)}))
+ [5]
+ >>> list(index.apply({'any_of':(42,)}))
+ []
+
+Another query is 'qny', If the key is None, all indexed document ids with any
+values are returned. If the key is an extent, the intersection of the extent
+and all document ids with any values is returned.
+
+ >>> list(index.apply({'any': None}))
+ [1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+ >>> from zc.catalog.extentcatalog import FilterExtent
+ >>> extent = FilterExtent(lambda extent, uid, obj: True)
+ >>> for i in range(15):
+ ... extent.add(i, i)
+ ...
+ >>> list(index.apply({'any': extent}))
+ [1, 2, 3, 4, 5, 6, 7, 8, 9]
+ >>> limited_extent = FilterExtent(lambda extent, uid, obj: True)
+ >>> for i in range(5):
+ ... limited_extent.add(i, i)
+ ...
+ >>> list(index.apply({'any': limited_extent}))
+ [1, 2, 3, 4]
+
+The 'between' argument takes from 1 to four values. The first is the
+minimum, and defaults to None, indicating no minimum; the second is the
+maximum, and defaults to None, indicating no maximum; the next is a boolean for
+whether the minimum value should be excluded, and defaults to False; and the
+last is a boolean for whether the maximum value should be excluded, and also
+defaults to False. The results are not weighted.
+
+ >>> list(index.apply({'between': ('b', 'd')}))
+ [2, 4, 5, 6, 7, 8, 9]
+ >>> list(index.apply({'between': ('c', None)}))
+ [4, 5, 6, 7, 9]
+ >>> list(index.apply({'between': ('c',)}))
+ [4, 5, 6, 7, 9]
+ >>> list(index.apply({'between': ('b', 'd', True, True)}))
+ [4, 6, 7, 9]
+
+The 'none' argument takes an extent and returns the ids in the extent
+that are not indexed; it is intended to be used to return docids that have
+no (or empty) values.
+
+ >>> list(index.apply({'none': extent}))
+ [0, 10, 11, 12, 13, 14]
+
+Trying to use more than one of these at a time generates an error.
+
+ >>> index.apply({'between': (5,), 'any_of': (3,)})
+ ... # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+
+Using none of them simply returns None.
+
+ >>> index.apply({}) # returns None
+
+Invalid query names cause ValueErrors.
+
+ >>> index.apply({'foo':()})
+ ... # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+
+When you unindex a document, the searches and statistics should be updated.
+
+ >>> index.unindex_doc(5)
+ >>> len(index.apply({'any_of': ('d',)}))
+ 0
+ >>> index.documentCount()
+ 8
+ >>> index.wordCount()
+ 3
+ >>> list(index.values())
+ ['a', 'b', 'c']
+ >>> list(index.ids())
+ [1, 2, 3, 4, 6, 7, 8, 9]
+
+Reindexing a document that has a changed value also is reflected in
+subsequent searches and statistic checks.
+
+ >>> list(index.apply({'any_of': ('b',)}))
+ [2, 8]
+ >>> data[8] = 'e'
+ >>> index.index_doc(8, data[8])
+ >>> index.documentCount()
+ 8
+ >>> index.wordCount()
+ 4
+ >>> list(index.apply({'any_of': ('e',)}))
+ [8]
+ >>> list(index.apply({'any_of': ('b',)}))
+ [2]
+ >>> data[2] = 'e'
+ >>> index.index_doc(2, data[2])
+ >>> index.documentCount()
+ 8
+ >>> index.wordCount()
+ 3
+ >>> list(index.apply({'any_of': ('e',)}))
+ [2, 8]
+ >>> list(index.apply({'any_of': ('b',)}))
+ []
+
+Reindexing a document for which the value is now None causes it to be removed
+from the statistics.
+
+ >>> data[3] = None
+ >>> index.index_doc(3, data[3])
+ >>> index.documentCount()
+ 7
+ >>> index.wordCount()
+ 3
+ >>> list(index.ids())
+ [1, 2, 4, 6, 7, 8, 9]
+
+This affects both ways of determining the ids that are and are not in the index
+(that do and do not have values).
+
+ >>> list(index.apply({'any': None}))
+ [1, 2, 4, 6, 7, 8, 9]
+ >>> list(index.apply({'any': extent}))
+ [1, 2, 4, 6, 7, 8, 9]
+ >>> list(index.apply({'none': extent}))
+ [0, 3, 5, 10, 11, 12, 13, 14]
+
+The values method can be used to examine the indexed values for a given
+document id. For a valueindex, the "values" for a given doc_id will always
+have a length of 0 or 1.
+
+ >>> index.values(doc_id=8)
+ ('e',)
+
+And the containsValue method provides a way of determining membership in the
+values.
+
+ >>> index.containsValue('a')
+ True
+ >>> index.containsValue('q')
+ False
+
Property changes on: python-zc.catalog/branches/upstream/current/src/zc/catalog/valueindex.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/PKG-INFO
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/PKG-INFO (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/PKG-INFO 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,14 @@
+Metadata-Version: 1.0
+Name: zc.catalog
+Version: 0.1.1
+Summary: zc.catalog contains a number of extensions to the Zope 3 catalog,
+such as some new indexes, improved globbing and stemming support,
+and an alternative catalog implementation.
+
+Home-page: UNKNOWN
+Author: Zope Project
+Author-email: zope3-dev at zope.org
+License: ZPL
+Description: UNKNOWN
+Keywords: zope zope3
+Platform: UNKNOWN
Added: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/SOURCES.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/SOURCES.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/SOURCES.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,26 @@
+Makefile
+ZopePublicLicense.txt
+setup.py
+zc.catalog-configure.zcml
+src/zc/__init__.py
+src/zc.catalog.egg-info/PKG-INFO
+src/zc.catalog.egg-info/SOURCES.txt
+src/zc.catalog.egg-info/namespace_packages.txt
+src/zc.catalog.egg-info/not-zip-safe
+src/zc.catalog.egg-info/top_level.txt
+src/zc/catalog/__init__.py
+src/zc/catalog/catalogindex.py
+src/zc/catalog/configure.zcml
+src/zc/catalog/extentcatalog.py
+src/zc/catalog/extentcatalog.txt
+src/zc/catalog/globber.py
+src/zc/catalog/globber.txt
+src/zc/catalog/i18n.py
+src/zc/catalog/index.py
+src/zc/catalog/interfaces.py
+src/zc/catalog/normalizedindex.txt
+src/zc/catalog/setindex.txt
+src/zc/catalog/stemmer.py
+src/zc/catalog/stemmer.txt
+src/zc/catalog/tests.py
+src/zc/catalog/valueindex.txt
Property changes on: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/SOURCES.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/namespace_packages.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/namespace_packages.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/namespace_packages.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1 @@
+zc
Property changes on: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/namespace_packages.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/not-zip-safe
===================================================================
Added: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/top_level.txt
===================================================================
--- python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/top_level.txt (rev 0)
+++ python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/top_level.txt 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1 @@
+zc
Property changes on: python-zc.catalog/branches/upstream/current/src/zc.catalog.egg-info/top_level.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: python-zc.catalog/branches/upstream/current/zc.catalog-configure.zcml
===================================================================
--- python-zc.catalog/branches/upstream/current/zc.catalog-configure.zcml (rev 0)
+++ python-zc.catalog/branches/upstream/current/zc.catalog-configure.zcml 2006-10-25 09:18:24 UTC (rev 331)
@@ -0,0 +1,3 @@
+<!-- install this into your Zope 3 instance's etc/package-includes
+ directory -->
+<include package="zc.catalog"/>
\ No newline at end of file
Property changes on: python-zc.catalog/branches/upstream/current/zc.catalog-configure.zcml
___________________________________________________________________
Name: svn:eol-style
+ native
More information about the pkg-zope-commits
mailing list