[Pkg-bazaar-commits] ./loggerhead/unstable r232: New upstream snapshot.
Jelmer Vernooij
jelmer at samba.org
Tue May 5 16:22:36 UTC 2009
------------------------------------------------------------
revno: 232
committer: Jelmer Vernooij <jelmer at samba.org>
branch nick: unstable
timestamp: Tue 2009-05-05 18:21:49 +0200
message:
New upstream snapshot.
renamed:
loggerhead/apps/filesystem.py => loggerhead/apps/transport.py
modified:
NEWS
__init__.py
debian/changelog
loggerhead/apps/branch.py
loggerhead/changecache.py
loggerhead/config.py
loggerhead/controllers/directory_ui.py
loggerhead/history.py
loggerhead/middleware/profile.py
loggerhead/search.py
loggerhead/wholehistory.py
serve-branches
loggerhead/apps/transport.py
------------------------------------------------------------
revno: 182.83.1
committer: Jelmer Vernooij <jelmer at samba.org>
branch nick: trunk
timestamp: Thu 2009-04-09 18:56:39 +0200
message:
Mention bug # for branches served through http in NEWS.
modified:
NEWS
------------------------------------------------------------
revno: 182.48.72
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Thu 2009-04-09 17:19:11 +0000
message:
Mention bug # for branches served through http in NEWS. (Jelmer Vernooij)
modified:
NEWS
------------------------------------------------------------
revno: 182.48.73
committer: Martin Albisetti <martin.albisetti at canonical.com>
branch nick: trunk
timestamp: Mon 2009-04-13 21:58:20 -0300
message:
Fix certain race conditions for loading bzr-search (Robert Collins)
modified:
loggerhead/search.py
------------------------------------------------------------
revno: 182.84.1
committer: Robert Collins <robertc at robertcollins.net>
branch nick: trunk
timestamp: Thu 2009-04-09 19:33:40 +1000
message:
Fix search when bzr-search is only in ~/.bazaar/plugins, by dealing with the race condition with load_plugins being called, and the default search path having changed in 1.12 or so.
modified:
loggerhead/search.py
------------------------------------------------------------
revno: 182.48.74
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Thu 2009-04-16 21:49:48 +0000
message:
Fix incorrect calls to config.get_option() (bug #361238)
modified:
NEWS
serve-branches
------------------------------------------------------------
revno: 182.48.75
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Thu 2009-04-16 23:47:26 +0000
message:
Trivial PEP 8 tweak
modified:
loggerhead/apps/filesystem.py
------------------------------------------------------------
revno: 182.48.76
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Fri 2009-04-17 01:50:05 +0000
message:
Don't use the "sets" module.
1.) The code wasn't using the built-in "set" type when it was available.
2.) bzrlib isn't compatible with Python <2.4 anyway, so there's no need to fall back to the "sets" module.
modified:
loggerhead/search.py
------------------------------------------------------------
revno: 182.48.77
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: trunk
timestamp: Fri 2009-04-24 11:37:10 +1200
message:
merge Matt's one-config branch, which starts to use a single object for configuration
modified:
loggerhead/apps/filesystem.py
serve-branches
------------------------------------------------------------
revno: 182.85.1
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: one-config
timestamp: Fri 2009-04-17 00:14:16 +0000
message:
Pass serve-branches's config object to BranchesFromFileSystemServer instead of creating new ones.
modified:
loggerhead/apps/filesystem.py
serve-branches
------------------------------------------------------------
revno: 182.48.78
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: trunk
timestamp: Fri 2009-04-24 11:39:18 +1200
message:
fix bug #358322: only create one temporary sql dir per process (Matt Nordhoff)
modified:
loggerhead/config.py
------------------------------------------------------------
revno: 182.86.1
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: 358322-sql-dirs
timestamp: Thu 2009-04-09 09:37:27 +0000
message:
Quick fix: Create one, global SQL_DIR in loggerhead.config
modified:
loggerhead/config.py
------------------------------------------------------------
revno: 182.86.2
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: 358322-sql-dirs
timestamp: Thu 2009-04-16 22:22:25 +0000
message:
Add a --cache-dir option for where to put the SQL cache, defaulting to mkdtemp()
modified:
loggerhead/apps/filesystem.py
loggerhead/config.py
------------------------------------------------------------
revno: 182.86.3
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: 358322-sql-dirs
timestamp: Thu 2009-04-16 22:26:15 +0000
message:
Don't create the temporary SQL dir until it's actually needed.
This is a bit ugly, but not *that* bad. Plus it's closer to the old code.
modified:
loggerhead/apps/filesystem.py
loggerhead/config.py
------------------------------------------------------------
revno: 182.86.4
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: 358322-sql-dirs
timestamp: Thu 2009-04-16 22:28:17 +0000
message:
Whoops. Test, *then* commit! Not the other way around!
modified:
loggerhead/config.py
------------------------------------------------------------
revno: 182.48.79
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: trunk
timestamp: Fri 2009-04-24 12:10:56 +1200
message:
score one for the memory profiling we did at pycon: store the "where merged"
information in small tuples, not small sets.
this makes the 'whole history' cache for a launchpad branch take up about 15%
less RAM.
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.87.1
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: small-tuples-not-small-sets
timestamp: Fri 2009-04-24 12:00:03 +1200
message:
tiny change
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.87.2
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: small-tuples-not-small-sets
timestamp: Fri 2009-04-24 12:00:14 +1200
message:
merge trunk
modified:
NEWS
loggerhead/apps/filesystem.py
loggerhead/config.py
loggerhead/search.py
serve-branches
------------------------------------------------------------
revno: 182.48.80
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: trunk
timestamp: Mon 2009-04-27 15:22:23 +1200
message:
add the ability to cache the wholehistory data to disk as well as RAM.
cache a much smaller amount of wholehistory data in RAM.
modified:
loggerhead/apps/branch.py
loggerhead/apps/filesystem.py
loggerhead/changecache.py
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.88.1
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: less-stupid-whole-history-redundancy
timestamp: Fri 2009-04-24 13:15:31 +1200
message:
eliminate some of the more egregious memory usage.
reduces whole history size by another 14% for launchpad branches
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.88.2
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: less-stupid-whole-history-redundancy
timestamp: Fri 2009-04-24 14:53:25 +1200
message:
it takes < 0.1 second to derive _revno_revid from the _rev_info, so don't cache
it!
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.88.3
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: less-stupid-whole-history-redundancy
timestamp: Fri 2009-04-24 14:59:49 +1200
message:
don't cache _rev_indices either
modified:
loggerhead/history.py
------------------------------------------------------------
revno: 182.88.4
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: less-stupid-whole-history-redundancy
timestamp: Fri 2009-04-24 15:06:06 +1200
message:
probably daft optimization
modified:
loggerhead/history.py
------------------------------------------------------------
revno: 182.88.5
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: less-stupid-whole-history-redundancy
timestamp: Fri 2009-04-24 15:12:28 +1200
message:
don't cache _full_history either, which doesn't make much difference really...
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.88.6
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: less-stupid-whole-history-redundancy
timestamp: Fri 2009-04-24 15:44:30 +1200
message:
oh heh, _full_history wasn't what I thought it was
modified:
loggerhead/history.py
------------------------------------------------------------
revno: 182.89.1
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 16:27:49 +1200
message:
merge less-stupid-whole-history-redundancy
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.89.2
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 16:32:31 +1200
message:
make it possible to persist _rev_info with marshal
modified:
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.89.3
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 17:36:08 +1200
message:
fairly tortuous two level caching
modified:
loggerhead/apps/branch.py
loggerhead/changecache.py
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.89.4
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 17:37:45 +1200
message:
remove semantic-free changes
modified:
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.89.5
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 21:29:13 +1200
message:
refactor in the direction of maybe making sense one day
modified:
loggerhead/apps/branch.py
loggerhead/changecache.py
loggerhead/history.py
------------------------------------------------------------
revno: 182.89.6
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 21:45:31 +1200
message:
zlib compress the marshalled data -- the data seems to compress by about 4:1
modified:
loggerhead/changecache.py
------------------------------------------------------------
revno: 182.89.7
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 22:23:44 +1200
message:
use smaller LRUCaches
modified:
loggerhead/apps/branch.py
loggerhead/apps/filesystem.py
------------------------------------------------------------
revno: 182.89.8
committer: Michael Hudson <michael.hudson at canonical.com>
branch nick: persist-_rev_info
timestamp: Fri 2009-04-24 22:30:15 +1200
message:
docstrings (omg!!)
modified:
loggerhead/changecache.py
loggerhead/history.py
loggerhead/wholehistory.py
------------------------------------------------------------
revno: 182.48.81
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Wed 2009-04-29 23:30:35 +0000
message:
Use Dozer for memory profiling (Paul Hummer)
modified:
loggerhead/middleware/profile.py
serve-branches
------------------------------------------------------------
revno: 182.90.1
committer: Paul Hummer <paul at canonical.com>
branch nick: dozer
timestamp: Wed 2009-04-01 16:41:00 -0600
message:
Memory profiling. Dozer is doing it.
modified:
serve-branches
------------------------------------------------------------
revno: 182.90.2
committer: Paul Hummer <paul at canonical.com>
branch nick: dozer
timestamp: Wed 2009-04-01 16:50:25 -0600
message:
Eated MemoryProfileMiddleware
modified:
loggerhead/middleware/profile.py
------------------------------------------------------------
revno: 182.48.82
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Sat 2009-05-02 02:42:49 +0000
message:
Use the branch's public_branch as the default for served_url (bug #369767)
modified:
loggerhead/apps/branch.py
------------------------------------------------------------
revno: 182.91.1
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: serve_public_branch
timestamp: Thu 2009-04-30 11:15:58 +0000
message:
Use the branch's public_branch as the default for served_url (bug #369767)
modified:
loggerhead/apps/branch.py
------------------------------------------------------------
revno: 182.48.83
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Sat 2009-05-02 14:01:05 +0000
message:
Make sure to close mkstemp's file descriptor (bug #370845)
modified:
loggerhead/changecache.py
------------------------------------------------------------
revno: 182.48.84
committer: Martin Albisetti <martin.albisetti at canonical.com>
branch nick: trunk
timestamp: Mon 2009-05-04 17:12:01 -0300
message:
Use Branch._get_nick(local=True) rather than Branch.get_config().get_nickname() as the latter may not be available in some implementations like svn. (Jelmer Vernooij)
modified:
loggerhead/apps/filesystem.py
------------------------------------------------------------
revno: 182.92.1
committer: Jelmer Vernooij <jelmer at samba.org>
branch nick: trunk
timestamp: Mon 2009-05-04 21:47:36 +0200
message:
Use Branch._get_nick(local=True) rather than Branch.get_config().get_nickname() as the latter may not be available in some implementations (read: svn).
modified:
loggerhead/apps/filesystem.py
------------------------------------------------------------
revno: 182.48.85
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Mon 2009-05-04 20:28:50 +0000
message:
Use transports internally rather than local file system. (Jelmer Vernooij)
renamed:
loggerhead/apps/filesystem.py => loggerhead/apps/transport.py
modified:
NEWS
__init__.py
loggerhead/controllers/directory_ui.py
serve-branches
loggerhead/apps/transport.py
------------------------------------------------------------
revno: 182.93.1
committer: Jelmer Vernooij <jelmer at samba.org>
branch nick: trunk
timestamp: Mon 2009-05-04 20:53:21 +0200
message:
Use transports internally rather than local file system.
modified:
loggerhead/apps/filesystem.py
loggerhead/controllers/directory_ui.py
serve-branches
------------------------------------------------------------
revno: 182.93.2
committer: Jelmer Vernooij <jelmer at samba.org>
branch nick: trunk
timestamp: Mon 2009-05-04 21:13:43 +0200
message:
add NEWS item
modified:
NEWS
------------------------------------------------------------
revno: 182.93.3
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: transport
timestamp: Mon 2009-05-04 20:21:08 +0000
message:
Review:
* Fix hiding directories that start with "." in DirectoryUI.
* Rename loggerhead.apps.filesystem to loggerhead.apps.transport, along with its classes.
* Fix NEWS syntax and mention bug number.
renamed:
loggerhead/apps/filesystem.py => loggerhead/apps/transport.py
modified:
NEWS
__init__.py
loggerhead/controllers/directory_ui.py
serve-branches
loggerhead/apps/transport.py
------------------------------------------------------------
revno: 182.48.86
committer: Matt Nordhoff <mnordhoff at mattnordhoff.com>
branch nick: trunk
timestamp: Mon 2009-05-04 20:29:13 +0000
message:
Fix typo in NEWS
modified:
NEWS
-------------- next part --------------
=== modified file 'NEWS'
--- a/NEWS 2009-04-07 18:35:19 +0000
+++ b/NEWS 2009-05-04 20:29:13 +0000
@@ -5,7 +5,7 @@
---------------
- Loggerhead now serves bzr branches over HTTP and exposes the URL
- to branch them. (Jonathan Lange)
+ to branch them. Addresses bug #240577. (Jonathan Lange)
- Leading blank lines in commit messages no longer result in an
empty summary. (Colin Watson)
@@ -46,6 +46,12 @@
- serve-branches now has an option, --use-cdn, to load YUI from
Yahoo!'s CDN. (Matt Nordhoff)
+ - Fix bug #361238 (serve-branches was calling config.get_option()
+ incorrectly and failing). (Matt Nordhoff)
+
+ - Use transport API internally, so it is possible to specify a remote
+ URL to serve-branches. (Jelmer Vernooij, #371787)
+
1.10 [22Dec2008]
---------------
=== modified file '__init__.py'
--- a/__init__.py 2009-03-10 01:01:04 +0000
+++ b/__init__.py 2009-05-04 20:21:08 +0000
@@ -71,10 +71,10 @@
import os.path, sys
sys.path.append(os.path.dirname(__file__))
- from loggerhead.apps.filesystem import BranchesFromFileSystemRoot
+ from loggerhead.apps.transport import BranchesFromTransportRoot
from paste.httpexceptions import HTTPExceptionHandler
from paste.httpserver import serve
- a = HTTPExceptionHandler(BranchesFromFileSystemRoot('.'))
+ a = HTTPExceptionHandler(BranchesFromTransportRoot('.'))
port = kw.get('port', DEFAULT_PORT)
# port might be an int already...
if isinstance(port, basestring) and ':' in port:
=== modified file 'debian/changelog'
--- a/debian/changelog 2009-04-09 17:08:56 +0000
+++ b/debian/changelog 2009-05-05 16:21:49 +0000
@@ -1,4 +1,4 @@
-loggerhead (1.10+bzr324-2) UNRELEASED; urgency=low
+loggerhead (1.10+bzr339-1) UNRELEASED; urgency=low
[ Roland Mas ]
* Use the YUI library provided by libjs-yui. (Closes: #511286)
@@ -8,8 +8,9 @@
* Add ${misc:Depends} to please lintian.
* Suggest recent version of paste, which doesn't expose internal port
numbers in links. (Closes: #507000)
+ * New upstream snapshot.
- -- Jelmer Vernooij <jelmer at debian.org> Thu, 09 Apr 2009 19:08:49 +0200
+ -- Jelmer Vernooij <jelmer at debian.org> Tue, 05 May 2009 18:21:22 +0200
loggerhead (1.10-1) unstable; urgency=low
=== modified file 'loggerhead/apps/branch.py'
--- a/loggerhead/apps/branch.py 2009-04-07 18:29:10 +0000
+++ b/loggerhead/apps/branch.py 2009-04-30 11:15:58 +0000
@@ -38,27 +38,31 @@
self.branch_link = branch_link # Currently only used in Launchpad
self.log = logging.getLogger('loggerhead.%s' % friendly_name)
if graph_cache is None:
- graph_cache = bzrlib.lru_cache.LRUCache()
+ graph_cache = bzrlib.lru_cache.LRUCache(10)
self.graph_cache = graph_cache
self.is_root = is_root
self.served_url = served_url
self.use_cdn = use_cdn
def get_history(self):
- _history = History(self.branch, self.graph_cache)
+ file_cache = None
+ revinfo_disk_cache = None
cache_path = self._config.get('cachepath', None)
if cache_path is not None:
# Only import the cache if we're going to use it.
# This makes sqlite optional
try:
- from loggerhead.changecache import FileChangeCache
+ from loggerhead.changecache import (
+ FileChangeCache, RevInfoDiskCache)
except ImportError:
self.log.debug("Couldn't load python-sqlite,"
" continuing without using a cache")
else:
- _history.use_file_cache(
- FileChangeCache(_history, cache_path))
- return _history
+ file_cache = FileChangeCache(cache_path)
+ revinfo_disk_cache = RevInfoDiskCache(cache_path)
+ return History(
+ self.branch, self.graph_cache, file_cache=file_cache,
+ revinfo_disk_cache=revinfo_disk_cache, cache_key=self.friendly_name)
def url(self, *args, **kw):
if isinstance(args[0], list):
@@ -115,7 +119,11 @@
self._static_url_base = self._url_base
self._environ = environ
if self.served_url is _DEFAULT:
- self.served_url = self.url([])
+ public_branch = self.branch_url()
+ if public_branch is not None:
+ self.served_url = public_branch
+ else:
+ self.served_url = self.url([])
path = request.path_info_pop(environ)
if not path:
raise httpexceptions.HTTPMovedPermanently(
=== renamed file 'loggerhead/apps/filesystem.py' => 'loggerhead/apps/transport.py'
--- a/loggerhead/apps/filesystem.py 2009-04-07 18:29:10 +0000
+++ b/loggerhead/apps/transport.py 2009-05-04 20:28:50 +0000
@@ -1,8 +1,6 @@
-"""Serve branches at urls that mimic the file system layout."""
-
-import os
-
-from bzrlib import branch, errors, lru_cache
+"""Serve branches at urls that mimic a transport's file system layout."""
+
+from bzrlib import branch, errors, lru_cache, urlutils
from paste.request import path_info_pop
from paste import httpexceptions
@@ -14,17 +12,17 @@
from loggerhead.controllers.directory_ui import DirectoryUI
-class BranchesFromFileSystemServer(object):
+class BranchesFromTransportServer(object):
- def __init__(self, path, root, name=None):
- self.path = path
+ def __init__(self, transport, root, name=None):
+ self.transport = transport
self.root = root
self.name = name
- self._config = LoggerheadConfig()
+ self._config = root._config
def app_for_branch(self, branch):
if not self.name:
- name = branch.get_config().get_nickname()
+ name = branch._get_nick(local=True)
is_root = True
else:
name = self.name
@@ -47,32 +45,33 @@
else:
name = '/'
return DirectoryUI(environ['loggerhead.static.url'],
- self.path,
+ self.transport,
name)
else:
- new_path = os.path.join(self.path, segment)
+ new_transport = self.transport.clone(segment)
if self.name:
- new_name = os.path.join(self.name, segment)
+ new_name = urlutils.join(self.name, segment)
else:
new_name = '/' + segment
- return BranchesFromFileSystemServer(new_path, self.root, new_name)
+ return BranchesFromTransportServer(new_transport, self.root, new_name)
def __call__(self, environ, start_response):
- if not os.path.isdir(self.path):
- raise httpexceptions.HTTPNotFound()
try:
- b = branch.Branch.open(self.path)
+ b = branch.Branch.open_from_transport(self.transport)
except errors.NotBranchError:
+ if not self.transport.listable() or not self.transport.has('.'):
+ raise httpexceptions.HTTPNotFound()
return self.app_for_non_branch(environ)(environ, start_response)
else:
return self.app_for_branch(b)(environ, start_response)
-class BranchesFromFileSystemRoot(object):
+class BranchesFromTransportRoot(object):
- def __init__(self, folder):
- self.graph_cache = lru_cache.LRUCache()
- self.folder = folder
+ def __init__(self, transport, config):
+ self.graph_cache = lru_cache.LRUCache(10)
+ self.transport = transport
+ self._config = config
def __call__(self, environ, start_response):
environ['loggerhead.static.url'] = environ['SCRIPT_NAME']
@@ -83,23 +82,24 @@
elif environ['PATH_INFO'] == '/favicon.ico':
return favicon_app(environ, start_response)
elif '/.bzr/' in environ['PATH_INFO']:
- app = urlparser.make_static(None, self.folder)
+ app = urlparser.make_static(None, self.transport)
return app(environ, start_response)
else:
- return BranchesFromFileSystemServer(
- self.folder, self)(environ, start_response)
-
-
-class UserBranchesFromFileSystemRoot(object):
-
- def __init__(self, folder, trunk_dir):
- self.graph_cache = lru_cache.LRUCache()
- self.folder = folder
- self.trunk_dir = trunk_dir
+ return BranchesFromTransportServer(
+ self.transport, self)(environ, start_response)
+
+
+class UserBranchesFromTransportRoot(object):
+
+ def __init__(self, transport, config):
+ self.graph_cache = lru_cache.LRUCache(10)
+ self.transport = transport
+ self._config = config
+ self.trunk_dir = config.get_option('trunk_dir')
def __call__(self, environ, start_response):
environ['loggerhead.static.url'] = environ['SCRIPT_NAME']
- path_info= environ['PATH_INFO']
+ path_info = environ['PATH_INFO']
if path_info.startswith('/static/'):
segment = path_info_pop(environ)
assert segment == 'static'
@@ -110,10 +110,10 @@
# segments starting with ~ are user branches
if path_info.startswith('/~'):
segment = path_info_pop(environ)
- new_path = os.path.join(self.folder, segment[1:])
- return BranchesFromFileSystemServer(
- new_path, self, segment)(environ, start_response)
+ new_transport = self.transport.clone(segment[1:])
+ return BranchesFromTransportServer(
+ new_transport, self, segment)(environ, start_response)
else:
- new_path = os.path.join(self.folder, self.trunk_dir)
- return BranchesFromFileSystemServer(
- new_path, self)(environ, start_response)
+ new_transport = self.transport.clone(self.trunk_dir)
+ return BranchesFromTransportServer(
+ new_transport, self)(environ, start_response)
=== modified file 'loggerhead/changecache.py'
--- a/loggerhead/changecache.py 2009-03-19 21:01:11 +0000
+++ b/loggerhead/changecache.py 2009-05-02 14:01:05 +0000
@@ -27,8 +27,10 @@
"""
import cPickle
+import marshal
import os
import tempfile
+import zlib
try:
from sqlite3 import dbapi2
@@ -38,16 +40,26 @@
# We take an optimistic approach to concurrency here: we might do work twice
# in the case of races, but not crash or corrupt data.
+def safe_init_db(filename, init_sql):
+ # To avoid races around creating the database, we create the db in
+ # a temporary file and rename it into the ultimate location.
+ fd, temp_path = tempfile.mkstemp(dir=os.path.dirname(filename))
+ os.close(fd)
+ con = dbapi2.connect(temp_path)
+ cur = con.cursor()
+ cur.execute(init_sql)
+ con.commit()
+ con.close()
+ os.rename(temp_path, filename)
+
class FakeShelf(object):
def __init__(self, filename):
create_table = not os.path.exists(filename)
if create_table:
- # To avoid races around creating the database, we create the db in
- # a temporary file and rename it into the ultimate location.
- fd, path = tempfile.mkstemp(dir=os.path.dirname(filename))
- self._create_table(path)
- os.rename(path, filename)
+ safe_init_db(
+ filename, "create table RevisionData "
+ "(revid binary primary key, data binary)")
self.connection = dbapi2.connect(filename)
self.cursor = self.connection.cursor()
@@ -89,8 +101,7 @@
class FileChangeCache(object):
- def __init__(self, history, cache_path):
- self.history = history
+ def __init__(self, cache_path):
if not os.path.exists(cache_path):
os.mkdir(cache_path)
@@ -104,3 +115,44 @@
changes = self.history.get_file_changes_uncached(entry)
cache.add(entry.revid, changes)
return changes
+
+
+class RevInfoDiskCache(object):
+ """Like `RevInfoMemoryCache` but backed in a sqlite DB."""
+
+ def __init__(self, cache_path):
+ if not os.path.exists(cache_path):
+ os.mkdir(cache_path)
+ filename = os.path.join(cache_path, 'revinfo.sql')
+ create_table = not os.path.exists(filename)
+ if create_table:
+ safe_init_db(
+ filename, "create table Data "
+ "(key binary primary key, revid binary, data binary)")
+ self.connection = dbapi2.connect(filename)
+ self.cursor = self.connection.cursor()
+
+ def get(self, key, revid):
+ self.cursor.execute(
+ "select revid, data from data where key = ?", (dbapi2.Binary(key),))
+ row = self.cursor.fetchone()
+ if row is None:
+ return None
+ elif str(row[0]) != revid:
+ return None
+ else:
+ return marshal.loads(zlib.decompress(row[1]))
+
+ def set(self, key, revid, data):
+ try:
+ self.cursor.execute(
+ 'delete from data where key = ?', (dbapi2.Binary(key), ))
+ blob = zlib.compress(marshal.dumps(data))
+ self.cursor.execute(
+ "insert into data (key, revid, data) values (?, ?, ?)",
+ map(dbapi2.Binary, [key, revid, blob]))
+ self.connection.commit()
+ except dbapi2.IntegrityError:
+ # If another thread or process attempted to set the same key, we
+ # don't care too much -- it's only a cache after all!
+ pass
=== modified file 'loggerhead/config.py'
--- a/loggerhead/config.py 2009-04-07 19:00:26 +0000
+++ b/loggerhead/config.py 2009-04-16 22:28:17 +0000
@@ -4,6 +4,14 @@
import tempfile
+_temporary_sql_dir = None
+
+def _get_temporary_sql_dir():
+ global _temporary_sql_dir
+ if _temporary_sql_dir is None:
+ _temporary_sql_dir = tempfile.mkdtemp(prefix='loggerhead-cache-')
+ return _temporary_sql_dir
+
def command_line_parser():
parser = OptionParser("%prog [options] <path>")
parser.set_defaults(
@@ -11,6 +19,7 @@
show_version=False,
log_folder=None,
use_cdn=False,
+ sql_dir=None,
)
parser.add_option("--user-dirs", action="store_true", dest="user_dirs",
help="Serve user directories as ~user.")
@@ -38,6 +47,8 @@
help="Print the software version and exit")
parser.add_option('--use-cdn', action='store_true',
help="Serve YUI from Yahoo!'s CDN")
+ parser.add_option('--cache-dir', dest='sql_dir',
+ help="The directory to place the SQL cache in")
return parser
@@ -48,7 +59,10 @@
self._parser = command_line_parser()
self._options, self._args = self._parser.parse_args(sys.argv[1:])
- self.SQL_DIR = tempfile.mkdtemp(prefix='loggerhead-cache-')
+ sql_dir = self.get_option('sql_dir')
+ if sql_dir is None:
+ sql_dir = _get_temporary_sql_dir()
+ self.SQL_DIR = sql_dir
def get_option(self, option):
'''Get an option from the options dict.'''
=== modified file 'loggerhead/controllers/directory_ui.py'
--- a/loggerhead/controllers/directory_ui.py 2008-12-05 18:52:44 +0000
+++ b/loggerhead/controllers/directory_ui.py 2009-05-04 20:21:08 +0000
@@ -18,7 +18,7 @@
import datetime
import logging
-import os
+import stat
from bzrlib import branch
@@ -48,7 +48,7 @@
template_path = 'loggerhead.templates.directory'
- def __init__(self, static_url_base, path, name):
+ def __init__(self, static_url_base, transport, name):
class _branch(object):
context_url = 1
@@ -58,22 +58,21 @@
return self._static_url_base + path
self._branch = _branch
self._history_callable = lambda:None
- self._path = path
self._name = name
self._static_url_base = static_url_base
+ self.transport = transport
self.log = logging.getLogger('')
def get_values(self, path, kwargs, response):
- listing = [d for d in os.listdir(self._path)
+ listing = [d for d in self.transport.list_dir('.')
if not d.startswith('.')
- and os.path.isdir(os.path.join(self._path, d))]
+ and stat.S_ISDIR(self.transport.stat(d).st_mode)]
listing.sort(key=lambda x: x.lower())
dirs = []
parity = 0
for d in listing:
- p = os.path.join(self._path, d)
try:
- b = branch.Branch.open(p)
+ b = branch.Branch.open_from_transport(self.transport.clone(d))
except:
b = None
dirs.append(DirEntry(d, parity, b))
=== modified file 'loggerhead/history.py'
--- a/loggerhead/history.py 2009-03-19 20:04:24 +0000
+++ b/loggerhead/history.py 2009-04-24 10:30:15 +0000
@@ -31,6 +31,7 @@
import bisect
import datetime
import logging
+import marshal
import re
import textwrap
import threading
@@ -47,6 +48,7 @@
import bzrlib.delta
import bzrlib.diff
import bzrlib.errors
+import bzrlib.lru_cache
import bzrlib.progress
import bzrlib.revision
import bzrlib.textfile
@@ -177,6 +179,43 @@
file_id=file_id))
+class RevInfoMemoryCache(object):
+ """A store that validates values against the revids they were stored with.
+
+ We use a unique key for each branch.
+
+ The reason for not just using the revid as the key is so that when a new
+ value is provided for a branch, we replace the old value used for the
+ branch.
+
+ There is another implementation of the same interface in
+ loggerhead.changecache.RevInfoDiskCache.
+ """
+
+ def __init__(self, cache):
+ self._cache = cache
+
+ def get(self, key, revid):
+ """Return the data associated with `key`, subject to a revid check.
+
+ If a value was stored under `key`, with the same revid, return it.
+ Otherwise return None.
+ """
+ cached = self._cache.get(key)
+ if cached is None:
+ return None
+ stored_revid, data = cached
+ if revid == stored_revid:
+ return data
+ else:
+ return None
+
+ def set(self, key, revid, data):
+ """Store `data` under `key`, to be checked against `revid` on get().
+ """
+ self._cache[key] = (revid, data)
+
+
class History (object):
"""Decorate a branch to provide information for rendering.
@@ -185,13 +224,76 @@
around it, serve the request, throw the History object away, unlock the
branch and throw it away.
- :ivar _file_change_cache: xx
+ :ivar _file_change_cache: An object that caches information about the
+ files that changed between two revisions.
+ :ivar _rev_info: A list of information about revisions. This is by far
+ the most cryptic data structure in loggerhead. At the top level, it
+ is a list of 3-tuples [(merge-info, where-merged, parents)].
+ `merge-info` is (seq, revid, merge_depth, revno_str, end_of_merge) --
+ like a merged sorted list, but the revno is stringified.
+ `where-merged` is a tuple of revisions that have this revision as a
+ non-lefthand parent. Finally, `parents` is just the usual list of
+ parents of this revision.
+ :ivar _rev_indices: A dictionary mapping each revision id to the index of
+ the information about it in _rev_info.
+ :ivar _full_history: A list of all revision ids in the ancestry of the
+ branch, in merge-sorted order. This is a bit silly, and shouldn't
+ really be stored on the instance...
+ :ivar _revno_revid: A dictionary mapping stringified revnos to revision
+ ids.
"""
- def __init__(self, branch, whole_history_data_cache):
+ def _load_whole_history_data(self, caches, cache_key):
+ """Set the attributes relating to the whole history of the branch.
+
+ :param caches: a list of caches with interfaces like
+ `RevInfoMemoryCache` and be ordered from fastest to slowest.
+ :param cache_key: the key to use with the caches.
+ """
+ self._rev_indices = None
+ self._rev_info = None
+
+ missed_caches = []
+ def update_missed_caches():
+ for cache in missed_caches:
+ cache.set(cache_key, self.last_revid, self._rev_info)
+ for cache in caches:
+ data = cache.get(cache_key, self.last_revid)
+ if data is not None:
+ self._rev_info = data
+ update_missed_caches()
+ break
+ else:
+ missed_caches.append(cache)
+ else:
+ whole_history_data = compute_whole_history_data(self._branch)
+ self._rev_info, self._rev_indices = whole_history_data
+ update_missed_caches()
+
+ if self._rev_indices is not None:
+ self._full_history = []
+ self._revno_revid = {}
+ for ((_, revid, _, revno_str, _), _, _) in self._rev_info:
+ self._revno_revid[revno_str] = revid
+ self._full_history.append(revid)
+ else:
+ self._full_history = []
+ self._revno_revid = {}
+ self._rev_indices = {}
+ for ((seq, revid, _, revno_str, _), _, _) in self._rev_info:
+ self._rev_indices[revid] = seq
+ self._revno_revid[revno_str] = revid
+ self._full_history.append(revid)
+
+ def __init__(self, branch, whole_history_data_cache, file_cache=None,
+ revinfo_disk_cache=None, cache_key=None):
assert branch.is_locked(), (
"Can only construct a History object with a read-locked branch.")
- self._file_change_cache = None
+ if file_cache is not None:
+ self._file_change_cache = file_cache
+ file_cache.history = self
+ else:
+ self._file_change_cache = None
self._branch = branch
self._inventory_cache = {}
self._branch_nick = self._branch.get_config().get_nickname()
@@ -199,17 +301,10 @@
self.last_revid = branch.last_revision()
- whole_history_data = whole_history_data_cache.get(self.last_revid)
- if whole_history_data is None:
- whole_history_data = compute_whole_history_data(branch)
- whole_history_data_cache[self.last_revid] = whole_history_data
-
- (self._revision_graph, self._full_history, self._revision_info,
- self._revno_revid, self._merge_sort, self._where_merged,
- ) = whole_history_data
-
- def use_file_cache(self, cache):
- self._file_change_cache = cache
+ caches = [RevInfoMemoryCache(whole_history_data_cache)]
+ if revinfo_disk_cache:
+ caches.append(revinfo_disk_cache)
+ self._load_whole_history_data(caches, cache_key)
@property
def has_revisions(self):
@@ -219,12 +314,12 @@
return self._branch.get_config()
def get_revno(self, revid):
- if revid not in self._revision_info:
+ if revid not in self._rev_indices:
# ghost parent?
return 'unknown'
- (seq, revid, merge_depth,
- revno_str, end_of_merge) = self._revision_info[revid]
- return revno_str
+ seq = self._rev_indices[revid]
+ revno = self._rev_info[seq][0][3]
+ return revno
def get_revids_from(self, revid_list, start_revid):
"""
@@ -238,10 +333,11 @@
def introduced_revisions(revid):
r = set([revid])
- seq, revid, md, revno, end_of_merge = self._revision_info[revid]
+ seq = self._rev_indices[revid]
+ md = self._rev_info[seq][0][2]
i = seq + 1
- while i < len(self._merge_sort) and self._merge_sort[i][2] > md:
- r.add(self._merge_sort[i][1])
+ while i < len(self._rev_info) and self._rev_info[i][0][2] > md:
+ r.add(self._rev_info[i][0][1])
i += 1
return r
while 1:
@@ -249,7 +345,7 @@
return
if introduced_revisions(revid) & revid_set:
yield revid
- parents = self._revision_graph[revid]
+ parents = self._rev_info[self._rev_indices[revid]][2]
if len(parents) == 0:
return
revid = parents[0]
@@ -466,10 +562,10 @@
merge_point = []
while True:
- children = self._where_merged.get(revid, [])
+ children = self._rev_info[self._rev_indices[revid]][1]
nexts = []
for child in children:
- child_parents = self._revision_graph[child]
+ child_parents = self._rev_info[self._rev_indices[child]][2]
if child_parents[0] == revid:
nexts.append(child)
else:
=== modified file 'loggerhead/middleware/profile.py'
--- a/loggerhead/middleware/profile.py 2009-03-30 22:05:16 +0000
+++ b/loggerhead/middleware/profile.py 2009-04-01 22:50:25 +0000
@@ -35,75 +35,3 @@
self.lock.release()
-class MemoryProfileMiddleware(object):
- '''Paste middleware for profiling memory with heapy.'''
-
- def __init__(self, app, limit=40):
- self.app = app
- self.lock = threading.Lock()
-
- self.type2count = {}
- self.type2all = {}
- self.limit = limit
-
- def update(self):
- try:
- obs = sys.getobjects(0)
- except AttributeError:
- logging.error(
- 'Python does not have debug symbols compiled. Memory will '
- 'not be profiled.')
- return
- type2count = {}
- type2all = {}
- for o in obs:
- all = sys.getrefcount(o)
-
- if type(o) is str and o == '<dummy key>':
- # avoid dictionary madness
- continue
- t = type(o)
- if t in type2count:
- type2count[t] += 1
- type2all[t] += all
- else:
- type2count[t] = 1
- type2all[t] = all
-
- ct = [(type2count[t] - self.type2count.get(t, 0),
- type2all[t] - self.type2all.get(t, 0),
- t)
- for t in type2count.iterkeys()]
- ct.sort()
- ct.reverse()
- printed = False
-
- logger = logging.getLogger('loggerhead-memprofile')
- logger.debug('*' * 20)
- logger.debug("Loggerhead Memory Profiling")
- i = 0
- for delta1, delta2, t in ct:
- if delta1 or delta2:
- if not printed:
- logger.debug("%-55s %8s %8s" % ('', 'insts', 'refs'))
- printed = True
-
- logger.debug("%-55s %8d %8d" % (t, delta1, delta2))
-
- i += 1
- if i >= self.limit:
- break
-
- self.type2count = type2count
- self.type2all = type2all
-
- def __call__(self, environ, start_response):
- self.lock.acquire()
- try:
- # We don't want to be working with the static images
- # TODO: There needs to be a better way to do this.
- self.update()
- return self.app(environ, start_response)
- finally:
- self.lock.release()
-
=== modified file 'loggerhead/search.py'
--- a/loggerhead/search.py 2008-11-05 00:50:06 +0000
+++ b/loggerhead/search.py 2009-04-17 01:50:05 +0000
@@ -17,13 +17,17 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
-import sets
-try:
- from bzrlib.plugins.search import errors
- from bzrlib.plugins.search import index as _mod_index
- from bzrlib.plugins.search.index import FileTextHit, RevisionHit
-except ImportError:
- _mod_index = None
+_mod_index = None
+def import_search():
+ global errors, _mod_index, FileTextHit, RevisionHit
+ if _mod_index is not None:
+ return
+ try:
+ from bzrlib.plugins.search import errors
+ from bzrlib.plugins.search import index as _mod_index
+ from bzrlib.plugins.search.index import FileTextHit, RevisionHit
+ except ImportError:
+ _mod_index = None
def search_revisions(branch, query_list, suggest=False):
@@ -36,6 +40,7 @@
param suggest: Optional flag to request suggestions instead of results
return: A list for results, either revision ids or terms
"""
+ import_search()
if _mod_index is None:
return None # None indicates could-not-search
try:
@@ -59,6 +64,6 @@
revid_list.append(result.text_key[1])
elif isinstance(result, RevisionHit):
revid_list.append(result.revision_key[0])
- return list(sets.Set(revid_list))
+ return list(set(revid_list))
finally:
index._branch.unlock()
=== modified file 'loggerhead/wholehistory.py'
--- a/loggerhead/wholehistory.py 2008-12-05 20:09:14 +0000
+++ b/loggerhead/wholehistory.py 2009-04-24 10:30:15 +0000
@@ -37,6 +37,10 @@
def compute_whole_history_data(branch):
+ """Compute _rev_info and _rev_indices for a branch.
+
+ See History.__doc__ for what these data structures mean.
+ """
z = time.time()
last_revid = branch.last_revision()
@@ -50,30 +54,32 @@
_revision_graph = _strip_NULL_ghosts(parent_map)
_full_history = []
- _revision_info = {}
- _revno_revid = {}
+
+ _rev_info = []
+ _rev_indices = {}
+
if is_null(last_revid):
_merge_sort = []
else:
_merge_sort = merge_sort(
_revision_graph, last_revid, generate_revno=True)
- for (seq, revid, merge_depth, revno, end_of_merge) in _merge_sort:
+ for info in _merge_sort:
+ seq, revid, merge_depth, revno, end_of_merge = info
_full_history.append(revid)
revno_str = '.'.join(str(n) for n in revno)
- _revno_revid[revno_str] = revid
- _revision_info[revid] = (
- seq, revid, merge_depth, revno_str, end_of_merge)
-
- _where_merged = {}
+ parents = _revision_graph[revid]
+ _rev_indices[revid] = len(_rev_info)
+ _rev_info.append([(seq, revid, merge_depth, revno_str, end_of_merge), (), parents])
for revid in _revision_graph.keys():
- if _revision_info[revid][2] == 0:
+ if _rev_info[_rev_indices[revid]][0][2] == 0:
continue
for parent in _revision_graph[revid]:
- _where_merged.setdefault(parent, set()).add(revid)
+ c = _rev_info[_rev_indices[parent]]
+ if revid not in c[1]:
+ c[1] = c[1] + (revid,)
log.info('built revision graph cache: %r secs' % (time.time() - z))
- return (_revision_graph, _full_history, _revision_info,
- _revno_revid, _merge_sort, _where_merged)
+ return (_rev_info, _rev_indices)
=== modified file 'serve-branches'
--- a/serve-branches 2009-04-01 15:56:24 +0000
+++ b/serve-branches 2009-05-04 20:21:08 +0000
@@ -20,14 +20,15 @@
import sys
from bzrlib.plugin import load_plugins
+from bzrlib.transport import get_transport
from paste import httpserver
from paste.httpexceptions import HTTPExceptionHandler, HTTPInternalServerError
from paste.translogger import TransLogger
from loggerhead import __version__
-from loggerhead.apps.filesystem import (
- BranchesFromFileSystemRoot, UserBranchesFromFileSystemRoot)
+from loggerhead.apps.transport import (
+ BranchesFromTransportRoot, UserBranchesFromTransportRoot)
from loggerhead.config import LoggerheadConfig
from loggerhead.util import Reloader
from loggerhead.apps.error import ErrorHandlerApp
@@ -48,9 +49,9 @@
else:
path = '.'
- if not os.path.isdir(path):
- print "%s is not a directory" % path
- sys.exit(1)
+ load_plugins()
+
+ transport = get_transport(path)
if config.get_option('trunk_dir') and not config.get_option('user_dirs'):
print "--trunk-dir is only valid with --user-dirs"
@@ -63,13 +64,12 @@
return Reloader.restart_with_reloader()
if config.get_option('user_dirs'):
- if not config.get_option['trunk_dir']:
+ if not config.get_option('trunk_dir'):
print "You didn't specify a directory for the trunk directories."
sys.exit(1)
- app = UserBranchesFromFileSystemRoot(
- path, config.get_option('trunk_dir'))
+ app = UserBranchesFromTransportRoot(transport, config)
else:
- app = BranchesFromFileSystemRoot(path)
+ app = BranchesFromTransportRoot(transport, config)
# setup_logging()
logging.basicConfig()
@@ -77,7 +77,7 @@
logger = getattr(app, 'log', logging.getLogger('loggerhead'))
if config.get_option('log_folder'):
logfile_path = os.path.join(
- config.get_option['log_folder'], 'serve-branches.log')
+ config.get_option('log_folder'), 'serve-branches.log')
else:
logfile_path = 'serve-branches.log'
logfile = logging.FileHandler(logfile_path, 'a')
@@ -87,11 +87,6 @@
logfile.setLevel(logging.DEBUG)
logger.addHandler(logfile)
- if config.get_option('memory_profile'):
- memprofile = logging.getLogger('loggerhead-memprofile')
- memprofile.setLevel(logging.DEBUG)
- memprofile.addHandler(logging.FileHandler('loggerhead-memprofile'))
-
# setup_logging() #end
app = TransLogger(app, logger=logger)
@@ -99,8 +94,8 @@
from loggerhead.middleware.profile import LSProfMiddleware
app = LSProfMiddleware(app)
if config.get_option('memory_profile'):
- from loggerhead.middleware.profile import MemoryProfileMiddleware
- app = MemoryProfileMiddleware(app)
+ from dozer import Dozer
+ app = Dozer(app)
if not config.get_option('user_prefix'):
prefix = '/'
@@ -140,8 +135,6 @@
else:
host = config.get_option('user_host')
- load_plugins()
-
httpserver.serve(app, host=host, port=port)
More information about the Pkg-bazaar-commits
mailing list