r2954 - in zodb/trunk/debian/patches (2 files)

[jmuchemb-guest at users.alioth.debian.org]

    Date: Tuesday, August 11, 2015 @ 13:45:49
  Author: jmuchemb-guest
Revision: 2954

Add patch to fix upstream regression in FileStorage

Added:
  zodb/trunk/debian/patches/Fix-possible-data-corruption-after-FileStorage-is-tr.patch
Modified:
  zodb/trunk/debian/patches/series

Added: zodb/trunk/debian/patches/Fix-possible-data-corruption-after-FileStorage-is-tr.patch
===================================================================
--- zodb/trunk/debian/patches/Fix-possible-data-corruption-after-FileStorage-is-tr.patch	                        (rev 0)
+++ zodb/trunk/debian/patches/Fix-possible-data-corruption-after-FileStorage-is-tr.patch	2015-08-11 13:45:49 UTC (rev 2954)
@@ -0,0 +1,76 @@
+From: Julien Muchembled <jm at nexedi.com>
+Date: Tue, 11 Feb 2014 19:17:07 +0100
+Subject: Fix possible data corruption after FileStorage is truncated to roll
+ back a transaction
+
+Multi-threaded IO support, which is new to ZODB 3.10, allows clients to read
+data (load & loadBefore) even after tpc_vote has started to write a new
+transaction to disk. This is done by using different 'file' objects.
+
+Issues start when a transaction is rolled back after data has been appended
+(using the writing file object). Truncating is not enough because the FilePool
+may have been used concurrently to read the end of the last transaction:
+file objects have their own read buffers which, in this case, may also contain
+the beginning of the aborted transaction.
+
+So a solution is to invalidate read buffers whenever they may contain wrong
+data. This patch does it on truncation, which happens rarely enough to not
+affect performance.
+
+We discovered this bug in the following conditions:
+- ZODB splitted in several FileStorage
+- many conflicts in the first committed DB, but always resolved
+- unresolved conflict in another DB
+If the transaction is replayed with success (no more conflict in the other DB),
+a subsequent load of the object that could be resolved in the first DB may, for
+example, return a wrong serial (tid of the aborted transaction) if the layout
+of the committed transaction matches that of the aborted one.
+
+The bug usually manifests with POSKeyError & CorruptedDataError exceptions in
+ZEO logs (+ ZEO freeze due to https://github.com/zopefoundation/ZODB/pull/15),
+for example while trying to resolve a conflict (and restarting the transaction
+does not help, causing Site Errors in Zope).
+But theorically, this could also cause silent corruption or unpickling errors
+at client side.
+
+Forwarded: http://thread.gmane.org/gmane.comp.web.zope.zodb/12685/focus=12751
+---
+ src/ZODB/FileStorage/FileStorage.py | 11 +++++++++++
+ 1 file changed, 11 insertions(+)
+
+diff --git a/src/ZODB/FileStorage/FileStorage.py b/src/ZODB/FileStorage/FileStorage.py
+index be9d58b..241cfdb 100644
+--- a/src/ZODB/FileStorage/FileStorage.py
++++ b/src/ZODB/FileStorage/FileStorage.py
+@@ -683,6 +683,7 @@ def tpc_vote(self, transaction):
+                 # Hm, an error occurred writing out the data. Maybe the
+                 # disk is full. We don't want any turd at the end.
+                 self._file.truncate(self._pos)
++                self._files.flush()
+                 raise
+             self._nextpos = self._pos + (tl + 8)
+ 
+@@ -737,6 +738,7 @@ def _finish_finish(self, tid):
+     def _abort(self):
+         if self._nextpos:
+             self._file.truncate(self._pos)
++            self._files.flush()
+             self._nextpos=0
+             self._blob_tpc_abort()
+ 
+@@ -1997,6 +1999,15 @@ def __init__(self, file_name):
+         self._out = []
+         self._cond = threading.Condition()
+ 
++    def flush(self):
++        """Empty read buffers.
++
++        This is required if they contain data of rolled back transactions.
++        """
++        with self.write_lock():
++            for f in self._files:
++                f.flush()
++
+     @contextlib.contextmanager
+     def write_lock(self):
+         with self._cond:

Modified: zodb/trunk/debian/patches/series
===================================================================
--- zodb/trunk/debian/patches/series	2015-08-11 13:45:43 UTC (rev 2953)
+++ zodb/trunk/debian/patches/series	2015-08-11 13:45:49 UTC (rev 2954)
@@ -1,3 +1,4 @@
 Make-compatible-with-python-persistent-4.x.patch
 Don-t-provide-persistent-headers-in-python-zodb.patch
 manuel-is-not-packaged-for-Debian.patch
+Fix-possible-data-corruption-after-FileStorage-is-tr.patch