[h5py] 95/455: Cleanup in preparation for 0.3
Ghislain Vaillant
ghisvail-guest at moszumanska.debian.org
Thu Jul 2 18:19:21 UTC 2015
This is an automated email from the git hooks/post-receive script.
ghisvail-guest pushed a commit to annotated tag 1.3.0
in repository h5py.
commit e4c92004aec9a35664e934083ed4538c28794a62
Author: andrewcollette <andrew.collette at gmail.com>
Date: Thu Aug 7 02:05:05 2008 +0000
Cleanup in preparation for 0.3
---
CHANGES.txt | 19 ++++-
INSTALL.txt | 48 ------------
README.txt | 27 +++----
autotest.py | 36 +++++++++
docs/source/build.rst | 138 ++++++++++++++++++++++++++++++++++
docs/source/conf.py | 4 +-
docs/source/index.rst | 36 +++++++--
docs/source/{info.rst => threads.rst} | 35 +++++----
h5py/highlevel.py | 3 +
setup.py | 37 ++-------
10 files changed, 266 insertions(+), 117 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index c323355..8db6bbe 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,7 +1,24 @@
-Version 0.2.3
+Version 0.3.0
=============
+General
+-------
+ - Now compiles on Windows; installer available
+ - Sphinx-based manual started
+
+Low-level
+---------
+ - Threading infrastructure & non-blocking I/O patch
+ - Threading unit tests
+ - New module h5fd for constants needed by h5p
- Moved get_objinfo out of GroupID into h5g, as it accepts datasets, etc.
+ - Work around for 1.8.1 Windows misbehavior: statbuf.linklen always 0
+ - ObjectID instances can be hashed and compared sanely
+ - New global configuration object "h5py.config"
+
+High-level
+----------
+ - All methods are now thread-safe
Version 0.2.2
=============
diff --git a/INSTALL.txt b/INSTALL.txt
deleted file mode 100644
index b35a6a3..0000000
--- a/INSTALL.txt
+++ /dev/null
@@ -1,48 +0,0 @@
-Installation instructions for the h5py package
-==============================================
-
-Version 0.2.2
-
-This package is designed to be installed from source. You will need
-Python and a C compiler, for distutils to build the extensions. Pyrex_ is
-not required unless you want to change compile-time options, like the
-debugging level.
-
-It's strongly recommended you use the versions of these packages provided
-by your operating system's package manager/finder. In particular, HDF5 can
-be painful to install manually.
-
-Requires
---------
-- Unix-like environment (created/tested on 32-bit Intel linux)
-- Python with headers for development ("python-dev")
-- Numpy_ 1.0.3 or higher
-- HDF5_ 1.6.5 or higher (1.8 support experimental)
-- A working compiler for distutils
-- (Optionally) Pyrex_ 0.9.8.4 or higher
-
-.. _Numpy: http://numpy.scipy.org/
-.. _HDF5: http://hdf.ncsa.uiuc.edu/
-.. _Pyrex: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/
-
-Procedure
----------
-1. Unpack the tarball and cd to the resulting directory
-2. Run ``python setup.py build`` to build the package
-3. Run ``python setup.py test`` to run unit tests in-place (optional)
-4. Run ``sudo python setup.py install`` to install into your main Python
- package directory.
-
-Additional options
-------------------
- --pyrex Have Pyrex recompile changed pyx files.
- --pyrex-only Have Pyrex recompile changed pyx files, and stop.
- --pyrex-force Recompile all pyx files, regardless of timestamps.
- --no-pyrex Don't run Pyrex, no matter what
-
- --hdf5=path Use alternate HDF5 directory (containing bin, include, lib)
- --api=<n> Specifies API version. Only --api=16 is currently useful.
- --debug=<n> If nonzero, compile in debug mode. The number is
- interpreted as a logging-module level number. Requires
- Pyrex for recompilation.
-
diff --git a/README.txt b/README.txt
index ca92ba8..2b0581d 100644
--- a/README.txt
+++ b/README.txt
@@ -2,40 +2,38 @@ README for the "h5py" Python/HDF5 interface
===========================================
Copyright (c) 2008 Andrew Collette
-* http://h5py.alfven.org
-* mail: "h5py" at the domain "alfven dot org"
-
-**Version 0.2.2**
+Version 0.3.0
-* `Introduction`_
-* `Features`_
+* http://h5py.alfven.org Main site, docs
+* http://h5py.googlecode.com Downloads, FAQ and bug tracker
-Introduction
-============
+* mail: "h5py" at the domain "alfven dot org"
The h5py package provides both a high- and low-level interface to the
-`NCSA HDF5 library`__ from Python. The low-level interface is
+HDF5 library from Python. The low-level interface is
intended to be a complete wrapping of the HDF5 1.6 API, while the high-
level component supports Python-style object-oriented access to HDF5 files,
datasets and groups.
-__ http://hdf.ncsa.uiuc.edu
-
The goal of this package is not to provide yet another scientific data
model. It is an attempt to create as straightforward a binding as possible
to the existing HDF5 API and abstractions, so that Python programs can
easily deal with HDF5 files and exchange data with other HDF5-aware
applications.
+Quick installation
+------------------
+On Unix-like systems with gcc and distutils, run "python setup.py build"
+followed by "sudo python setup.py install". See INSTALL.txt or the online
+guide at h5py.alfven.org for more details.
+
Documentation
-------------
Extensive documentation is available through docstrings, as well as in
-`HTML format on the web`__ and in the "docs/" directory in the
+HTML format on the web and in the "docs/" directory in the
distribution. This document is an overview of some of the package's
features and highlights.
-__ http://h5py.alfven.org/docs
-
Features
========
@@ -85,7 +83,6 @@ Low-Level
- The majority of the HDF5 C-API documentation is still valid for the
h5py interface
-
Bugs
----
I expect there are still a few. :) A FAQ page will soon be created at the
diff --git a/autotest.py b/autotest.py
new file mode 100644
index 0000000..7029205
--- /dev/null
+++ b/autotest.py
@@ -0,0 +1,36 @@
+# Development script to test setup options
+import os.path as op
+import os, sys
+
+# HDF5 library versions
+libnames = ['h167','h180','h181']
+lib_base = op.abspath(op.join(op.curdir, '..'))
+libs = [''] + [' --hdf5='+op.join(lib_base, x) for x in libnames]
+
+# Experimental non-blocking I/O
+nonblock = ['',' --io-nonblock']
+
+# API versions
+api = [' --api=16', ' --api=18']
+
+linebase = "python setup.py test --pyrex-force%s%s%s"
+print libs
+print nonblock
+print api
+
+idx=0
+for l in libs:
+ for n in nonblock:
+ for a in api:
+ # Only allow --api=18 if 1.8.X library being tested
+ if '8' in a and not '8' in l:
+ continue
+ line = linebase % (l, n, a)
+ outfile = "autotest%d.txt" % idx
+ print 'Testing config %d "%s"...' % (idx, line)
+ retval = os.system(line+" > %s 2>&1" % outfile)
+ if retval !=0:
+ print '!!! Line failed; output saved to "%s"' % outfile
+ else:
+ os.unlink(outfile)
+ idx += 1
diff --git a/docs/source/build.rst b/docs/source/build.rst
new file mode 100644
index 0000000..98b800e
--- /dev/null
+++ b/docs/source/build.rst
@@ -0,0 +1,138 @@
+******************
+Installation guide
+******************
+
+Installing on Linux/Mac OS-X
+============================
+
+This package is designed to be installed from source. You will need
+Python and a C compiler, for distutils to build the extensions. Pyrex_ is
+required only if you want to change compile-time options, like the
+debugging level.
+
+It's strongly recommended you use the versions of these packages provided
+by your operating system's package manager/finder. In particular, HDF5 can
+be painful to install manually.
+
+Requires
+--------
+- Unix-like environment (created/tested on 32-bit Intel linux)
+- Python with headers for development
+- Numpy_ 1.0.3 or higher
+- HDF5_ 1.6.5 or higher, including 1.8.X versions
+- A working compiler for distutils
+- (Optionally) Pyrex_ 0.9.8.4 or higher
+
+.. _Numpy: http://numpy.scipy.org/
+.. _HDF5: http://www.hdfgroup.com/HDF5/
+.. _Pyrex: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/
+
+Procedure
+---------
+1. Unpack the tarball and cd to the resulting directory
+2. Run ``python setup.py build`` to build the package
+3. Run ``python setup.py test`` to run unit tests in-place (optional)
+4. Run ``sudo python setup.py install`` to install into your main Python
+ package directory.
+
+Additional options
+------------------
+
+::
+
+ --pyrex Have Pyrex recompile changed pyx files.
+ --pyrex-only Have Pyrex recompile changed pyx files, and stop.
+ --pyrex-force Recompile all pyx files, regardless of timestamps.
+ --no-pyrex Don't run Pyrex, no matter what
+
+ --hdf5=path Use alternate HDF5 directory (containing bin, include, lib)
+
+ --api=<n> Specifies API version. The 1.8.X API (--api=18) is a
+ work in progress.
+
+ --debug=<n> If nonzero, compile in debug mode. The number is
+ interpreted as a logging-module level number. Requires
+ Pyrex for recompilation.
+
+ --io-nonblock Enable experimental non-blocking I/O feature. The GIL will
+ be released around lengthy HDF5 reads/writes. See the
+ "Threading" manual entry for caveats.
+
+
+Installing on Windows
+=====================
+
+**It's strongly recommended that you use the pre-built .exe installer.** It
+will install h5py, a private copy of HDF5 1.8.1 with ZLIB and SZIP compression
+enabled, and the proper C runtime dependencies. You must have the following
+already installed:
+
+- Python 2.5
+- Numpy_ 1.0.3 or higher
+
+If for some reason you want to build from source (for example, to change the
+compile-time options), read the following instructions carefully.
+
+Environment for source build
+----------------------------
+
+1. Install Python 2.5 and Numpy 1.0.3 or higher
+2. Install MinGW to ``C:\MinGW``
+3. Add ``C:\MinGW\bin`` to the Windows PATH
+4. Download the ``pexports`` utility and place it in ``C:\MinGW\bin``
+5. Go to ``C:\Python25\Lib\distutils`` (or equivalent path for your Python install)
+ and create the file "distutils.cfg" with the following text::
+
+ [build]
+ compiler=mingw32
+
+6. Add ``C:\Python25`` (or wherever python.exe lives) to your Windows PATH
+
+Get HDF5 and create import file
+-------------------------------
+
+1. Download the pre-built version of HDF5 1.8.1, along with SZIP and ZLIB.
+ You should use the versions built with Visual Studio 2005. Make sure you
+ get the version of SZIP with compression enabled.
+2. Unpack the HDF5 archive to e.g. ``C:\hdf5``; this directory should now
+ contain ``include`` and ``dll``, among other things.
+3. Open a command prompt in ``C:\hdf5\dll`` and run
+ ``pexports hdf5dll.dll > hdf5dll.def``
+4. Create the directory ``C:\hdf5\dll2`` and move ``hdf5dll.def`` there
+
+Compile h5py
+------------
+
+1. Download h5py and extract it to ``C:\h5py``.
+2. In ``C:\h5py``, run ``python setup.py build --hdf5=C:\hdf5``
+3. Copy the following files to ``C:\h5py\h5py``:
+
+ * hdf5dll.dll (from ``C:\hdf5\dll``)
+ * zlib1.dll (from the HDF group zlib archive)
+ * szipdll.dll (from the HDF group szip archive)
+
+4. Run unit tests via ``python setup.py test --hdf5=C:\hdf5``
+
+.. note::
+
+ If you get the message "DLL import failed: configuration incorrect" or
+ some variant, you need to install the package
+ "Microsoft Visual C++ 2005 SP1 Redistributable" from Microsoft's
+ web site. This is needed by the pre-built HDF5 library for some
+ reason.
+
+5. Install via ``python setup.py install --hdf5=C:\hdf5``.
+
+After you're done, you can delete the ``C:\hdf5`` directory. It isn't needed at
+runtime.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/source/conf.py b/docs/source/conf.py
index dc76071..d28b1b1 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
#
# h5py documentation build configuration file, created by
-# sphinx-quickstart on Wed Jul 30 15:11:11 2008.
+# sphinx-quickstart on Tue Aug 5 14:14:15 2008.
#
# This file is execfile()d with the current directory set to its containing dir.
#
@@ -23,7 +23,7 @@ import sys, os
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = []
+extensions = ['sphinx.ext.autodoc']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 0900350..4d67bfd 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -1,16 +1,42 @@
-.. h5py documentation master file, created by sphinx-quickstart on Wed Jul 30 15:11:11 2008.
+.. h5py documentation master file, created by sphinx-quickstart on Tue Aug 5 14:14:15 2008.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
-Welcome to h5py's documentation!
-================================
+Python bindings for the HDF5 library
+====================================
+
+The h5py package provides both a high- and low-level interface to the
+`HDF5 library`__ from Python. The low-level interface is
+intended to be a complete wrapping of the HDF5 1.6 API, while the high-
+level component supports Python-style object-oriented access to HDF5 files,
+datasets and groups.
+
+__ http://www.hdfgroup.com/HDF5
+
+The goal of this package is not to provide yet another scientific data
+model. It is an attempt to create as straightforward a binding as possible
+to the existing HDF5 API and abstractions, so that Python programs can
+easily deal with HDF5 files and exchange data with other HDF5-aware
+applications.
+
+Important links:
+ * `Main project page`__
+ * `Auto-generated API docs`__
+ * `Google Code development page`__
+ * Mail: "h5py" at the domain "alfven" dot "org"
+
+__ http://h5py.alfven.org
+__ http://h5py.alfven.org/docs
+__ http://h5py.googlecode.com
Contents:
.. toctree::
- :maxdepth: 2
+ :maxdepth: 2
+
+ build
+ threads
- info
Indices and tables
==================
diff --git a/docs/source/info.rst b/docs/source/threads.rst
similarity index 68%
rename from docs/source/info.rst
rename to docs/source/threads.rst
index d4eed5b..1efd03f 100644
--- a/docs/source/info.rst
+++ b/docs/source/threads.rst
@@ -1,7 +1,6 @@
-
-
+*********
Threading
-=========
+*********
Threading is an issue in h5py because HDF5 doesn't support thread-level
concurrency. Some versions of HDF5 are not even thread-safe. The package
@@ -25,26 +24,29 @@ is available on the global configuration object at "h5py.config.lock". The
decorator "h5sync" in h5py.extras can wrap functions to do this automatically.
-More about Non-Blocking Routines
---------------------------------
+Non-Blocking Routines
+---------------------
By default, all low-level HDF5 routines will lock the entire interpreter
until they complete, even in the case of lengthy I/O operations. This is
unnecessarily restrictive, as it means even non-HDF5 threads cannot execute.
-When the package is compiled with the option "--io-nonblock", a few C methods
+When the package is compiled with the option ``--io-nonblock``, a few C methods
involving I/O will release the global interpreter lock. These methods always
acquire the global HDF5 lock before yielding control to other threads. While
-no other HDF5 operation can acquire the HDF5 lock until the write completes,
-other Python threads (GUIs, pure computation threads, etc) will execute in
-a normal fashion.
+another thread seeking to acquire the HDF5 lock will block until the write
+completes, other Python threads (GUIs, pure computation threads, etc) will
+execute in a normal fashion.
-However, if another thread skips acquiring the HDF5 lock and blindly calls a
-low-level HDF5 routine while such I/O is in progress, the results are
-undefined. In the worst case, irreversible data corruption and/or a crash of
-the interpreter is possible. Therefore, it's very important to always acquire
-the global HDF5 lock before calling into the h5py.h5* API when (1) more than
-one thread is performing HDF5 operations, and (2) non-blocking I/O is enabled.
+However, this defeats the thread safety provided by the GIL. If another thread
+skips acquiring the HDF5 lock and blindly calls a low-level HDF5 routine while
+such I/O is in progress, the results are undefined. In the worst case,
+irreversible data corruption and/or a crash of the interpreter is possible.
+Therefore, it's very important to always acquire the global HDF5 lock before
+calling into the h5py.h5* API when all the following are true:
+
+ 1. More than one thread is performing HDF5 operations
+ 2. Non-blocking I/O is enabled
This is not an issue for the h5py.highlevel components (Dataset, Group,
File objects, etc.) as they acquire the lock automatically.
@@ -73,7 +75,8 @@ The default lock type is the native Python threading.RLock, but h5py makes no
assumptions about the behavior or implementation of locks beyond reentrance and
the existence of the four required methods above.
-
+It remains to be seen whether this is even necessary. In future versions of
+h5py, this attribute may disappear or become non-writable.
diff --git a/h5py/highlevel.py b/h5py/highlevel.py
index 0b09db6..506c94f 100644
--- a/h5py/highlevel.py
+++ b/h5py/highlevel.py
@@ -40,6 +40,9 @@
It is safe to import this module using "from h5py.highlevel import *"; it
will export only the major classes.
+
+ Everything in this module is thread-safe, regardless of the HDF5
+ configuration or compile options.
"""
from __future__ import with_statement
diff --git a/setup.py b/setup.py
index 2560c27..4e9fd27 100644
--- a/setup.py
+++ b/setup.py
@@ -29,28 +29,13 @@
To run the test suite locally (won't install anything):
python setup.py test
- Additional options (for all modes):
- --pyrex Have Pyrex recompile changed pyx files.
- --pyrex-only Have Pyrex recompile changed pyx files, and stop.
- --pyrex-force Recompile all pyx files, regardless of timestamps.
- --no-pyrex Don't run Pyrex, no matter what
-
- --hdf5=path Use alternate HDF5 directory (contains bin, include, lib)
- --api=<n> Specifies API version. Only "16" is currently useful.
- --debug=<n> If nonzero, compile in debug mode. The number is
- interpreted as a logging-module level number.
-
- Advanced developer options:
- python setup.py dev [--doc] [--clean] [--readme=<name.html>]
- --doc: Rebuild HTML documentation (requires epydoc)
- --clean: Wipe out build/ and Pyrex-created .c, .dep files
- --readme: Compile the RST readme file into an HTML fragment
+ See INSTALL.txt or the h5py manual for additional build options.
"""
# === Global constants ========================================================
NAME = 'h5py'
-VERSION = '0.2.2'
+VERSION = '0.3.0'
MIN_PYREX = '0.9.8.4' # for compile_multiple
MIN_NUMPY = '1.0.3'
@@ -138,9 +123,11 @@ for arg in sys.argv[:]:
opts.IO_NONBLOCK = True
sys.argv.remove(arg)
-if 'sdist' in sys.argv and os.path.exists('MANIFEST'):
- warn("Cleaning up stale MANIFEST file")
- os.remove('MANIFEST')
+if 'sdist' in sys.argv:
+ if os.path.exists('MANIFEST'):
+ warn("Cleaning up stale MANIFEST file")
+ os.remove('MANIFEST')
+ shutil.copyfile(reduce(op.join, ('docs', 'source', 'build.rst')), 'INSTALL.txt')
# === Required imports ========================================================
@@ -304,7 +291,6 @@ class dev(Command):
description = "Developer commands (--doc, --clean, --readme=<file>)"
user_options = [('doc','d','Rebuild documentation'),
- ('readme=','r','Compile HTML file from README.txt'),
('clean', 'c', 'Remove built files and Pyrex temp files.')]
boolean_options = ['doc']
@@ -353,15 +339,6 @@ class dev(Command):
else:
shutil.copytree('docs/build/html', 'docs/manual-html')
- if self.readme:
- import docutils.core
- fh = open('README.txt','r')
- parts = docutils.core.publish_parts(fh.read(),writer_name='html')
- fh.close()
- fh = open(self.readme,'w')
- fh.write(parts['body'])
- fh.close()
-
# New commands for setup (e.g. "python setup.py test")
if os.name == 'nt':
CMD_CLASS = {'test': test}
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-science/packages/h5py.git
More information about the debian-science-commits
mailing list