[h5py] 107/455: Initial fix for 32-bit indexing restriction
Ghislain Vaillant
ghisvail-guest at moszumanska.debian.org
Thu Jul 2 18:19:23 UTC 2015
This is an automated email from the git hooks/post-receive script.
ghisvail-guest pushed a commit to annotated tag 1.3.0
in repository h5py.
commit 32aa6ab7775456f12719e47a54e8783945f02b5a
Author: andrewcollette <andrew.collette at gmail.com>
Date: Mon Aug 25 00:22:36 2008 +0000
Initial fix for 32-bit indexing restriction
---
docs/source/index.rst | 3 +
docs/source/low.rst | 108 ++++++++++++++++++++++++++++++
docs/source/overview.rst | 162 ---------------------------------------------
h5py/tests/test_threads.py | 3 +-
h5py/utils.pxd | 2 +-
h5py/utils.pyx | 18 +++++
h5py/utils_low.c | 41 ------------
h5py/utils_low.h | 1 -
8 files changed, 132 insertions(+), 206 deletions(-)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index a273a01..1c93ec0 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -36,6 +36,9 @@ Contents:
build
quick
+ low
threads
licenses
+
+
diff --git a/docs/source/low.rst b/docs/source/low.rst
new file mode 100644
index 0000000..16bb962
--- /dev/null
+++ b/docs/source/low.rst
@@ -0,0 +1,108 @@
+==============================
+Low-level (``h5py.h5*``) guide
+==============================
+
+This is a general overview of the lowest-level API in h5py, the layer that
+calls into HDF5 directly. A lot of effort has been put into making even this
+component useful in a Python context. It provides the most general interface
+to HDF5, including the vast majority of the C library.
+
+The definitive documentation for this level is available through docstrings.
+`Auto-generated HTML documentation`__ based on these is available.
+You'll probably also find the `official HDF5 documentation`__ useful as a guide
+to how the library itself operates. In particular, the HDF5 User Guide is
+an excellent description of each major component.
+
+__ http://h5py.alfven.org/docs
+__ http://hdf.ncsa.uiuc.edu/HDF5/doc/index.html
+
+
+Library organization
+====================
+
+Modules
+-------
+
+While HDF5 is a C library, and therefore uses on global namespace for all
+functions and constants, their naming scheme is designed to partition the API
+into modules of related code. H5py uses this as a guide for the Python-side
+module organization. For example, the Python wrapping of the HDF5 function
+``H5Dwrite`` is contained in module ``h5d``, while the Python equivalent of
+``H5Aiterate`` is in module ``h5a``.
+
+Identifier wrapping
+-------------------
+
+No matter how complete, a library full of C functions is not very fun to use.
+Additionally, since HDF5 identifiers are natively handled as integers, their
+lifespan must be manually tracked by the library user. This quickly becomes
+impossible for applications of even a moderate size; errors will lead to
+resource leaks or (in the worst case) accidentally invalidating identifiers.
+
+Rather than a straight C-API mapping, all HDF5 identifiers are presented as
+Python extension types. The root type ``h5.ObjectID`` provides a container
+for an integer identifier, which allows Python reference counting to manage
+the lifespan of the identifer. When no more references exist to the Python
+object, the HDF5 identifier is automatically closed.
+
+A side benefit is that many HDF5 functions take an identifier as their first
+argument. They are naturally expressed as methods on an identifier object.
+For example, the HDF5 function``H5Dwrite`` becomes the method
+``h5d.DatasetID.write``. Code using this technique is easier to write and
+maintain.
+
+State & Hashing
+---------------
+
+Since the ``h5py.h5*`` family of modules is intended to be a straightforward
+interface to HDF5, almost all state information resides with the HDF5 library
+itself. A side effect of this is that the hash and equality operations on
+ObjectID instances are determined by the status of the underlying HDF5 object.
+For example, if two GroupID objects with different HDF5 integer identifiers
+point to the same group, they will have identical hashes and compare equal.
+Among other things, this means that if you can use
+ObjectID/GroupID/DatasetID/etc. instances as keys in a dictionary.
+
+
+Data Conversion
+===============
+
+The natural numerical layer between h5py and the Python user is NumPy. It
+provides the mechanism to transfer large datasets between HDF5 and Python
+analysis routines. Additionally, its type infrastructure ("dtype" objects)
+closely matches the HDF5 hierarchy of types. With very few exceptions, there
+is good mapping between NumPy dtypes and HDF5 basic types.
+
+The actual conversion between datatypes is performed by the optimised routines
+inside the HDF5 library; all h5py does is provide the mapping between NumPy
+and HDF5 type objects. Because the HDF5 typing system is more comprehensive
+than the NumPy system, this is an asymmetrical process. While translating
+from ``NumPy => HDF5`` always results in a bit-for-bit identical description,
+the reverse process ``HDF5 => NumPy`` cannot be guaranteed to result in an
+exact description. In the vast majority of cases, this does not matter; HDF5
+can natively auto-convert a huge variety of representations.
+
+API Versioning
+==============
+
+HDF5 recently went though a major release, in the form of version 1.8.0.
+In addition to various stability improvements, it introduces a number of
+new and changed functions. Rather than force people to use a particular
+version, h5py deals with this by specifying an "API compatibility" level.
+In "1.6" mode, the extension can be compiled with either 1.6.X or 1.8.X, and
+will function identically. In this mode, only the functions from the 1.6.X
+series are exposed. In "1.8" mode, new features and function signatures from
+HDF5 1.8.X are available.
+
+Currently, while h5py can be built in both modes, not many 1.8.X features are
+available.
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/source/overview.rst b/docs/source/overview.rst
deleted file mode 100644
index 7172c16..0000000
--- a/docs/source/overview.rst
+++ /dev/null
@@ -1,162 +0,0 @@
-********
-Overview
-********
-
-.. note::
-
- This document assumes a basic familiarity with HDF5, including groups,
- datasets, and the file hierarchy. For more information, the `user guide
- distributed by the HDF Group`__ is an excellent introduction. It also
- assumes you have used `Numpy`_ before.
-
-__ http://hdf.ncsa.uiuc.edu/HDF5/doc/UG/index.html
-.. _Numpy: http://numpy.scipy.org
-
-High-level classes
-==================
-
-While the ``h5py.h5*`` modules provide access to the guts of the HDF5 library,
-they are not very convenient for everyday use. For example, creating a new
-dataset with chunking and compression requires creating datatype, dataspace and
-property list objects, assigning the correct values to the property list in
-the right order, and managing the group object to which the dataset will be
-attached. To make interacting with HDF5 less painful, a pure-Python
-"high-level" interface is provided to encapsulate these common operations.
-
-It consists of three classes:
-
-* File: Represents an HDF5 file on disk
-* Group: Represents an HDF5 group, containing datasets and other groups
-* Dataset: An HDF5 dataset
-
-All communication with HDF5 is done through Numpy arrays, dtype objects, and
-slicing conventions. No low-level HDF5 objects (datatype/dataspace objects,
-etc.) are exposed, and no additional Python-side abstractions are introduced.
-Objects typically have a small number of methods.
-
-Paths in HDF5 files
--------------------
-
-HDF5 files are organized like a filesystem. Groups are analagous to
-directories, while datasets are like the files stored in them. Paths are
-always specified UNIX-style, starting at ``/`` (the "root" group). It's good
-to limit yourself to ASCII, but you're welcome to use spaces, quotes,
-punctuation and other symbol characters.
-
-
-Group objects
--------------
-
-These represent HDF5 *groups*, directory-like objects which contain *links* to
-HDF5 objects like datasets, or other groups. To a good approximation, you
-can think of them as dictionaries which map a string name to an HDF5 object.
-Like objects stored in Python dictionaries, the same HDF5 object can be
-referred to by more than one group. Groups can even contain themselves!
-
-This dictionary metaphor is how h5py treats groups. They support the following
-Python behavior:
-
-* Item access (``group["name"]``)
-
- Accessing a member by name returns the appropriate HDF5 object; usually a
- dataset or another group. Assigning to a name stores the object in the file
- in the appropriate way (see the docstring for details). Deleting an item
- "unlinks" it from the group. Like Python objects in dictionaries, when zero
- groups refer to an object, it's permanently gone.
-
-* Iteration (``for x in group...``, etc.)
-
- Iterating over a group yields the *names* of its members, like a Python dict.
- You can use the method ``iteritems()`` to get ``(name, obj)`` tuples. The
- same restrictions as in Python apply for iteration; don't modify the group
- while iterating.
-
-* Length (``len(group)``)
-
- This is just the number of entries in the group.
-
-* Membership (``if "name" in group...``)
-
- Test if a name appears in the group.
-
-They also support the following methods:
-
-.. method:: create_group(name)
-
- Create a new, empty group attached to this one, called "name".
-
-.. method:: create_dataset(name, *args, **kwds)
-
- Create a new dataset attached to this group. The arguments are passed to
- the Dataset constructor below.
-
-
-File objects
-------------
-
-These represent the HDF5 file itself. Since every HDF5 file contains at least
-the *root group* (called ``/``, as in a POSIX filesystem), it also provides
-your entry point into the file. File objects inherit from Group objects; all
-the Group behavior and methods are available, and will operate on the root
-(``/``) group. For example, to access datasets::
-
- ds1 = file_obj["/ds1"] (or file_obj["ds1"])
- ds2 = file_obj["/grp1/ds2"] (or file_obj["grp1/ds2"])
-
-
-Opening (or creating) a file
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You use the File constructor directly to open or create an HDF5 file. Like the
-standard Python function ``open()``, the syntax is:
-
-.. method:: File(name, mode='a')
-
- Allowed modes are:
-
- - r Readonly, file must exist
- - r+ Read/write, file must exist
- - w Create file, truncate if exists
- - w- Create file, fail if exists
- - a Read/write if exists, create otherwise (default)
-
-.. method:: close()
-
- When you're done, as with Python files, it's important to close the file so
- that all the data gets written.
-
-
-Python attributes
-~~~~~~~~~~~~~~~~~
-
-.. attribute:: name
-
- Name used to open the file
-
-.. attribute:: mode
-
- Mode character used to open the file
-
-
-Browsing a file
-~~~~~~~~~~~~~~~
-
-.. method:: browse()
-
- Specifying the full name of an HDF5 resource can be tedious and error-prone.
- Therefore, h5py includes a small command-line browser which can be used like
- a UNIX shell to explore an HDF5 file and import datasets and groups into an
- interactive session. It includes things like ``ls`` and tab-completion. To
- open the browser, simply call browse(). Type ``help`` at the prompt for a
- list of commands.
-
-
-
-
-
-
-
-
-
-
-
diff --git a/h5py/tests/test_threads.py b/h5py/tests/test_threads.py
index e29dfda..b8f6587 100644
--- a/h5py/tests/test_threads.py
+++ b/h5py/tests/test_threads.py
@@ -230,12 +230,13 @@ class TestThreads(unittest.TestCase):
thread_b = SleeperThread(1)
thread_a.start()
+ time.sleep(0.2)
thread_b.start()
thread_a.join()
thread_b.join()
- self.assert_(thread_a.time < thread_b.time)
+ self.assert_(thread_a.time < thread_b.time, "%f :: %f" % (thread_a.time, thread_b.time))
@h5sync
def thisismyname(foo):
diff --git a/h5py/utils.pxd b/h5py/utils.pxd
index 65f8660..34d6b65 100644
--- a/h5py/utils.pxd
+++ b/h5py/utils.pxd
@@ -21,7 +21,6 @@ cdef extern from "utils_low.h":
hid_t create_ieee_complex128(char byteorder, char* real_name, char* img_name) except -1
# Tuple conversion
- int convert_tuple(object tpl, hsize_t *dims, hsize_t rank) except -1
object convert_dims(hsize_t* dims, hsize_t rank) # automatic except
# Numpy array validation
@@ -34,6 +33,7 @@ cdef extern from "utils_low.h":
# === Custom API ==============================================================
+cdef int convert_tuple(object tuple, hsize_t *dims, hsize_t rank) except -1
cdef int require_tuple(object tpl, int none_allowed, int size, char* name) except -1
cdef int require_list(object lst, int none_allowed, int size, char* name) except -1
cdef object pybool(long long val)
diff --git a/h5py/utils.pyx b/h5py/utils.pyx
index 6d3901b..b59f573 100644
--- a/h5py/utils.pyx
+++ b/h5py/utils.pyx
@@ -18,6 +18,24 @@ from numpy cimport import_array, NPY_UINT16, NPY_UINT32, NPY_UINT64, \
import_array()
+cdef int convert_tuple(object tpl, hsize_t *dims, hsize_t rank) except -1:
+ # Convert a Python tuple to an hsize_t array. You must allocate
+ # the array yourself and pass both it and the size to this function.
+ # Returns 0 on success, -1 on failure and raises an exception.
+ cdef int i
+
+ if len(tpl) != rank:
+ raise ValueError("Tuple length incompatible with array")
+
+ try:
+ for i from 0<=i<rank:
+ dims[i] = long(tpl[i])
+ except TypeError:
+ raise TypeError("Can't convert element %d (%s) to a long" % (i, tpl[i]))
+
+ return 0
+
+
cdef object create_numpy_hsize(int rank, hsize_t* dims):
# Create an empty Numpy array which can hold HDF5 hsize_t entries
diff --git a/h5py/utils_low.c b/h5py/utils_low.c
index 927d588..1811b39 100644
--- a/h5py/utils_low.c
+++ b/h5py/utils_low.c
@@ -84,47 +84,6 @@ PyObject* convert_dims(hsize_t* dims, hsize_t rank) {
return NULL;
}
-/* Convert a Python tuple to an hsize_t array. You must allocate
- the array yourself and pass both it and the size to this function.
- Returns 0 on success, -1 on failure and raises an exception.
-*/
-int convert_tuple(PyObject* tpl, hsize_t *dims, hsize_t rank_in){
-
- PyObject* temp = NULL;
- int rank;
- int i;
-
- if(tpl == NULL) goto err;
- if(!PyTuple_Check(tpl)) goto err;
-
- rank = (int)PyTuple_GET_SIZE(tpl);
- if(rank != rank_in) {
- PyErr_SetString(PyExc_RuntimeError, "Allocated space does not match tuple length");
- goto err;
- }
-
- for(i=0; i<rank; i++){
- temp = PyTuple_GetItem(tpl, i);
- if(temp == NULL) goto err;
-
- if PyLong_Check(temp)
- dims[i] = (hsize_t)PyLong_AsLong(temp);
- else if PyInt_Check(temp)
- dims[i] = (hsize_t)PyLong_AsLong(temp);
- else if PyFloat_Check(temp)
- dims[i] = (hsize_t)PyFloat_AsDouble(temp);
- else
- goto err;
- }
-
- return 0;
-
- err:
- if(!PyErr_Occurred()){
- PyErr_SetString(PyExc_ValueError, "Illegal argument (must be a tuple of numbers).");
- }
- return -1;
-}
/* The functions
diff --git a/h5py/utils_low.h b/h5py/utils_low.h
index 72c3108..c3b68a2 100644
--- a/h5py/utils_low.h
+++ b/h5py/utils_low.h
@@ -27,7 +27,6 @@
hid_t create_ieee_complex64(const char byteorder, const char* real_name, const char* img_name);
hid_t create_ieee_complex128(const char byteorder, const char* real_name, const char* img_name);
-int convert_tuple(PyObject* tpl, hsize_t *dims, hsize_t rank);
PyObject* convert_dims(hsize_t* dims, hsize_t rank);
int check_numpy_read(PyArrayObject* arr, hid_t space_id);
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-science/packages/h5py.git
More information about the debian-science-commits
mailing list