[Pkg-ofed-commits] r295 - branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current
Guy Coates
gmpc-guest at alioth.debian.org
Fri May 29 02:22:31 UTC 2009
Author: gmpc-guest
Date: 2009-05-29 14:22:31 +0000 (Fri, 29 May 2009)
New Revision: 295
Modified:
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/BUILD_ID
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MLNX_EN_README.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MPI_README.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_Installation_Guide.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/PERF_TEST_README.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/README.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/cxgb3_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ehca_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ib-bonding.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ibutils_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipath_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipoib_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mlx4_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mstflint_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mthca_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mvapich_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nes_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nfs-rdma.release-notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/open_mpi_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/opensm_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/sdp_release_notes.txt
branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/uDAPL_release_notes.txt
Log:
[svn-upgrade] Integrating new upstream version, ofed-docs (1.4.1)
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/BUILD_ID
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/BUILD_ID 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/BUILD_ID 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,4 +1,4 @@
-OFED-1.4
+OFED-1.4.1
libibverbs:
git://git.openfabrics.org/ofed_1_4/libibverbs.git ofed_1_4
commit b00dc7d2f79e0660ac40160607c9c4937a895433
@@ -7,63 +7,63 @@
commit be5eef3895eb7864db6395b885a19f770fde7234
libmlx4:
git://git.openfabrics.org/ofed_1_4/libmlx4.git ofed_1_4
-commit bd28f5307c3782b41cf6dfcbb6714df03c9f7025
+commit d5e5026e2bd3bbd7648199a48c4245daf313aa48
libehca:
git://git.openfabrics.org/ofed_1_4/libehca.git ofed_1_4
-commit e0c2d7e8ee2aa5dd3f3511270521fb0c206167c6
+commit 0249815e9b6f134f33546da6fa2e84e1185eea6d
libipathverbs:
git://git.openfabrics.org/~ralphc/libipathverbs ofed_1_4
-commit 65e5701dbe7b511f796cb0026b0cd51831a62318
+commit 337df3c1cbe43c3e9cb58e7f6e91f44603dd23fb
libcxgb3:
git://git.openfabrics.org/~swise/libcxgb3.git ofed_1_4
commit f685c8fe7e77e64614d825e563dd9f02a0b1ae16
libnes:
git://git.openfabrics.org/~glenn/libnes.git master
-commit 07fb9dfbbb36b28b5ea6caa14a1a5e215386b3e8
+commit 379cccb4484f39b99c974eb6910d3a0407c0bbd1
libibcm:
git://git.openfabrics.org/~shefty/libibcm.git master
commit 7fb57e005b3eae2feb83b3fd369aeba700a5bcf8
librdmacm:
git://git.openfabrics.org/~shefty/librdmacm.git master
-commit e0b1ece1dc0518b2a5232872e0c48d3e2e354e47
+commit 62c2bddeaf5275425e1a7e3add59c3913ccdb4e9
libsdp:
git://git.openfabrics.org/ofed_1_4/libsdp.git ofed_1_4
-commit 02404fb0266082f5b64412c3c25a71cb9d39442d
+commit b1eaecb7806d60922b2fe7a2592cea4ae56cc2ab
sdpnetstat:
git://git.openfabrics.org/~amirv/sdpnetstat.git ofed_1_4
-commit 75a033a9512127449f141411b0b7516f72351f95
+commit 798e44f6d5ff8b15b2a86bc36768bd2ad473a6d7
srptools:
-git://git.openfabrics.org/ofed_1_3/srptools.git ofed_1_3
-commit d3025d0771317584e51490a419a79ab55650ebc9
+git://git.openfabrics.org/~ishai/srptools.git master
+commit ce1f64c8dd63c93d56c1cc5fbcdaaadd4f74a1e3
perftest:
git://git.openfabrics.org/~orenmeron/perftest.git master
-commit ca629627c7a26005a1a4c8775cc01f483524f1c4
+commit 1cd38e844dc50d670b48200bcda91937df5f5a92
qlvnictools:
git://git.openfabrics.org/~ramachandrak/qlvnictools.git ofed_1_4
-commit 1dc6e51a728cbfbdd2018260602b8bebde618da9
+commit 4ce9789273896d0e67430c330eb3703405b59951
tvflash:
git://git.openfabrics.org/ofed_1_4/tvflash.git ofed_1_4
commit e1b50b3b8af52b0bc55b2825bb4d6ce699d5c43b
mstflint:
git://git.openfabrics.org/~orenk/mstflint.git master
-commit 9ddeea464e946cd425e05b0d1fdd9ec003fca824
+commit 3352f8997591c6955430b3e68adba33e80a974e3
qperf:
git://git.openfabrics.org/~johann/qperf.git/.git master
-commit bee05d35b09b0349cf4734ae43fc9c2e970ada8c
+commit 18e1c1e8af96cd8bcacced3c4c2a4fd90f880792
ibutils:
-git://git.openfabrics.org/~orenk/ibutils.git master
-commit 6516d16e815c68fa405562ea773b0c5215c1b70c
+git://git.openfabrics.org/~kliteyn/ibutils.git ofed_1_4
+commit 9d4bfc3ba19875dfa4583dfaef6f0f579bb013bb
ibsim:
-git://git.openfabrics.org/~sashak/ibsim.git master
+git://git.openfabrics.org/ofed_1_4/ibsim.git ofed_1_4
commit a76132ae36dde8302552d896e35bd29608ac9524
-ofa_kernel-1.4:
+ofa_kernel-1.4.1:
Git:
git://git.openfabrics.org/ofed_1_4/linux-2.6.git ofed_kernel
-commit 88ab7955605c5e769e760f6bec980e0c2e72aa5c
+commit 868661b127c355c64066a796460a7380a722dd84
# MPI
-mvapich-1.1.0-3143.src.rpm
+mvapich-1.1.0-3355.src.rpm
mvapich2-1.2p1-1.src.rpm
-openmpi-1.2.8-1.src.rpm
+openmpi-1.3.2-1.src.rpm
mpitests-3.1-891.src.rpm
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MLNX_EN_README.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MLNX_EN_README.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MLNX_EN_README.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,8 +1,8 @@
===============================================================================
MLNX_EN driver for Mellanox Adapter Cards with 10GigE Support
- README for OFED 1.4
+ README for MLNX_OFED 1.4
- December 2008
+ March 2009
===============================================================================
Contents:
@@ -29,6 +29,7 @@
- HW Tx/Rx checksum calculation
- Large Send Offload (i.e., TCP Segmentation Offload)
- Large Receive Offload
+- IP reassembly offload for fragmented IP packets
- Multi-core NAPI support
- VLAN Tx/Rx acceleration (HW VLAN stripping/insertion)
- HW VLAN filtering
@@ -61,7 +62,7 @@
#> ethtool -i eth2
driver: mlx4_en (MT_0BD0110004)
version: 1.4.0 (Dec 2008)
- firmware-version: 2.5.952
+ firmware-version: 2.6.0
bus-info: 0000:0e:00.0
- To query stateless offload status run:
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MPI_README.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MPI_README.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/MPI_README.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,7 +1,7 @@
Open Fabrics Enterprise Distribution (OFED)
- MPI in OFED 1.4.0 README
+ MPI in OFED 1.4.1 README
- December 2008
+ May 2009
===============================================================================
@@ -17,8 +17,8 @@
1. Overview
===============================================================================
Three MPI stacks are included in this release of OFED:
-- MVAPICH 1.1.0-3143
-- Open MPI 1.2.8
+- MVAPICH 1.1.0-3355
+- Open MPI 1.3.2
- MVAPICH2 1.2p1
Setup, compilation and run information of MVAPICH, Open MPI and MVAPICH2 is
@@ -194,7 +194,7 @@
===============================================================================
Open MPI is a next-generation MPI implementation from the Open MPI
-Project (http://www.open-mpi.org/). Version 1.2.8 of Open MPI is
+Project (http://www.open-mpi.org/). Version 1.3.2 of Open MPI is
included in this release, which is also available directly from the
main Open MPI web site.
@@ -208,8 +208,8 @@
than one compiler can be selected simultaneously, if desired.
Users should check the main Open MPI web site for additional
-documentation and support. (Note: The FAQ file considers
-InfiniBand tuning among other issues.)
+documentation and support. (Note: The FAQ file considers OpenFabrics
+tuning among other issues.)
3.1 Setting up for Open MPI
---------------------------
@@ -338,34 +338,28 @@
applications (see the Open MPI web site / FAQ for more information:
http://www.open-mpi.org/faq/).
-It is worth noting that the "mpi_leave_pinned" run-time tunable
-parameter is usually *very* good for running benchmarks, but can
-actually be detrimental to real-world MPI applications -- and is
-therefore disabled by default. When running the benchmarks listed
-below, it is advistable enable the "mpi_leave_pinned" option in order
-to see maximum performance (*).
+ - <N> is an integer indicating how many MPI processes to run (e.g., 2)
+ - <HOSTFILE> is the filename of a hostfile, as described above
Example 1: Running the OSU bandwidth:
- > cd /usr/mpi/gcc/openmpi-1.2.8/tests/osu_benchmarks-3.0
- > mpirun -np <N> --mca mpi_leave_pinned 1 -hostfile <HOSTFILE> osu_bw
+ > cd /usr/mpi/gcc/openmpi-1.3.2/tests/osu_benchmarks-3.0
+ > mpirun -np <N> -hostfile <HOSTFILE> osu_bw
Example 2: Running the Intel MPI Benchmark benchmarks:
- > cd /usr/mpi/gcc/openmpi-1.2.8/tests/IMB-3.1
- > mpirun -np <N> --mca mpi_leave_pinned 1 -hostfile <HOSTFILE> IMB-MPI1
+ > cd /usr/mpi/gcc/openmpi-1.3.2/tests/IMB-3.1
+ > mpirun -np <N> -hostfile <HOSTFILE> IMB-MPI1
+ --> Note that the version of IMB-EXT that ships in this version of
+ OFED contains a bug that will cause it to immediately error
+ out when run with Open MPI.
+
Example 3: Running the Presta benchmarks:
- > cd /usr/mpi/gcc/openmpi-1.2.8/tests/presta-1.4.0
- > mpirun -np <N> --mca mpi_leave_pinned 1 -hostfile <HOSTFILE> com -o 100
+ > cd /usr/mpi/gcc/openmpi-1.3.2/tests/presta-1.4.0
+ > mpirun -np <N> -hostfile <HOSTFILE> com -o 100
-(*) The "mpi_leave_pinned" option can increase bandwidth and decrease
- latency for applications that repeatedly send and/or receive from
- the same buffers. If your application does not repeatedly
- send/receive from the same buffers, mpi_leave_pinned will likely
- have little effect on your performance.
-
3.5 More Open MPI Information
-----------------------------
Much, much more information is available about using and tuning Open
@@ -381,8 +375,6 @@
http://www.open-mpi.org/community/help/
-
-
===============================================================================
4. MVAPICH2 MPI
===============================================================================
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_Installation_Guide.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_Installation_Guide.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_Installation_Guide.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,8 +1,8 @@
Open Fabrics Enterprise Distribution (OFED)
- Version 1.4
+ Version 1.4.1
Installation Guide
- December 2008
+ May 2009
==============================================================================
Table of contents
@@ -25,7 +25,7 @@
1. Overview
==============================================================================
-This is the OpenFabrics Enterprise Distribution (OFED) version 1.4
+This is the OpenFabrics Enterprise Distribution (OFED) version 1.4.1
software package supporting InfiniBand and iWARP fabrics. It is composed
of several software modules intended for use on a computer cluster
constructed as an InfiniBand subnet or an iWARP network.
@@ -42,7 +42,7 @@
2) When installing OFED on an entire [homogeneous] cluster, a common
strategy is to install the software on one of the cluster nodes
(perhaps on a shared file system such as NFS). The resulting RPMs,
- created under OFED-1.4/RPMS directory, can then be installed on all
+ created under OFED-1.4.1/RPMS directory, can then be installed on all
nodes in the cluster using any cluster-aware tools (such as pdsh).
==============================================================================
@@ -237,7 +237,7 @@
the Openfabrics stack. (Note that these includes, if needed, are
"backported" to your kernel).
o The raw package (un-backported) source files are placed under
- <prefix>/src/ofa_kernel-1.4
+ <prefix>/src/ofa_kernel-1.4.1
o The script "openibd" is installed under /etc/init.d/. This script can
be used to load and unload the software stack.
o The directory /etc/infiniband is created with the files "info" and
@@ -401,8 +401,8 @@
OFED documentation is located in the ofed-docs RPM. After
installation the documents are located under the directory:
-/usr/share/doc/ofed-docs-1.4 for RedHat
-/usr/share/doc/packages/ofed-docs-1.4 for SuSE
+/usr/share/doc/ofed-docs-1.4.1 for RedHat
+/usr/share/doc/packages/ofed-docs-1.4.1 for SuSE
Document list:
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/OFED_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,7 +1,7 @@
Open Fabrics Enterprise Distribution (OFED)
- Version 1.4
- Release Notes
- December 2008
+ Version 1.4.1
+ Release Notes
+ May 2009
===============================================================================
@@ -15,6 +15,7 @@
- Third party Test Packages
- OFED sources
2. Main Changes from OFED 1.3
+3. Main Changes from OFED 1.4
4. Known Issues
@@ -30,7 +31,7 @@
all of its nodes to this new version.
-1.1 OFED 1.4 Contents
+1.1 OFED 1.4.1 Contents
-----------------------
The OFED package contains the following components:
- OpenFabrics core and ULPs:
@@ -57,7 +58,7 @@
Notes:
1. iSER target is of Beta quality.
-2. NFS-RDMA is at technology preview state, thus it is not installed by default.
+2. NFS-RDMA is at Beta, thus it is not installed by default.
2. All other OFED components are of production quality.
3. See release notes for each package in the docs directory.
4. Any Topspin copyright belongs to Cisco Systems, Inc.
@@ -78,10 +79,12 @@
- RedHat EL5: 2.6.18-8.el5
- RedHat EL5 up1: 2.6.18-53.el5
- RedHat EL5 up2: 2.6.18-92.el5
+ - RedHat EL5 up3: 2.6.18-128.el5
- Fedora C9: 2.6.25-14.fc9 *
- SLES10: 2.6.16.21-0.8-smp
- SLES10 SP1: 2.6.16.46-0.12-smp
- SLES10 SP2: 2.6.16.60-0.21-smp
+ - SLES11: 2.6.27.19-5-default
- OpenSuSE 10.3: 2.6.22.5-31 *
- OEL 4 up7 2.6.9-78.ELsmp
- OEL 5 up2 2.6.18-92.el5
@@ -106,9 +109,10 @@
http://www.mellanox.com/content/pages.php?pg=firmware_download
o Qlogic HCAs:
- - QHT6040 (PathScale InfiniPath HT-460)
- - QHT6140 (PathScale InfiniPath HT-465)
- - QLE6140 (PathScale InfiniPath PE-880)
+ - QHT7140 QLogic InfiniPath SDR HTX HCA
+ - QLE7140 QLogic InfiniPath SDR PCIe HCA
+ - QLE7240 QLogic InfiniPath DDR x8 PCIe HCA
+ - QLE7280 QLogic IniniPath DDR x16 PCIe HCA
o IBM HCAs:
- GX Dual-port SDR 4x IB HCA
@@ -133,6 +137,7 @@
- Qlogic
- Flextronics
- Sun
+ - Mellanox
1.5 Third Party Packages
------------------------
@@ -232,8 +237,26 @@
- Updated IMB 3.1
===============================================================================
-3. Known Issues
+3. Main Changes from OFED 1.4
===============================================================================
+- Added support for RHEL 5.3 and SLES11
+- NFS/RDMA: In beta qaulity with backports for RHEL 5.2, 5.3 and SLES 10 SP2
+- Updated MPI packages:
+ MVAPICH 1.1.0-3355
+ Open MPI 1.3.2
+- Updated bonding package: ib-bonding-0.9.0-40
+- Updated DAPL: compat-dapl-1.2.14 and dapl-2.0.19
+- Updated opensm version to include critical bug fixes
+- Fixed RDS iWARP support
+- Low level drivers updated: ehca, mlx4, cxgb3, nes, ipath, mthca
+- Added a module parameter to control number of MTTs per segment in Mellanox
+ HCAs (mlx4 & mthca)
+- mstflint update
+- Bug fixes - see each component release notes for details
+
+===============================================================================
+4. Known Issues
+===============================================================================
The following is a list of general limitations and known issues of the various
components of the OFED 1.4 release.
@@ -260,6 +283,14 @@
echo "%__arch_install_post %{nil}" >> /root/.rpmmacros
7. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that
is that these utilities support devices of type Ethernet only.
+8. "openibd stop" can sometime fail with the error:
+ Unloading ib_cm [FAILED]
+ ERROR: Module ib_cm is in use by ib_ipoib
+ Workaround: run "openibd stop" again.
+9. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even
+ if IPoIB is set to connected mode). This is because there is a bug in older
+ kernels which causes a kernel panic.
+10. On SLES11 in case that uninstall is failing, need to look at the error log
+ and remove the RPMs manually using 'rpm -e <rpms list>'
-
Note: See the release notes of each component for additional issues.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/PERF_TEST_README.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/PERF_TEST_README.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/PERF_TEST_README.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -90,25 +90,6 @@
Client: ./<test name> <options> <server IP address>
o <server address> is IPv4 or IPv6 address. You can use the IPoIB
-diags_release_notes.txt
-mpi-selector_release_notes.txt
-rdma_cm_release_notes.txt
-MSTFLINT_README.txt
-open_mpi_release_notes.txt RDS_README.txt
-ib-bonding.txt
-mthca_release_notes.txt
-opensm_release_notes.txt
-rds_release_notes.txt
-ibutils_release_notes.txt*
-mvapich_release_notes.txt
-PERF_TEST_README.txt
-sdp_release_notes.txt
-ipoib_release_notes.txt
-srp_release_notes.txt
-QoS_in_OFED.txt
-SRPT_README.txt
-mlx4_release_notes.txt
-QoS_management_in_OpenSM.
address if IPoIB is configured.
o --help lists the available <options>
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/README.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/README.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/README.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -188,8 +188,8 @@
simultaneously -- they will not conflict with each other.
Three MPI stacks are included in this release of OFED:
-- MVAPICH 1.1.0
-- Open MPI 1.2.8
+- MVAPICH 1.1.0-3355
+- Open MPI 1.3.2
- MVAPICH2 1.2p1
OFED also includes 4 basic tests that can be run against each MPI
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/cxgb3_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/cxgb3_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/cxgb3_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,44 +1,22 @@
Open Fabrics Enterprise Distribution (OFED)
CHELSIO T3 RNIC RELEASE NOTES
- December 2008
+ May 2009
The iw_cxgb3 and cxgb3 modules provide RDMA and NIC support for the
-Chelsio S310/320 and R310/320 series adapters. Make sure you choose the
-'cxgb3' and 'libcxgb3' options when generating your ofed-1.4 rpms.
+Chelsio S series adapters. Make sure you choose the 'cxgb3' and
+'libcxgb3' options when generating your ofed-1.4.1 rpms.
============================================
-New for ofed-1.4
+New for ofed-1.4.1
============================================
-- 7.0 Firmware support. See below for more information on updating
+- NFSRDMA support.
+
+- 7.4 Firmware support. See below for more information on updating
your RNIC to the latest firmware.
-- Memory Managment Extensions including:
- - Fast register memory regions
- - Invalidate local memory region work request
- - Zero stag support via the local DMA lkey field
- - Read with invalidate local stag work request
-
-- RDS bcopy mode enabled for iWARP devices
-
============================================
-Recent Enhancements
-============================================
-
-- Various MPI libraries are enabled via a new iw_cxgb3 module option
-called peer2peer. When loading iw_cxgb3, set peer2peer=1 to enable Intel
-MPI version 3.1.038, HP MPI version 2.02.05.01, OpenMPI (will be released
-with OpenMPI-1.3), and Scali MPI (will be available in version 3.13.7).
-This option must be set on all systems in your cluster. See more info
-below on running these MPIs. NOTE: None of these MPIs are included in
-the ofed-1.4 release. Contact the specific vendors for obtaining the
-MPI code. Open MPI can be pulled from www.open-mpi.org.
-
-- Large memory registration. User applications can now register > 30MB
-memory regions.
-
-============================================
Enabling Various MPIs
============================================
@@ -275,3 +253,10 @@
server: ib_rdma_lat -c -I 64
client: ib_rdma_lat -c -I 64 server_ip_addr
+
+5) If you're running NFSRDMA over Chelsio's T3 RNIC and your cients are
+using a 64KB page size (like PPC64 and IA64 systems) and your server is
+using a 4KB page size (like i386 and X86_64), then you need to mount the
+server using rsize=32768,wsize=32768 to avoid overrunning the Chelsio
+RNIC fast register limits. This is a known firmware limitation in the
+Chelsio RNIC.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ehca_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ehca_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ehca_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,8 +1,8 @@
Open Fabrics Enterprise Distribution (OFED)
- ehca in OFED 1.4 Release Notes
+ ehca in OFED 1.4.1 Release Notes
- December 2008
+ May 2009
Overview
@@ -38,8 +38,12 @@
New Features
------------
-- Flush error completion support
+- none
+Fixed Bugs ofed-1.4.1
+---------------------
+- none
+
Fixed Bugs ofed-1.4
---------------------
- Reject send work requests only for RESET, INIT and RTR state
@@ -77,8 +81,9 @@
Available backports
-------------------
-- RedHat EL5 up1: 2.6.18-53.ELsmp
- RedHat EL5 up2: 2.6.18-92.ELsmp
+- RedHat EL5 up3: 2.6.18-128.ELsmp
+- SLES11: 2.6.27.19-5.1-smp
- SLES10SP1: 2.6.16-53-0.16-smp
- SLES10SP2: 2.6.16-60
- kernel.org: 2.6.24-27
@@ -93,4 +98,6 @@
loading the ehca device driver.
3. Dynamic memory operations are not supported with ehca
-
+
+4. Allocating a large number of queue pairs might be time consuming. This will
+be fixed in next OFED release.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ib-bonding.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ib-bonding.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ib-bonding.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -15,6 +15,8 @@
-------------------------------------------------------------------------------
ib-bonding is a High Availability solution for IPoIB interfaces. It is based
on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB.
+However, the support for for IPoIB interfaces is only for the active-backup
+mode, other modes should not be used.
ib-bonding package contains a bonding driver and a utility called ib-bond to
manage and control the driver operation.
@@ -70,6 +72,7 @@
-----------------------------------------------------------------
* In the master (bond) interface script add the line:
TYPE=Bonding
+MTU=<according to the slave's MTU>
Exmaple: for bond0 (master) the file is named /etc/sysconfig/network-scripts/ifcfg-bond0
with the following text in the file:
@@ -83,11 +86,19 @@
BOOTPROTO=none
USERCTL=no
TYPE=Bonding
+MTU=65520
+Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
+mode and are configured with the same value. For IPoIB slaves that work in
+datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
+all (and letting it to be set to the default value), performance of the
+interface might decrease.
+
* In the slave (ib) interface script put the following lines:
SLAVE=yes
MASTER=<bond name>
TYPE=InfiniBand
+PRIMARY=<yes|no>
Example: the script for ib0 (slave) would be named /etc/sysconfig/network-scripts/ifcfg-ib0
with the following text in the file:
@@ -99,7 +110,11 @@
SLAVE=yes
BOOTPROTO=none
TYPE=InfiniBand
+PRIMARY=yes
+Note: If the slave interface is not primary then the line PRIMARY= is not
+required and can be omitted.
+
After the configuration is saved, restart the network service by running:
/etc/init.d/network restart
@@ -108,6 +123,9 @@
Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4)
with the following changes:
* In the bondX (master) script - the line TYPE=Bonding is not needed.
+* In the bondX (master) script - you may add to the configuration more options
+with the following line
+BONDING_OPTS=" primary=ib0 updelay=0 downdelay=0"
* in the ibX (slave) script - the line TYPE=InfiniBand necessary when using
bonding over devices configured with partitions ( p_key)
Example:
@@ -137,6 +155,7 @@
BONDING_MODULE_OPTS="mode=active-backup miimon=<value>"
BONDING_SLAVE0=slave0
BONDING_SLAVE1=slave1
+MTU=<according to the slave's MTU>
Exmaple: for bond0 (master) the file is named /etc/sysconfig/network/ifcfg-bond0
with the following text in the file:
@@ -149,10 +168,21 @@
REMOTE_IPADDR=""
STARTMODE="onboot"
BONDING_MASTER="yes"
-BONDING_MODULE_OPTS="mode=active-backup miimon=100"
+BONDING_MODULE_OPTS="mode=active-backup miimon=100 primary=ib0 updelay=0 downdelay=0"
BONDING_SLAVE0=ib0
BONDING_SLAVE1=ib1
+MTU=65520
+Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
+mode and are configured with the same value. For IPoIB slaves that work in
+datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
+all (and letting it to be set to the default value), performance of the
+interface might decrease.
+
+Note: primary, downdelay and updelay is an optional bonding interface
+configuration. You may choose to use them, change them or delete them from the
+configuration script (by editing the line that starts with BONDING_OPTS)
+
* The slave (ib) interace script should look like this:
BOOTPROTO='none'
@@ -167,18 +197,36 @@
It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the
same bonding master. It is possible however that a bonding master of Ethernet
slaves and a bonding master of IPoIB slaves will co-exist in one machne.
-To configure Ethernet slaves under a bonding master use the same instructions
-as for IPoIB slaves (according to the OS) with one exception. When working
-under Redhat-AS4 do the following when configuring a bonding master with
-Ethernet slaves
+To configure Ethernet slaves under a bonding master use the following
+instructios (depending on the OS)
+* Under Redhat-AS4
+
+Use the same instructions as for IPoIB slaves with the following exceptions
+
- In the master configuration file add the line
SLAVEDEV=1
- In the slave configuration file leave the line
TYPE=InfiniBand
+- For Ethernet, it is possible to set parameters of the bonding module in /etc/modprobe.conf
+with the following line for example
+options bonding miimon=100 mode=1 primary=eth0
+Note that alias names for the bonding module (such as bond0) may not work.
-When using both type of bonding under SLES-10, it is neccessary to update the
+* Under Redhat-AS5
+
+No special instructions are required.
+
+* Under SLES10
+
+When using both type of bonding under, it is neccessary to update the
MANDATORY_DEVICES environment variable in /etc/sysconfig/network/config with the names
of the InfiniBand devices ( ib0, ib1, etc. ). Otherwise, bonding devices will be created
before InfiniBand devices at boot time.
+Note: If there is more than one Ethernet NIC installed then there might be a
+race for the interface name eth0, eth1 etc. This may lead to unexpected
+relation between logical and physical devices which may lead to wrong bonding
+configuration. This issue may be solved by binding a logical device name (e.g.
+eth0) to a physical (hardware) device by specifying the MAC address in the
+ethN configuration file.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ibutils_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ibutils_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ibutils_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -2,8 +2,8 @@
--------------------------------------------
*******************************************************************************
-RELEASE: OFED 1.4
-DATE: Dec 2008
+RELEASE: OFED 1.4.1
+DATE: May 2009
===============================================================================
Table of Contents
@@ -11,6 +11,7 @@
1. Overview
2. New features
3. Major Bugs Fixed
+3.1 Changes since OFED 1.4
4. Known Issues
===============================================================================
@@ -74,6 +75,25 @@
* ibdiagnet: Do not query port counters when local port is in INIT state.
===============================================================================
+3.1 Changes since OFED 1.4
+===============================================================================
+
+* PM csv files format fix
+
+* Fixed generating and parsing IBNL files
+
+* Fixed CC packet format to meet IBTA approved format
+
+* Set of changes to sync with OpenSM changes
+
+* Regenerated wrappers - fixed compilation errors on some distros
+
+* Fixed printing SM and mcast info in ibdiagnet
+
+* Other minor fixes/improvements
+
+
+===============================================================================
4. Known Issues
===============================================================================
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipath_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipath_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipath_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,8 +1,8 @@
- Open Fabrics Enterprise Distribution (OFED)
- ipath in OFED 1.4 Release Notes
-
- December 2008
+ Open Fabrics Enterprise Distribution (OFED)
+ ipath in OFED 1.4.1 Release Notes
+ May 2009
+
======================================================================
Table of Contents
======================================================================
@@ -20,65 +20,16 @@
======================================================================
2. Fixed Bugs and Enhancements
======================================================================
-2.1 (Bug 1283) Intel MPI fails on Qlogc HCA
+2.1 (Bug 1369) No results when running Open MPI bandwidth with msg size
+bigger than 2200 with Qlogic HCA
+This was fixed by a change submitted to OpenMPI 1.3, which is part of OFED
+1.4.1.
-Added memcpy_cachebypass support to IB/ipath
-----------------------------------
-2.2 (Bug 1117) ib_ipath module hangs on unload.
-
-This fix handles the case where posting a send is requested when the
-link is down.
-----------------------------------
-2.3 (Bug 1093) Compile fails because definition of valid_dma_direction
-is missing from file ipath_dma.c
-
-Marked as RESOLVED INVALID. XEN is not a supported kernel. It defines
-its own dma-mapping.h file and any driver which uses DMA probably
-needs to be modified to work specifically within XEN.
-
======================================================================
3. Known Issues
======================================================================
-3.1 MSI (Message Signalled Interrupt) support required with
- QLogic InfiniPath HCAs
-The QLogic InfiniPath adapter requires MSI (Message Signalled
-Interrupts) to be enabled in the kernel.
-
-If the InfiniPath driver is intalled on a machine without
-CONFIG_PCI_MSI=y configured, a warning similar to this will appear in
-dmesg at boot:
-
-[root at sqa-00 ~]# dmesg | grep ipath
- ib_ipath 0000:01:00.0: infinipath0: QLogic PCIE device 0x10 cannot
- work if CONFIG_PCI_MSI is not enabled
-----------------------------------
-3.2 (Bug 1369) No results when running Open MPI bandwidth with msg size
-bigger than 2200 with Qlogic HCA
-
-OpenMPI defaults to the IB MTU size specified in a configuration file
-found within the OpenMPI install directory at:
-
- share/openmpi/mca-btl-openib-hca-params.ini
-
-This file contains a mapping from adapter cards (identified by vendor
-ID and adapter part ID) to MTU size. Note that this is a static
-mapping and the available MTU on a particular fabric might not be the
-same as the maximum supported by the adapter card. This could happen
-due to switch MTU limitations, or use of different adapters. In this
-case the value should be set to the largest value that works for
-all HCAs and switches in the IB fabric (that will be part of the
-OpenMPI job). If the value specified is too large, the job will
-typically hang at run-time without a diagnostic.
-
-Current workaround: The two ways to work around this would be to
-either set your switch to run with a 4K MTU, or to edit the
-mca-btl-openib-hca-params.ini file and hardcode the MTU for
-InfiniPath (TrueScale) adapters to 2K.
-
-This will be resolved by joint effort between QLogic and OpenMPI.
-----------------------------------
-3.3 (Bug 1242) Kernel panic while running mpi2007 against
+3.1 (Bug 1242) Kernel panic while running mpi2007 against
ofed1.4 -- ib_ipath: ipath_sdma_verbs_send
Found while running mpi2007 over OpenMPI on stock OFED1.4 RC1 RHEL4.X
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipoib_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipoib_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/ipoib_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,7 +1,7 @@
Open Fabrics Enterprise Distribution (OFED)
- IPoIB in OFED 1.4 Release Notes
+ IPoIB in OFED 1.4.1 Release Notes
- December 2008
+ May 2009
===============================================================================
@@ -14,7 +14,8 @@
5. The ib-bonding driver
6. Bug Fixes and Enhancements Since OFED 1.3
7. Bug Fixes and Enhancements Since OFED 1.3.1
-8. Performance tuning
+8. Bug Fixes and Enhancements Since OFED 1.4
+9. Performance tuning
===============================================================================
1. Overview
@@ -124,13 +125,7 @@
and SuSE: /etc/sysconfig/network/), the option IPOIB_LOAD=no in openib.conf
does not prevent the loading of IPoIB on boot.
-6. On RedHat EL 4 up4, the IPOIB implementation is not spec-compliant:
- - ipoib multicast does not work
- - ipoib cannot inter-operate between RHEL4U4 and other hosts. This is due to
- missing code in the kernel which was available in U3 and U5 but removed in
- U4. As a workaround, upgrade to RHEL4U5.
-
-7. If IPoIB connected mode is enabled, it uses a large MTU for connected mode
+6. If IPoIB connected mode is enabled, it uses a large MTU for connected mode
messages and a small MTU for datagram (in particular, multicast) messages,
and relies on path MTU discovery to adjust MTU appropriately. Packets sent
in the window before MTU discovery automatically reduces the MTU for a
@@ -141,6 +136,13 @@
To warn about this, a message is produced in the system log each time MTU is
set to a value higher than 2K.
+7. IPoIB IPv6 support is broken for between systems with kernels < 2.6.12 and
+ kernels >= 2.6.12. The reason for that is that kernel 2.6.12 puts the link
+ layer address at an offset of two bytes with respect to older kernels. This
+ causes the other host to misinterpret the hardware address resulting in failure
+ to resolve path which are based on wrong GIDs. As an example, RH 4.x and RH
+ 5.x cannot interoperate.
+
8. In connected mode, TCP latency for short messages is larger by approx. 1usec
(~5%) than in datagram mode. As a workaround, use datagram mode.
@@ -153,37 +155,95 @@
11. The IPoIB module uses a Linux implementation for Large Receive Offload
(LRO) in kernel 2.6.24 and later. These kernels require installing the
"inet_lro" module.
+
+12. ConnectX only: If you have a port configured as ETH, and are running IPoIB
+ in connected mode -- and then change the port type to IB, the IPoIB mode
+ changes to datagram mode.
+
+13. When working with ISCSI, you must disable LRO (even if you are working in
+ connected mode). This is because there is a bug in older kernels which causes
+ a kernel panic.
+
+14. IPoIB datagram mode initial packet loss (bug #1287): When the datagram test
+ gets to packet size 8192 and larger it always loose the first packet in the
+ sequence.
+ Workaround: Increase the number of pending skb's before a neighbor is
+ resolved (default is 3). This value can be changed with:
+ sysctl net.ipv4.neigh.ib0.unres_qlen.
+15. IPoIB multicast support is broken in RH4.x kernels. This is because
+ ndisc_mc_map() does not handle IPOIB hardware addresses.
+
===============================================================================
-4. DHCP Support of IPoIB
+4. IPoIB Configuration Based on DHCP
===============================================================================
-Note: To use DHCP the user must apply a special patch (see "DHCP Notes" below).
-DHCP Supported Operating Systems
---------------------------------
-1. SLES 10
-2. RHEL 5
-3. All kernels from 2.6.14 and up
+Setting an IPoIB interface configuration based on DHCP (v3.1.2 which is available
+via www.isc.org) is performed similarly to the configuration of Ethernet
+interfaces. In other words, you need to make sure that IPoIB configuration files
+include the following line:
+ For RedHat:
+ BOOTPROTO=dhcp
+ For SLES:
+ BOOTPROTO=dchp
+Note: If IPoIB configuration files are included, ifcfg-ib<n> files will be
+installed under:
+/etc/sysconfig/network-scripts/ on a RedHat machine
+/etc/sysconfig/network/ on a SuSE machine
-DHCP Unsupported Operating Systems
-----------------------------------
-RedHat EL 4 distributions are supported.
+Note: A patch for DHCP is required for supporting IPoIB. The patch file for
+DHCP v3.1.2, dhcp.patch, is available under the docs/ directory.
+Standard DHCP fields holding MAC addresses are not large enough to contain an
+IPoIB hardware address. To overcome this problem, DHCP over InfiniBand messages
+convey a client identifier field used to identify the DHCP session. This client
+identifier field can be used to associate an IP address with a client identifier
+value, such that the DHCP server will grant the same IP address to any client
+that conveys this client identifier.
-DHCP Notes
-----------
-1. It may be required to run over different UDP ports than the well known ports
- (67 and 68). Free port numbers greater than 0x8000 must be chosen. To
- specify a server or a client port number, use the option -p <port number>.
- The client's port number must be the chosen server's port number plus one.
+Note: Refer to the DHCP documentation for more details how to make this
+association.
-2. For IPoIB to use DHCP, you must patch ISC's DHCP. The patch file can be
- found under OFED-1.3/docs/dhcp after extracting the distribution file.
- (After installation it can also be found under <prefix>/docs/dhcp.) The
- patch should be applied for the server and for each client. Tests were run
- on version 3.0.4 of the DHCP package.
+The length of the client identifier field is not fixed in the specification.
+4.1 DHCP Server
+In order for the DHCP server to provide configuration records for clients, an
+appropriate configuration file needs to be created. By default, the DHCP server
+looks for a configuration file called dhcpd.conf under /etc. You can either edit
+this file or create a new one and provide its full path to the DHCP server using
+the -cf flag. See a file example at docs/dhcpd.conf of this package.
+The DHCP server must run on a machine which has loaded the IPoIB module.
+To run the DHCP server from the command line, enter:
+dhcpd <IB network interface name> -d
+Example:
+host1# dhcpd ib0 -d
+
+4.2 DHCP Client (Optional)
+
+Note: A DHCP client can be used if you need to prepare a diskless machine with
+an IB driver.
+In order to use a DHCP client identifier, you need to first create a
+configuration file that defines the DHCP client identifier. Then run the DHCP
+client with this file using the following command:
+dhclient cf <client conf file> <IB network interface name>
+Example of a configuration file for the ConnectX (PCI Device ID 25418), called
+dhclient.conf:
+# The value indicates a hexadecimal number
+interface "ib1" {
+send dhcp-client-identifier 00:02:c9:03:00:00:10:39;
+}
+Example of a configuration file for InfiniHost III Ex (PCI Device ID 25218), called
+dhclient.conf:
+# The value indicates a hexadecimal number
+interface "ib1" {
+send dhcp-client-identifier 20:00:55:04:01:fe:80:00:00:00:00:00:00:00:02:c9:02:00:23:13:92;
+}
+
+In order to use the configuration file, run:
+host1# dhclient cf dhclient.conf ib1
+
+
===============================================================================
5. The ib-bonding driver
===============================================================================
@@ -210,6 +270,7 @@
Notes:
* Using /etc/infiniband/openib.conf to create a persistent configuration is
no longer supported
+* On RHEL4_U7, cannot set a slave interface as primary.
===============================================================================
@@ -243,21 +304,47 @@
- Bonding: Set default number of grat. ARP after failover to three (was one)
===============================================================================
-8. Performance tuning
+8. Bug Fixes and Enhancements Since OFED 1.4
===============================================================================
-- In IPoIB connected mode, the throughput of medium and large messages can be
- increased by setting the following TCP parameters as follows:
+- Performance tuning is enabled by default for IPOIB CM.
+- Clear IPOIB_FLAG_ADMIN_UP if ipoib_open fails
+- disable napi while cq is being drained (bugzilla #1587)
+- rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast
+ When joining IPoIB multicast group, use the same rate as in the broadcast
+ group. Otherwise, if rdma_cm creates this group before IPoIB does, it might get
+ a different rate. This will cause IPoIB to fail joining to the same group later
+ on, because IPoIB has a strict rate selection.
+- fix unprotected use of priv->broadcast in ipoib_mcast_join_task.
+- Do not join broadcast group if interface is brought down
- /sbin/sysctl -w net.ipv4.tcp_timestamps=0
- /sbin/sysctl -w net.ipv4.tcp_sack=0
- /sbin/sysctl -w net.core.netdev_max_backlog=250000
- /sbin/sysctl -w net.core.rmem_max=16777216
- /sbin/sysctl -w net.core.wmem_max=16777216
- /sbin/sysctl -w net.core.rmem_default=16777216
- /sbin/sysctl -w net.core.wmem_default=16777216
- /sbin/sysctl -w net.core.optmem_max=16777216
- /sbin/sysctl -w net.ipv4.tcp_mem="16777216 16777216 16777216"
- /sbin/sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
- /sbin/sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
+
+===============================================================================
+9. Performance tuning
+===============================================================================
+When IPoIB is configured to run in connected mode, tcp parameter tuning is
+performed at driver startup -- to improve the throughput of medium and large
+messages.
+The driver startup scripts set the following TCP parameters as follows:
+ net.ipv4.tcp_timestamps=0
+ net.ipv4.tcp_sack=0
+ net.core.netdev_max_backlog=250000
+ net.core.rmem_max=16777216
+ net.core.wmem_max=16777216
+ net.core.rmem_default=16777216
+ net.core.wmem_default=16777216
+ net.core.optmem_max=16777216
+ net.ipv4.tcp_mem="16777216 16777216 16777216"
+ net.ipv4.tcp_rmem="4096 87380 16777216"
+ net.ipv4.tcp_wmem="4096 65536 16777216"
+This tuning is effective only for connected mode. If you run in datagram mode,
+it actually reduces performance.
+
+If you change the IPoIB run mode to "datagram" while the driver is running,
+the tuned parameters do not get reset to their default values. We therefore
+recommend that you change the IPoIB mode only while the driver is down
+(by setting line "SET_IPOIB_CM=yes" to "SET_IPOIB_CM=no" in file
+/etc/infiniband/openib.conf, and then restarting the driver).
+
+
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mlx4_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mlx4_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mlx4_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,7 +1,7 @@
Open Fabrics Enterprise Distribution (OFED)
- ConnectX driver (mlx4) in OFED 1.4 Release Notes
+ ConnectX driver (mlx4) in OFED 1.4.1 Release Notes
- December 2008
+ May 2009
===============================================================================
@@ -10,8 +10,10 @@
1. Overview
2. Supported Firmware Versions
3. VPI (Virtual Process Interconnect)
-4. Infiniband new features and bug fixes
-5. Known Issues
+4. Infiniband new features and bug fixes since OFED 1.3.1
+5. Infiniband (mlx4_ib) new features and bug fixes since OFED 1.4
+6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
+7. Known Issues
===============================================================================
1. Overview
@@ -39,7 +41,7 @@
===============================================================================
- This release was tested with FW 2.6.000.
- The minimal version to use is 2.3.000.
-- To use both IB and Ethernet use FW version 2.6.0
+- To use both IB and Ethernet (VPI) use FW version 2.6.0
===============================================================================
3. VPI (Virtual Protocol Interconnect)
@@ -105,7 +107,7 @@
===============================================================================
-4. Infiniband new features and bug fixes
+4. Infiniband new features and bug fixes since OFED 1.3.1
===============================================================================
Features that are enabled with FW 2.5.0 only:
- Send with invalidate and Local invalidate send queue work requests.
@@ -129,8 +131,52 @@
===============================================================================
-5. Known Issues
+5. Infiniband new features and bug fixes since OFED 1.4
===============================================================================
+- Enable setting via module param (set_4k_mtu) 4K MTU for ConnectX ports.
+- Support optimized registration of huge pages backed memory.
+ With this optimization, the number of MTT entries used is significantly
+ lower than for regular memory, so the HCA will access registered memory with
+ fewer cache misses and improved performance.
+ For more information on this topic, please refer to Linux documentation file:
+ Documentation/vm/hugetlbpage.txt
+- Do not enable blueflame sends if write combining is not available
+- Add write combining support for for PPC64, and thus enable blueflame sends.
+- Unregister IB device before executing CLOSE_PORT.
+- Notify and exit if the kernel module used does not support XRC. This is done
+ to avoid libmlx4 compatibility problem.
+- Added a module parameter (log_mtts_per_seg) for number of MTTs per segment.
+ This enable to register more memory with the same number of segments.
+
+
+===============================================================================
+6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
+===============================================================================
+6.1 Changes and New Features
+----------------------------
+- Added Tx Multi-queue support which Improves multi-stream and bi-directional
+ TCP performance.
+- Added IP Reassembly to improve RX bandwidth for IP fragmented packets.
+- Added linear skb support which improves UDP performance.
+- Removed the following module parameters:
+ - rx/tx_ring_size
+ - rx_ring_num - number of RX rings
+ - pprx/pptx - global pause frames
+ The parameters above are controlled through the standard Ethtool interface.
+
+Bug Fixes
+---------
+- Memory leak when driver is unloaded without configuring interfaces first.
+- Setting flow control parameters for one ConnectX port through Ethtool
+ impacts the other port as well.
+- Adaptive interrupt moderation malfunctions after receiving/transmitting
+ around 7 Tera-bytes of data.
+- Firmware commands fail with bad flow messages when bringing an interface up.
+- Unexpected behavior in case of memory allocation failures.
+
+===============================================================================
+7. Known Issues
+===============================================================================
- mlx4_en driver is not supported on PPC64 and IA64
- The mlx4_en module uses a Linux implementation for Large Receive Offload
(LRO) in kernel 2.6.24 and later. These kernels require installing the
@@ -138,14 +184,33 @@
- The SQD feature is not supported:
- To load the driver on machines with 64KB default page size UAR bar must be
enlarged. 64KB page size is the default of PPC with RHEL5 and Itanium with
- 64KB page size enabled.
+ SLES 11 or when 64KB page size enabled.
Perform the following three steps:
1. Add the following line in the firmware configuration (INI) file under the
[HCA] section:
log2_uar_bar_megabytes = 5
2. Burn a modified firmware image with the changed INI file.
3. Reboot the system.
+- Ethernet in MLNX_OFED 1.4 is not supported for the following OPNs:
+ MHQH29-XTC
+ MHGH29-XTC
+ MHGH29-XSC
+ MHGH28-XTC
+ MHGH28-XTC
+ MHGH28-XSC
+ MHGH28-XSC
+ MHEH28-XTC
+ MHEH28-XSC
+ MHQH19-XTC
+ Attempting to use these cards as NICs will yield the following error
+ in /var/log/messages:
+ mlx4_core 0000:0d:00.0: command 0x9 failed: fw status = 0x8
+ <date & time> mtlx008 kernel: mlx4_en 0000:0d:00.0: Failed Initializing port
+ <date & time> mtlx008 kernel: mlx4_en 0000:0d:00.0: Failed starting port:1
+
+ As a workaround, use the MLNX_EN driver instead of OFED for Linux.
+
===============================================================================
6. mlx4 Available Parameters
@@ -158,6 +223,7 @@
options mlx4_en parameter=<value>
mlx4_core parameters:
+ set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (default 0)
msi_x: attempt to use MSI-X if nonzero (default 1)
enable_qos: Enable Quality of Service support in the HCA if > 0, (default 0)
block_loopback Block multicast loopback packets if > 0 (default: 1)
@@ -174,6 +240,7 @@
(default is 20; max is 20)
log_num_mac: log maximum number of MACs per ETH port (1-7) (int)
log_num_vlan: log maximum number of VLANs per ETH port (0-7) (int)
+ log_mtts_per_seg Log2 number of MTT entries per segment (1-5; default is 3)
use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool)
mlx4_ib parameters:
@@ -183,21 +250,8 @@
rss_xor: Use XOR hash function for RSS 0 (default is xor)
rss_mask: RSS hash type bitmask (default is 0xf)
num_lro: Number of LRO sessions per ring or disabled (0) (default is 32)
- pptx: Pause policy on TX: 0 never generate pause frames 1 generate pause
- frames according to RX buffer threshold (default is 1)
- pprx: Pause policy on RX: 0 ignore received pause frames 1 respect received
- pause frames (default is 1)
pfctx: Priority based Flow Control policy on TX[7:0].
Per priority bit mask (default is 0)
pfcrx: Priority based Flow Control policy on RX[7:0].
Per priority bit mask (default is 0)
- rx_moder_cnt: Max coalesced descriptors for Rx interrupt moderation
- rx_moder_time: Timeout following last packet for Rx interrupt moderation
- auto_moder: Enable dynamic interrupt moderation (default is 1)
- rx_ring_num1: Number or Rx rings for port 1 (0 = #cores) (default is 0)
- rx_ring_num2: Number or Rx rings for port 2 (0 = #cores) (default is 0)
- tx_ring_size1: Tx ring size for port 1 (default is 1024)
- tx_ring_size2: Tx ring size for port 2 (default is 1024)
- rx_ring_size1: Rx ring size for port 1 (default is 1024)
- rx_ring_size2: Rx ring size for port 2 (default is 1024)
- inline_thold: treshold for using inline data (default is 128)
+ inline_thold: threshold for using inline data (default is 128)
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mstflint_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mstflint_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mstflint_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -57,4 +57,11 @@
4. Known Issues
===============================================================================
+* In the very unlikely event that you get the following error message when
+ running mstflint:
+ Warning: memory access to device 0a:00.0 failed: Input/output error.
+ Warning: Fallback on IO: much slower, and unsafe if device in use.
+ *** buffer overflow detected ***: mstflint terminated
+ simply run "mst start" and then re-run mstflint.
+
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mthca_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mthca_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mthca_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,15 +1,16 @@
Open Fabrics Enterprise Distribution (OFED)
- mthca in OFED 1.4 Release Notes
+ mthca in OFED 1.4.1 Release Notes
- December 2008
+ May 2009
===============================================================================
Table of Contents
===============================================================================
1. Overview
-2. Fixed Bugs
-3. Known Issues
+2. Fixed Bugs since OFED 1.3.1
+3. Bug fixes and enhancements since OFED 1.4
+4. Known Issues
===============================================================================
1. Overview
@@ -45,6 +46,7 @@
num_udav - maximum number of UD address vectors per HCA (int)
fmr_reserved_mtts - number of memory translation table segments reserved
for FMR (int)
+ log_mtts_per_seg - log2 number of MTT entries per segment (1-5)
===============================================================================
2. Fixed Bugs
@@ -58,6 +60,14 @@
===============================================================================
+3. Bug fixes and enhancements since OFED 1.4
+===============================================================================
+- Added a module parameter (log_mtts_per_seg) for number of MTTs per segment.
+ This enable to register more memory with the same number of segments.
+- Bring INIT_HCA and other commands timeout into consistency with PRM. This
+ solve an issue when had more than 2^18 max qp's configured.
+
+===============================================================================
3. Known Issues
===============================================================================
1. A UAR size other than 8MB prevents mthca driver loading. The default UAR
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mvapich_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mvapich_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/mvapich_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,7 +1,7 @@
Open Fabrics Enterprise Distribution (OFED)
- OSU MPI MVAPICH-1.1.0, in OFED 1.4.0 Release Notes
+ OSU MPI MVAPICH-1.1.0, in OFED 1.4.r10 Release Notes
- December 2008
+ May 2009
===============================================================================
@@ -49,6 +49,8 @@
===============================================================================
4. Bug Fixes ( Compared to mvapich 1.0.0 )
===============================================================================
+- De-register stale memory regions earlier to prevent
+ excess allocations of physical memory
- Fixes for MPI_Query_thread and MPI_Is_thread_main
- Fixes for PGI compiler support
- Compilation warnings cleanup
@@ -59,6 +61,8 @@
===============================================================================
5. Known Issues
===============================================================================
+- Shared memory broadcast optimization is disabled by default.
+
- MVAPICH MPI compiled on AMD x86_64 does not work with MVAPICH MPI compiled
on Intel X86_64 (EM64t).
Workaround:
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nes_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nes_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nes_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,108 +1,301 @@
Open Fabrics Enterprise Distribution (OFED)
- Intel-NE RNIC RELEASE NOTES
- December 2008
+ NetEffect Ethernet Cluster Server Adapter Release Notes
+ May 2009
-The iw_nes and libnes modules provide RDMA and NIC support for the
-Intel-NE NE020 series of adapters.
+The iw_nes module and libnes user library provide RDMA and L2IF
+support for the NetEffect Ethernet Cluster Server Adapters.
+
============================================
-Loadable Module options
+Required Setting - RDMA Unify TCP port space
============================================
-The following options can be used when loading the iw_nes module:
+RDMA connections use the same TCP port space as the host stack. To avoid
+conflicts, set rdma_cm module option unify_tcp_port_sapce to 1 by adding
+the following to /etc/modprobe.conf:
-mpa_version = 1;
- "MPA version to be used int MPA Req/Resp (0 or 1)"
+ options rdma_cm unify_tcp_port_space=1
-disable_mpa_crc = 0;
- "Disable checking of MPA CRC"
-send_first = 0;
- "Send RDMA Message First on Active Connection"
+=======================
+Loadable Module Options
+=======================
+The following options can be used when loading the iw_nes module by modifying
+modprobe.conf file:
-nes_drv_opt = 0;
- "Driver option parameters"
+wide_ppm_offset = 0
+ Set to 1 will increase CX4 interface clock ppm offset to 300ppm.
+ Default setting 0 is 100ppm.
- NES_DRV_OPT_ENABLE_MSI 0x00000010
- NES_DRV_OPT_DUAL_LOGICAL_PORT 0x00000020
- NES_DRV_OPT_SUPRESS_OPTION_BC 0x00000040
- NES_DRV_OPT_NO_INLINE_DATA 0x00000080
- NES_DRV_OPT_DISABLE_INT_MOD 0x00000100
- NES_DRV_OPT_DISABLE_VIRT_WQ 0x00000200
- NES_DRV_OPT_DISABLE_LRO 0x00000400
+mpa_version = 1
+ MPA version to be used int MPA Req/Resp (0 or 1).
-nes_debug_level = 0;
- "Enable debug output level"
+disable_mpa_crc = 0
+ Disable checking of MPA CRC.
+send_first = 0
+ Send RDMA Message First on Active Connection.
+
+nes_drv_opt = 0x00000100
+ Following options are supported:
+
+ Enable MSI - 0x00000010
+ No Inline Data - 0x00000080
+ Disable Interrupt Moderation - 0x00000100
+ Disable Virtual Work Queue - 0x00000200
+
+nes_debug_level = 0
+ Enable debug output level.
+
wqm_quanta = 65536
- "Size of data to be transmitted at a time"
+ Set size of data to be transmitted at a time.
limit_maxrdreqsz = 0
- "Limit PCI read request size to 256 bytes"
+ Limit PCI read request size to 256 bytes.
-============================================
-Runtime Module options
-============================================
+===============
+Runtime Options
+===============
The following options can be used to alter the behavior of the iw_nes module:
+NOTE: Assuming NetEffect Ethernet Cluster Server Adapter is assigned eth2.
-tso
- ethtool -K eth2 tso on == enables tso
- ethtool -K eth2 tso off == disables tso
-
-jumbo
- ifconfig eth2 mtu 9000 == largest mtu supported
+ ifconfig eth2 mtu 9000 - largest mtu supported
-static interrupt moderation
- ethtool -C eth2 rx-usecs-irq 128
-
-dynamic interrupt moderation
- ethtool -C eth2 adaptive-rx on == enable
- ethtool -C eth2 adaptive-rx off == disable
-
-dynamic interrupt moderation
- ethtool -C eth2 rx-frames-low 12 == low watermark of rx queue
- ethtool -C eth2 rx-frames-high 255 == high watermark of rx queue
- ethtool -C eth2 rx-usecs-low 40 == smallest interrupt moderation timer
- ethtool -C eth2 rx-usecs-high 1500 == largest interrupt moderation timer
+ ethtool -K eth2 tso on - enables TSO
+ ethtool -K eth2 tso off - disables TSO
+ ethtool -C eth2 rx-usecs-irq 128 - set static interrupt moderation
-============================================
-Recommended setting
-============================================
-RDMA connections use the same TCP port space as the host stack. To avoid
-conflicts, set rdma_cm module option unify_tcp_port_sapce to 1 by adding
-the following to /etc/modprobe.conf:
+ ethtool -C eth2 adaptive-rx on - enable dynamic interrupt moderation
+ ethtool -C eth2 adaptive-rx off - disable dynamic interrupt moderation
+ ethtool -C eth2 rx-frames-low 16 - low watermark of rx queue for dynamic
+ interrupt moderation
+ ethtool -C eth2 rx-frames-high 256 - high watermark of rx queue for
+ dynamic interrupt moderation
+ ethtool -C eth2 rx-usecs-low 40 - smallest interrupt moderation timer
+ for dynamic interrupt moderation
+ ethtool -C eth2 rx-usecs-high 1000 - largest interrupt moderation timer
+ for dynamic interrupt moderation
- options rdma_cm unify_tcp_port_space=1
+===================
+uDAPL Configuration
+===================
+Rest of the document assumes the following uDAPL settings in dat.conf:
-============================================
-Known issues
-============================================
-On RHEL4 update 4, we have observed /dev/infiniband/uverbs0 does not
-always get created. This device file is used for user-mode access to
-accelerated interface. Current workaround is to change the start order
-for openibd(S05openibd) to after network(S10network). For systems that
-start at runlevel 3 do the following:
+ OpenIB-cma-nes u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" ""
+ ofa-v2-nes u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
- mv /etc/rc.d/rc3.d/S05openibd /etc/rc.d/rc3.d/S11openibd
-For runlevel 5 do:
+=======================================
+Recommended Settings for HP MPI 2.2.7
+=======================================
+Add the following to mpirun command:
- mv /etc/rc.d/rc5.d/S05openibd /etc/rc.d/rc5.d/S11openibd
+ -1sided
+Example mpirun command with uDAPL-2.0:
-Some MPIs require the node that initiated the RDMA connection to send
-the first RDMA message. Enable this feature by adding the following
-to /etc/modprobe.conf:
+ mpirun -UDAPL -prot -intra=shm
+ -e MPI_ICLIB_UDAPL=libdaplofa.so.1
+ -e MPI_HASIC_UDAPL=ofa-v2-nes
+ -1sided
+ -f /opt/hpmpi/appfile
- options iw_nes send_first=1
+Example mpirun command with uDAPL-1.2:
+ mpirun -UDAPL -prot -intra=shm
+ -e MPI_ICLIB_UDAPL=libdaplcma.so.1
+ -e MPI_HASIC_UDAPL=OpenIB-cma-nes
+ -1sided
+ -f /opt/hpmpi/appfile
-For Intel MPI, iw_nes currently does not support dynamic connection
-establishment feature. Turn it off by setting/exporting the
-I_MPI_USE_DYNAMIC_CONNECTIONS variable to 0:
- export I_MPI_USE_DYNAMIC_CONNECTIONS=0
+=======================================
+Recommended Settings for Intel MPI 3.2
+=======================================
+Add the following to mpiexec command:
+
+ -genv I_MPI_FALLBACK_DEVICE 0
+ -genv I_MPI_DEVICE rdma:OpenIB-cma-nes
+ -genv I_MPI_RENDEZVOUS_RDMA_WRITE
+
+Example mpiexec command line for uDAPL-2.0:
+
+ mpiexec -genv I_MPI_FALLBACK_DEVICE 0
+ -genv I_MPI_DEVICE rdma:ofa-v2-nes
+ -genv I_MPI_RENDEZVOUS_RDMA_WRITE
+ -ppn 1 -n 2
+ /opt/intel/impi/3.2.0.011/bin64/IMB-MPI1
+
+Example mpiexec command line for uDAPL-1.2:
+
+ mpiexec -genv I_MPI_FALLBACK_DEVICE 0
+ -genv I_MPI_DEVICE rdma:OpenIB-cma-nes
+ -genv I_MPI_RENDEZVOUS_RDMA_WRITE
+ -ppn 1 -n 2
+ /opt/intel/impi/3.2.0.011/bin64/IMB-MPI1
+
+
+========================================
+Recommended Setting for MVAPICH2 and OFA
+========================================
+Add the following to the mpirun command:
+
+ -env MV2_USE_RDMA_CM 1
+ -env MV2_USE_IWARP_MODE 1
+
+For larger number of processes, it is also recommended to set the following:
+
+ -env MV2_MAX_INLINE_SIZE 64
+ -env MV2_USE_SRQ 0
+
+Example mpiexec command line:
+
+ mpiexec -l -n 2
+ -env MV2_USE_RDMA_CM 1
+ -env MV2_USE_IWARP_MODE 1
+ /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_latency
+
+
+==========================================
+Recommended Setting for MVAPICH2 and uDAPL
+==========================================
+Add the following to the mpirun command:
+
+ -env MV2_PREPOST_DEPTH 59
+
+Example mpiexec command line:
+
+ mpiexec -l -n 2
+ -env MV2_DAPL_PROVIDER ofa-v2-nes
+ -env MV2_PREPOST_DEPTH 59
+ /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_latency
+
+ mpiexec -l -n 2
+ -env MV2_DAPL_PROVIDER OpenIB-cma-nes
+ -env MV2_PREPOST_DEPTH 59
+ /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_latency
+
+
+===========================
+Modify Settings in Open MPI
+===========================
+There is more than one way to specify MCA parameters in
+Open MPI. Please visit this link and use the best method
+for your environment:
+
+http://www.open-mpi.org/faq/?category=tuning#setting-mca-params
+
+
+=======================================
+Recommended Settings for Open MPI 1.3.2
+=======================================
+Caching pinned memory is enabled by default but it may be necessary
+to limit the size of the cache to prevent running out of memory by
+adding the following parameter:
+
+ mpool_rdma_rcache_size_limit = <cache size>
+
+The cache size depends on the number of processes and nodes, e.g. for
+64 processes with 8 nodes, limit the pinned cache size to
+104857600 (100 MBytes).
+
+Example mpirun command line:
+
+ mpirun -np 2 -hostfile /opt/mpd.hosts
+ -mca btl openib,self,sm
+ -mca mpool_rdma_rcache_size_limit 104857600
+ /usr/mpi/gcc/openmpi-1.3.2/tests/IMB-3.1/IMB-MPI1
+
+
+=======================================
+Recommended Settings for Open MPI 1.3.1
+=======================================
+There is a known problem with cached pinned memory. It is recommended
+that pinned memory caching be disabled. For more information, see
+https://svn.open-mpi.org/trac/ompi/ticket/1853
+
+To disable pinned memory caching, add the following parameter:
+
+ mpi_leave_pinned = 0
+
+Example mpirun command line:
+
+ mpirun -np 2 -hostfile /opt/mpd.hosts
+ -mca btl openib,self,sm
+ -mca btl_mpi_leave_pinned 0
+ /usr/mpi/gcc/openmpi-1.3.1/tests/IMB-3.1/IMB-MPI1
+
+
+=====================================
+Recommended Settings for Open MPI 1.3
+=====================================
+There is a known problem with cached pinned memory. It is recommended
+that pinned memory caching be disabled. For more information, see
+https://svn.open-mpi.org/trac/ompi/ticket/1853
+
+To disable pinned memory caching, add the following parameter:
+
+ mpi_leave_pinned = 0
+
+Receive Queue setting:
+
+ btl_openib_receive_queues = P,65536,256,192,128
+
+Set maximum size of inline data segment to 64:
+
+ btl_openib_max_inline_data = 64
+
+Example mpirun command:
+
+ mpirun -np 2 -hostfile /root/mpd.hosts
+ -mca btl openib,self,sm
+ -mca btl_mpi_leave_pinned 0
+ -mca btl_openib_receive_queues P,65536,256,192,128
+ -mca btl_openib_max_inline_data 64
+ /usr/mpi/gcc/openmpi-1.3/tests/IMB-3.1/IMB-MPI1
+
+
+============
+Known Issues
+============
+The following is a list of known issues with Linux kernel and
+OFED 1.4.1 release.
+
+1. We have observed "__qdisc_run" softlockup crash running UDP
+ traffic on RHEL5.1 systems with more than 8 cores. The issue
+ is in Linux network stack. The fix for this is available from
+ the following link:
+
+http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git
+;a=commitdiff;h=2ba2506ca7ca62c56edaa334b0fe61eb5eab6ab0
+;hp=32aced7509cb20ef3ec67c9b56f5b55c41dd4f8d
+
+
+2. Running Pallas test suite and MVAPICH2 (OFA/uDAPL) for more
+ than 64 processes will abnormally terminate. The workaround is
+ add the following to mpirun command:
+
+ -env MV2_ON_DEMAND_THRESHOLD <total processes>
+
+ e.g. For 72 total processes, -env MV2_ON_DEMAND_THRESHOLD 72
+
+
+3. For MVAPICH2 (OFA/uDAPL) IMB-EXT (part of Pallas suite) "Window" test
+ may show high latency numbers. It is recommended to turn off one sided
+ communication by adding following to the mpirun command:
+
+ -env MV2_USE_RDMA_ONE_SIDED 0
+
+
+4. IMB-EXT does not run with Open MPI 1.3.1 or 1.3. The workaround is
+ to turn off message coalescing by adding the following to mpirun
+ command:
+
+ -mca btl_openib_use_message_coalescing 0
+
+
+NetEffect is a trademark of Intel Corporation in the U.S. and other countries.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nfs-rdma.release-notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nfs-rdma.release-notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/nfs-rdma.release-notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -29,25 +29,30 @@
In our testing, we have obtained excellent performance results (full 10Gbit
wire bandwidth at minimal client CPU) under many workloads. The code passes
- the full Connection test suite and operates over both Infiniband and iWARP
+ the full Connectathon test suite and operates over both Infiniband and iWARP
RDMA adapters.
-OFED 1.4 limitations:
+OFED 1.4.1 limitations:
~~~~~~~~~~~~~~~~~~~~~
-- NFS/RDMA is at technology preview state.
-- NFS/RDMA supported only kernel 2.6.26 and 2.6.27.
-- NFS/RDMA is not installed by default; to install is use custom install or add
- nfsrdma=y to ofed.conf
+ NFS-RDMA is supported for the following releases:
+ - Redhat Enterprise Linux (RHEL) version 5.1
+ - Redhat Enterprise Linux (RHEL) version 5.2
+ - Redhat Enterprise Linux (RHEL) version 5.3
+ - SUSE Linux Enterprise Server (SLES) version 10, Service Pack 2
+ - SUSE Linux Enterprise Server (SLES) version 11
-
+ And the following kernel.org kernels:
+ - 2.6.22
+ - 2.6.26
+ - 2.6.27
+ All other Linux Distrubutions and kernel versions are NOT supported on OFED 1.4.1
+
Getting Help
~~~~~~~~~~~~
If you get stuck, you can ask questions on the
-
- nfs-rdma-devel at lists.sourceforge.net, or general at lists.openfabrics.org
-
+ nfs-rdma-devel at lists.sourceforge.net, or general at lists.openfabrics.org
mailing lists.
Installation
@@ -60,65 +65,25 @@
Any device supported by the drivers in drivers/infiniband/hw is acceptable.
- Testing has been performed using several Mellanox-based IB cards, the
- Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter.
+ Testing has been performed using several Mellanox-based IB cards and
+ the Chelsio cxgb3 iWARP adapter.
- - Install OFED 1.4
+ - Install OFED 1.4.1
- OFED 1.4 comes with NFS/RDMA as a technology preview. It has been tested on
- linux 2.6.26 and 2.6.27. If you would like to try it, please select "custom"
- from the install.pl Install menu, and select NFS/RDMA as well as IP over IB.
- Note that NFS/RDMA is NOT installed if you simply select "install all."
+ NFS/RDMA has been tested on RHEL5.1, RHEL5.2, RHEL 5.3, SLES10SP2, SLES11,
+ kernels 2.6.22, 2.6.26, and 2.6.27. On these kernels, NFS-RDMA will be
+ installed by default if you simply select "install all", and can be
+ specifically included by a "custom" install.
- In addition, the install script will check the installed version of mount.nfs to
- ensure that it is from nfs-utils 1.1 or later, as that is required for NFS/RDMA.
+ In addition, the install script will install a version of the nfs-utils that
+ is required for NFS/RDMA. The binary installed will be named "mount.rnfs".
+ This version is not necessary for Linux Distributions with nfs-utils 1.1 or
+ later.
- Upon successful installation, the nfs kernel modules will be placed in the
- directory /lib/modules/'uname -a'/updates. It is recommended that you reboot to
+ Upon successful installation, the nfs kernel modules will be placed in the
+ directory /lib/modules/'uname -a'/updates. It is recommended that you reboot to
ensure that the correct modules are loaded.
- - Install nfs-utils-1.1.2 or greater on the client
-
- An NFS/RDMA mount point can be obtained by using the mount.nfs command in
- nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils
- version with support for NFS/RDMA mounts, but for various reasons we
- recommend using nfs-utils-1.1.2 or greater). To see which version of
- mount.nfs you are using, type:
-
- $ /sbin/mount.nfs -V
-
- If the version is less than 1.1.2 or the command does not exist,
- you should install the latest version of nfs-utils.
-
- Download the latest package from:
-
- http://www.kernel.org/pub/linux/utils/nfs
-
- Uncompress the package and follow the installation instructions.
-
- If you will not need the idmapper and gssd executables (you do not need
- these to create an NFS/RDMA enabled mount command), the installation
- process can be simplified by disabling these features when running
- configure:
-
- $ ./configure --disable-gss --disable-nfsv4
-
- To build nfs-utils you will need the tcp_wrappers package installed. For
- more information on this see the package's README and INSTALL files.
-
- After building the nfs-utils package, there will be a mount.nfs binary in
- the utils/mount directory. This binary can be used to initiate NFS v2, v3,
- or v4 mounts. To initiate a v4 mount, the binary must be called
- mount.nfs4. The standard technique is to create a symlink called
- mount.nfs4 to mount.nfs.
-
- This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
-
- $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
-
- In this location, mount.nfs will be invoked automatically for NFS mounts
- by the system mount command.
-
Check RDMA and NFS Setup
~~~~~~~~~~~~~~~~~~~~~~~~
@@ -214,19 +179,28 @@
Instruct the server to listen on the RDMA transport:
- $ echo rdma 2050 > /proc/fs/nfsd/portlist
+ $ echo rdma 20049 > /proc/fs/nfsd/portlist
- On the client system
Load the RDMA client module:
- $ modprobe xprtrdma.ko
+ $ modprobe xprtrdma
Mount the NFS/RDMA server:
- $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt
+ $ mount -o rdma,port=20049 <IPoIB-server-name-or-address>:/<export> /mnt
To verify that the mount is using RDMA, run "cat /proc/mounts" and check
the "proto" field for the given mount.
Congratulations! You're using NFS/RDMA!
+
+Known Issues
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you're running NFSRDMA over Chelsio's T3 RNIC and your cients are using
+a 64KB page size (like PPC64 and IA64 systems) and your server is using a
+4KB page size (like i386 and X86_64), then you need to mount the server
+using rsize=32768,wsize=32768 to avoid overrunning the Chelsio RNIC fast
+register limits. This is a known firmware limitation in the Chelsio RNIC.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/open_mpi_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/open_mpi_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/open_mpi_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,30 +1,47 @@
Open Fabrics Enterprise Distribution (OFED)
- Open MPI in OFED 1.4 Copyrights, License, and Release Notes
+ Open MPI in OFED 1.4.1 Copyrights, License, and Release Notes
- December 2008
+ May 2009
-
Open MPI Copyrights
-------------------
-Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
+Most files in this release are marked with the copyrights of the
+organizations who have edited them. The copyrights below generally
+reflect members of the Open MPI core team who have contributed code to
+this release. The copyrights for code used under license from other
+parties are included in the corresponding files.
+
+Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
University Research and Technology
Corporation. All rights reserved.
-Copyright (c) 2004-2007 The University of Tennessee and The University
+Copyright (c) 2004-2009 The University of Tennessee and The University
of Tennessee Research Foundation. All rights
reserved.
-Copyright (c) 2004-2006 High Performance Computing Center Stuttgart,
+Copyright (c) 2004-2008 High Performance Computing Center Stuttgart,
University of Stuttgart. All rights reserved.
-Copyright (c) 2004-2006 The Regents of the University of California.
+Copyright (c) 2004-2007 The Regents of the University of California.
All rights reserved.
-Copyright (c) 2006-2007 Los Alamos National Security, LLC. All rights
+Copyright (c) 2006-2009 Los Alamos National Security, LLC. All rights
reserved.
-Copyright (c) 2006-2007 Cisco Systems, Inc. All rights reserved.
-Copyright (c) 2006-2007 Voltaire, Inc. All rights reserved.
-Copyright (c) 2006 Sandia National Laboratories. All rights reserved.
-Copyright (c) 2006-2007 Sun Microsystems, Inc. All rights reserved.
+Copyright (c) 2006-2009 Cisco Systems, Inc. All rights reserved.
+Copyright (c) 2006-2008 Voltaire, Inc. All rights reserved.
+Copyright (c) 2006-2008 Sandia National Laboratories. All rights reserved.
+Copyright (c) 2006-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
-Copyright (c) 2006-2007 The University of Houston. All rights reserved.
-Copyright (c) 2006 Myricom, Inc. All rights reserved.
+Copyright (c) 2006-2009 The University of Houston. All rights reserved.
+Copyright (c) 2006-2008 Myricom, Inc. All rights reserved.
+Copyright (c) 2007-2008 UT-Battelle, LLC. All rights reserved.
+Copyright (c) 2007-2008 IBM Corporation. All rights reserved.
+Copyright (c) 1998-2005 Forschungszentrum Juelich, Juelich Supercomputing
+ Centre, Federal Republic of Germany
+Copyright (c) 2005-2008 ZIH, TU Dresden, Federal Republic of Germany
+Copyright (c) 2007 Evergrid, Inc. All rights reserved.
+Copyright (c) 2008 Institut National de Recherche en
+ Informatique. All rights reserved.
+Copyright (c) 2007 Lawrence Livermore National Security, LLC.
+ All rights reserved.
+Copyright (c) 2007-2009 Mellanox Technologies. All rights reserved.
+Copyright (c) 2006 QLogic Corporation. All rights reserved.
Additional copyrights may follow
@@ -67,6 +84,12 @@
===========================================================================
+When submitting questions and problems, be sure to include as much
+extra information as possible. This web page details all the
+information that we request in order to provide assistance:
+
+ http://www.open-mpi.org/community/help/
+
The best way to report bugs, send comments, or ask questions is to
sign up on the user's and/or developer's mailing list (for user-level
and developer-level questions; when in doubt, send to the user's
@@ -97,17 +120,6 @@
OFED-Specific Release Notes
---------------------------
-** iWARP support
-
-The version of Open MPI included in OFED 1.4 does not include iWARP
-support. iWARP support is included in the upcoming Open MPI v1.3
-release (which was not ready in time for the OFED 1.4 release);
-please see http://www.open-mpi.org/ for updates.
-
-See the "Installing newer versions of Open MPI after OFED is
-installed" section, below, for details about how to download and
-install newer versions of Open MPI from its web site.
-
** SLES 10 with Pathscale compiler support:
Using the Pathscale compiler to build Open MPI on SLES10 may result in
@@ -142,33 +154,265 @@
important part is that the "openib" BTL is displayed):
shell$ ompi_info | grep openib
- MCA btl: openib (MCA v1.0, API v1.0.1, Component v1.2.8)
+ MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2)
See the rest of the documentation below for other configure command
line options and installation instructions.
-** OFED 1.4 bug fixes (corresponding to Open MPI v1.2.7 and v1.2.8):
+** Changelog summary
-v1.2.8
-- Tweaked one memory barrier in the openib component to be more
- conservative. May fix a problem observed on PPC machines. See
- ticket #1532.
+Showing versions 1.2.7 - 1.3.2; see the "NEWS" file in an Open MPI
+distribution for the full list.
+
+1.3.2
+-----
+
+- Fixed a potential infinite loop in the openib BTL that could occur
+ in senders in some frequent-communication scenarios. Thanks to Don
+ Wood for reporting the problem.
+- Add a new checksum PML variation on ob1 (main MPI point-to-point
+ communication engine) to detect memory corruption in node-to-node
+ messages
+- Add a new configuration option to add padding to the openib
+ header so the data is aligned
+- Add a new configuration option to use an alternative checksum algo
+ when using the checksum PML
+- Fixed a problem reported by multiple users on the mailing list that
+ the LSF support would fail to find the appropriate libraries at
+ run-time.
+- Allow empty shell designations from getpwuid(). Thanks to Sergey
+ Koposov for the bug report.
+- Ensure that mpirun exits with non-zero status when applications die
+ due to user signal. Thanks to Geoffroy Pignot for suggesting the
+ fix.
+- Ensure that MPI_VERSION / MPI_SUBVERSION match what is returned by
+ MPI_GET_VERSION. Thanks to Rob Egan for reporting the error.
+- Updated MPI_*KEYVAL_CREATE functions to properly handle Fortran
+ extra state.
+- A variety of ob1 (main MPI point-to-point communication engine) bug
+ fixes that could have caused hangs or seg faults.
+- Do not install Open MPI's signal handlers in MPI_INIT if there are
+ already signal handlers installed. Thanks to Kees Verstoep for
+ bringing the issue to our attention.
+- Fix GM support to not seg fault in MPI_INIT.
+- Various VampirTrace fixes.
+- Various PLPA fixes.
+- No longer create BTLs for invalid (TCP) devices.
+- Various man page style and lint cleanups.
+- Fix critical OpenFabrics-related bug noted here:
+ http://www.open-mpi.org/community/lists/announce/2009/03/0029.php.
+ Open MPI now uses a much more robust memory intercept scheme that is
+ quite similar to what is used by MX. The use of "-lopenmpi-malloc"
+ is no longer necessary, is deprecated, and is expected to disappear
+ in a future release. -lopenmpi-malloc will continue to work for the
+ duration of the Open MPI v1.3 and v1.4 series.
+- Fix some OpenFabrics shutdown errors, both regarding iWARP and SRQ.
+- Allow the udapl BTL to work on Solaris platforms that support
+ relaxed PCI ordering.
+- Fix problem where the mpirun would sometimes use rsh/ssh to launch on
+ the localhost (instead of simply forking).
+- Minor SLURM stdin fixes.
+- Fix to run properly under SGE jobs.
+- Scalability and latency improvements for shared memory jobs: convert
+ to using one message queue instead of N queues.
+- Automatically size the shared-memory area (mmap file) to match
+ better what is needed; specifically, so that large-np jobs will start.
+- Use fixed-length MPI predefined handles in order to provide ABI
+ compatibility between Open MPI releases.
+- Fix building of the posix paffinity component to properly get the
+ number of processors in loosely tested environments (e.g.,
+ FreeBSD). Thanks to Steve Kargl for reporting the issue.
+- Fix --with-libnuma handling in configure. Thanks to Gus Correa for
+ reporting the problem.
+
+
+1.3.1
+-----
+
+- Added "sync" coll component to allow users to synchronize every N
+ collective operations on a given communicator.
+- Increased the default values of the IB and RNR timeout MCA parameters.
+- Fix a compiler error noted by Mostyn Lewis with the PGI 8.0 compiler.
+- Fix an error that prevented stdin from being forwarded if the
+ rsh launcher was in use. Thanks to Branden Moore for pointing out
+ the problem.
+- Correct a case where the added datatype is considered as contiguous but
+ has gaps in the beginning.
+- Fix an error that limited the number of comm_spawns that could
+ simultaneously be running in some environments
+- Correct a corner case in OB1's GET protocol for long messages; the
+ error could sometimes cause MPI jobs using the openib BTL to hang.
+- Fix a bunch of bugs in the IO forwarding (IOF) subsystem and add some
+ new options to output to files and redirect output to xterm. Thanks to
+ Jody Weissmann for helping test out many of the new fixes and
+ features.
+- Fix SLURM race condition.
+- Fix MPI_File_c2f(MPI_FILE_NULL) to return 0, not -1. Thanks to
+ Lisandro Dalcin for the bug report.
+- Fix the DSO build of tm PLM.
+- Various fixes for size disparity between C int's and Fortran
+ INTEGER's. Thanks to Christoph van Wullen for the bug report.
+- Ensure that mpirun exits with a non-zero exit status when daemons or
+ processes abort or fail to launch.
+- Various fixes to work around Intel (NetEffect) RNIC behavior.
+- Various fixes for mpirun's --preload-files and --preload-binary
+ options.
+- Fix the string name in MPI::ERRORS_THROW_EXCEPTIONS.
+- Add ability to forward SIFTSTP and SIGCONT to MPI processes if you
+ set the MCA parameter orte_forward_job_control to 1.
+- Allow the sm BTL to allocate larger amounts of shared memory if
+ desired (helpful for very large multi-core boxen).
+- Fix a few places where we used PATH_MAX instead of OMPI_PATH_MAX,
+ leading to compile problems on some platforms. Thanks to Andrea Iob
+ for the bug report.
+- Fix mca_btl_openib_warn_no_device_params_found MCA parameter; it
+ was accidentally being ignored.
+- Fix some run-time issues with the sctp BTL.
+- Ensure that RTLD_NEXT exists before trying to use it (e.g., it
+ doesn't exist on Cygwin). Thanks to Gustavo Seabra for reporting
+ the issue.
+- Various fixes to VampirTrace, including fixing compile errors on
+ some platforms.
+- Fixed missing MPI_Comm_accept.3 man page; fixed minor issue in
+ orterun.1 man page. Thanks to Dirk Eddelbuettel for identifying the
+ problem and submitting a patch.
+- Implement the XML formatted output of stdout/stderr/stddiag.
+- Fixed mpirun's -wdir switch to ensure that working directories for
+ multiple app contexts are properly handled. Thanks to Geoffroy
+ Pignot for reporting the problem.
+- Improvements to the MPI C++ integer constants:
+ - Allow MPI::SEEK_* constants to be used as constants
+ - Allow other MPI C++ constants to be used as array sizes
+- Fix minor problem with orte-restart's command line options. See
+ ticket #1761 for details. Thanks to Gregor Dschung for reporting
+ the problem.
+
+1.3
+---
+
+- Extended the OS X 10.5.x (Leopard) workaround for a problem when
+ assembly code is compiled with -g[0-9]. Thanks to Barry Smith for
+ reporting the problem. See ticket #1701.
+- Disabled MPI_REAL16 and MPI_COMPLEX32 support on platforms where the
+ bit representation of REAL*16 is different than that of the C type
+ of the same size (usually long double). Thanks to Julien Devriendt
+ for reporting the issue. See ticket #1603.
+- Increased the size of MPI_MAX_PORT_NAME to 1024 from 36. See ticket #1533.
+- Added "notify debugger on abort" feature. See tickets #1509 and #1510.
+ Thanks to Seppo Sahrakropi for the bug report.
+- Upgraded Open MPI tarballs to use Autoconf 2.63, Automake 1.10.1,
+ Libtool 2.2.6a.
+- Added missing MPI::Comm::Call_errhandler() function. Thanks to Dave
+ Goodell for bringing this to our attention.
+- Increased MPI_SUBVERSION value in mpi.h to 1 (i.e., MPI 2.1).
+- Changed behavior of MPI_GRAPH_CREATE, MPI_TOPO_CREATE, and several
+ other topology functions per MPI-2.1.
+- Fix the type of the C++ constant MPI::IN_PLACE.
+- Various enhancements to the openib BTL:
+ - Added btl_openib_if_[in|ex]clude MCA parameters for
+ including/excluding comma-delimited lists of HCAs and ports.
+ - Added RDMA CM support, includng btl_openib_cpc_[in|ex]clude MCA
+ parameters
+ - Added NUMA support to only use "near" network adapters
+ - Added "Bucket SRQ" (BSRQ) support to better utilize registered
+ memory, including btl_openib_receive_queues MCA parameter
+ - Added ConnectX XRC support (and integrated with BSRQ)
+ - Added btl_openib_ib_max_inline_data MCA parameter
+ - Added iWARP support
+ - Revamped flow control mechansisms to be more efficient
+ - "mpi_leave_pinned=1" is now the default when possible,
+ automatically improving performance for large messages when
+ application buffers are re-used
+- Elimiated duplicated error messages when multiple MPI processes fail
+ with the same error.
+- Added NUMA support to the shared memory BTL.
+- Add Valgrind-based memory checking for MPI-semantic checks.
+- Add support for some optional Fortran datatypes (MPI_LOGICAL1,
+ MPI_LOGICAL2, MPI_LOGICAL4 and MPI_LOGICAL8).
+- Remove the use of the STL from the C++ bindings.
+- Added support for Platform/LSF job launchers. Must be Platform LSF
+ v7.0.2 or later.
+- Updated ROMIO with the version from MPICH2 1.0.7.
+- Added RDMA capable one-sided component (called rdma), which
+ can be used with BTL components that expose a full one-sided
+ interface.
+- Added the optional datatype MPI_REAL2. As this is added to the "end of"
+ predefined datatypes in the fortran header files, there will not be
+ any compatibility issues.
+- Added Portable Linux Processor Affinity (PLPA) for Linux.
+- Addition of a finer symbols export control via the visibiliy feature
+ offered by some compilers.
+- Added checkpoint/restart process fault tolerance support. Initially
+ support a LAM/MPI-like protocol.
+- Removed "mvapi" BTL; all InfiniBand support now uses the OpenFabrics
+ driver stacks ("openib" BTL).
+- Added more stringent MPI API parameter checking to help user-level
+ debugging.
+- The ptmalloc2 memory manager component is now by default built as
+ a standalone library named libopenmpi-malloc. Users wanting to
+ use leave_pinned with ptmalloc2 will now need to link the library
+ into their application explicitly. All other users will use the
+ libc-provided allocator instead of Open MPI's ptmalloc2. This change
+ may be overriden with the configure option enable-ptmalloc2-internal
+- The leave_pinned options will now default to using mallopt on
+ Linux in the cases where ptmalloc2 was not linked in. mallopt
+ will also only be available if munmap can be intercepted (the
+ default whenever Open MPI is not compiled with --without-memory-
+ manager.
+- Open MPI will now complain and refuse to use leave_pinned if
+ no memory intercept / mallopt option is available.
+- Add option of using Perl-based wrapper compilers instead of the
+ C-based wrapper compilers. The Perl-based version does not
+ have the features of the C-based version, but does work better
+ in cross-compile environments.
+
+
+1.2.9
+-----
+
+- Fix a segfault when using one-sided communications on some forms of derived
+ datatypes. Thanks to Dorian Krause for reporting the bug. See #1715.
+- Fix an alignment problem affecting one-sided communications on
+ some architectures (e.g., SPARC64). See #1738.
+- Fix compilation on Solaris when thread support is enabled in Open MPI
+ (e.g., when using --with-threads). See #1736.
+- Correctly take into account the MTU that an OpenFabrics device port
+ is using. See #1722 and
+ https://bugs.openfabrics.org/show_bug.cgi?id=1369.
+- Fix two datatype engine bugs. See #1677.
+ Thanks to Peter Kjellstrom for the bugreport.
+- Fix the bml r2 help filename so the help message can be found. See #1623.
+- Fix a compilation problem on RHEL4U3 with the PGI 32 bit compiler
+ caused by <infiniband/driver.h>. See ticket #1613.
+- Fix the --enable-cxx-exceptions configure option. See ticket #1607.
+- Properly handle when the MX BTL cannot open an endpoint. See ticket #1621.
+- Fix a double free of events on the tcp_events list. See ticket #1631.
+- Fix a buffer overun in opal_free_list_grow (called by MPI_Init).
+ Thanks to Patrick Farrell for the bugreport and Stephan Kramer for
+ the bugfix. See ticket #1583.
+- Fix a problem setting OPAL_PREFIX for remote sh-based shells.
+ See ticket #1580.
+
+
+1.2.8
+-----
+
+- Tweaked one memory barrier in the openib component to be more conservative.
+ May fix a problem observed on PPC machines. See ticket #1532.
- Fix OpenFabrics IB partition support. See ticket #1557.
-- Restore v1.1 feature that sourced .profile on remote nodes if the
- default shell will not do so (e.g. /bin/sh and /bin/ksh). See
- ticket #1560.
-- Fix segfault in MPI_Init_thread() if ompi_mpi_init() fails. See
- ticket #1562.
-- Adjust SLURM support to first look for $SLURM_JOB_CPUS_PER_NODE
- instead of the deprecated $SLURM_TASKS_PER_NODE environment
- variable. This change may be *required* when using SLURM v1.2 and
- above. See ticket #1536.
-- Fix the MPIR_Proctable to be in process rank order. See ticket
- #1529.
-- Fix a regression introduced in 1.2.6 for the IBM eHCA. See ticket
- #1526.
+- Restore v1.1 feature that sourced .profile on remote nodes if the default
+ shell will not do so (e.g. /bin/sh and /bin/ksh). See ticket #1560.
+- Fix segfault in MPI_Init_thread() if ompi_mpi_init() fails. See ticket #1562.
+- Adjust SLURM support to first look for $SLURM_JOB_CPUS_PER_NODE instead of
+ the deprecated $SLURM_TASKS_PER_NODE environment variable. This change
+ may be *required* when using SLURM v1.2 and above. See ticket #1536.
+- Fix the MPIR_Proctable to be in process rank order. See ticket #1529.
+- Fix a regression introduced in 1.2.6 for the IBM eHCA. See ticket #1526.
-v1.2.7
+
+1.2.7
+-----
+
- Add some Sun HCA vendor IDs. See ticket #1461.
- Fixed a memory leak in MPI_Alltoallw when called from Fortran.
Thanks to Dave Grote for the bugreport. See ticket #1457.
@@ -210,73 +454,183 @@
General Release Notes
---------------------
+Detailed Open MPI v1.3 Feature List:
+
+ o Open MPI RunTime Environment (ORTE) improvements
+ - General robustness improvements
+ - Scalable job launch (we've seen ~16K processes in less than a
+ minute in a highly-optimized configuration)
+ - New process mappers
+ - Support for Platform/LSF environments (v7.0.2 and later)
+ - More flexible processing of host lists
+ - new mpirun cmd line options and associated functionality
+
+ o Fault-Tolerance Features
+ - Asynchronous, transparent checkpoint/restart support
+ - Fully coordinated checkpoint/restart coordination component
+ - Support for the following checkpoint/restart services:
+ - blcr: Berkley Lab's Checkpoint/Restart
+ - self: Application level callbacks
+ - Support for the following interconnects:
+ - tcp
+ - mx
+ - openib
+ - sm
+ - self
+ - Improved Message Logging
+
+ o MPI_THREAD_MULTIPLE support for point-to-point messaging in the
+ following BTLs (note that only MPI point-to-point messaging API
+ functions support MPI_THREAD_MULTIPLE; other API functions likely
+ do not):
+ - tcp
+ - sm
+ - mx
+ - elan
+ - self
+
+ o Point-to-point Messaging Layer (PML) improvements
+ - Memory footprint reduction
+ - Improved latency
+ - Improved algorithm for multiple communication device
+ ("multi-rail") support
+
+ o Numerous Open Fabrics improvements/enhancements
+ - Added iWARP support (including RDMA CM)
+ - Memory footprint and performance improvements
+ - "Bucket" SRQ support for better registered memory utilization
+ - XRC/ConnectX support
+ - Message coalescing
+ - Improved error report mechanism with Asynchronous events
+ - Automatic Path Migration (APM)
+ - Improved processor/port binding
+ - Infrastructure for additional wireup strategies
+ - mpi_leave_pinned is now enabled by default
+
+ o uDAPL BTL enhancements
+ - Multi-rail support
+ - Subnet checking
+ - Interface include/exclude capabilities
+
+ o Processor affinity
+ - Linux processor affinity improvements
+ - Core/socket <--> process mappings
+
+ o Collectives
+ - Performance improvements
+ - Support for hierarchical collectives (must be activated
+ manually; see below)
+
+ o Miscellaneous
+ - MPI 2.1 compliant
+ - Sparse process groups and communicators
+ - Support for Cray Compute Node Linux (CNL)
+ - One-sided RDMA component (BTL-level based rather than PML-level
+ based)
+ - Aggregate MCA parameter sets
+ - MPI handle debugging
+ - Many small improvements to the MPI C++ bindings
+ - Valgrind support
+ - VampirTrace support
+ - Updated ROMIO to the version from MPICH2 1.0.7
+ - Removed the mVAPI IB stacks
+ - Display most error messages only once (vs. once for each
+ process)
+ - Many other small improvements and bug fixes, too numerous to
+ list here
+
+Known issues
+------------
+
+ o There is a segfault that sometimes occurs on one of our x86_64 test
+ clusters when using MPI onesided communications over Myrinet MX.
+ Since no one else has reported this problem we are not holding
+ up the 1.3 release. See ticket #1757 for the details, and any
+ possible workarounds.
+
+ o XGrid support is currently broken.
+ https://svn.open-mpi.org/trac/ompi/ticket/1777
+
+ o MPI_REDUCE_SCATTER does not work with counts of 0.
+ https://svn.open-mpi.org/trac/ompi/ticket/1559
+
+ o Please also see the Open MPI bug tracker for bugs beyond this release.
+ https://svn.open-mpi.org/trac/ompi/report
+
+===========================================================================
+
The following abbreviated list of release notes applies to this code
-base as of this writing (19 September 2007):
+base as of this writing (14 April 2009):
+General notes
+-------------
+
- Open MPI includes support for a wide variety of supplemental
hardware and software package. When configuring Open MPI, you may
need to supply additional flags to the "configure" script in order
to tell Open MPI where the header files, libraries, and any other
required files are located. As such, running "configure" by itself
- may include support for all the devices (etc.) that you expect,
+ may not include support for all the devices (etc.) that you expect,
especially if their support headers / libraries are installed in
non-standard locations. Network interconnects are an easy example
- to discuss -- Myrinet and InfiniBand, for example, both have
- supplemental headers and libraries that must be found before Open
- MPI can build support for them. You must specify where these files
- are with the appropriate options to configure. See the listing of
- configure command-line switches, below, for more details.
+ to discuss -- Myrinet and OpenFabrics networks, for example, both
+ have supplemental headers and libraries that must be found before
+ Open MPI can build support for them. You must specify where these
+ files are with the appropriate options to configure. See the
+ listing of configure command-line switches, below, for more details.
-- The Open MPI installation must be in your PATH on all nodes (and
- potentially LD_LIBRARY_PATH, if libmpi is a shared library), unless
- using the --prefix or --enable-mpirun-prefix-by-default
- functionality (see below).
+- The majority of Open MPI's documentation is here in this file, the
+ included man pages, and on the web site FAQ
+ (http://www.open-mpi.org/). This will eventually be supplemented
+ with cohesive installation and user documentation files.
-- LAM/MPI-like mpirun notation of "C" and "N" is not yet supported.
+- Note that Open MPI documentation uses the word "component"
+ frequently; the word "plugin" is probably more familiar to most
+ users. As such, end users can probably completely substitute the
+ word "plugin" wherever you see "component" in our documentation.
+ For what it's worth, we use the word "component" for historical
+ reasons, mainly because it is part of our acronyms and internal API
+ functionc calls.
-- Striping MPI messages across multiple networks is supported (and
- happens automatically when multiple networks are available), but
- needs performance tuning.
-
- The run-time systems that are currently supported are:
- rsh / ssh
- - BProc versions 3 and 4 with LSF
- LoadLeveler
- PBS Pro, Open PBS, Torque
+ - Platform LSF (v7.0.2 and later)
- SLURM
- - XGrid
+ - XGrid (known to be broken in 1.3 through 1.3.2)
- Cray XT-3 and XT-4
- - Sun N1 Grid Engine (N1GE) 6 and open source Grid Engine
+ - Sun Grid Engine (SGE) 6.1, 6.2 and open source Grid Engine
+ - Microsoft Windows CCP (Microsoft Windows server 2003 and 2008)
-- The majority of Open MPI's documentation is here in this file, the
- included man pages, and on the web site FAQ
- (http://www.open-mpi.org/). This will eventually be supplemented
- with cohesive installation and user documentation files.
-
- Systems that have been tested are:
- - Linux, 32 bit, with gcc
- - Linux, 64 bit (x86), with gcc
+ - Linux (various flavors/distros), 32 bit, with gcc, and Sun Studio 12
+ - Linux (various flavors/distros), 64 bit (x86), with gcc, Absoft,
+ Intel, Portland, Pathscale, and Sun Studio 12 compilers (*)
- OS X (10.4), 32 and 64 bit (i386, PPC, PPC64, x86_64), with gcc
- - Solaris 10 updates 2 and 3, SPARC and AMD, 32 and 64 bit, with Sun
- Studio 10 and 11
+ and Absoft compilers (*)
+ - Solaris 10 update 2, 3 and 4, 32 and 64 bit (SPARC, i386, x86_64),
+ with Sun Studio 10, 11 and 12
+ (*) Be sure to read the Compiler Notes, below.
+
- Other systems have been lightly (but not fully tested):
- - Other compilers on Linux, 32 and 64 bit
- Other 64 bit platforms (e.g., Linux on PPC64)
+ - Microsoft Windows CCP (Microsoft Windows server 2003 and 2008);
+ more testing and support is expected later in the Open MPI v1.3.x
+ series.
-- Some MCA parameters can be set in a way that renders Open MPI
- inoperable (see notes about MCA parameters later in this file). In
- particular, some parameters have required options that must be
- included.
- - If specified, the "btl" parameter must include the "self"
- component, or Open MPI will not be able to deliver messages to the
- same rank as the sender. For example: "mpirun --mca btl tcp,self
- ..."
- - If specified, the "btl_tcp_if_exclude" paramater must include the
- loopback device ("lo" on many Linux platforms), or Open MPI will
- not be able to route MPI messages using the TCP BTL. For example:
- "mpirun --mca btl_tcp_if_exclude lo,eth1 ..."
+Compiler Notes
+--------------
+- Mixing compilers from different vendors when building Open MPI
+ (e.g., using the C/C++ compiler from one vendor and the F77/F90
+ compiler from a different vendor) has been successfully employed by
+ some Open MPI users (discussed on the Open MPI user's mailing list),
+ but such configurations are not tested and not documented. For
+ example, such configurations may require additional compiler /
+ linker flags to make Open MPI build properly.
+
- Open MPI does not support the Sparc v8 CPU target, which is the
default on Sun Solaris. The v8plus (32 bit) or v9 (64 bit)
targets must be used to build Open MPI on Solaris. This can be
@@ -314,6 +668,12 @@
also automatically add "-Msignextend" when the C and C++ MPI wrapper
compilers are used to compile user MPI applications.
+- Using the MPI C++ bindings with the Pathscale compiler is known
+ to fail, possibly due to Pathscale compiler issues.
+
+- Using the Absoft compiler to build the MPI Fortran bindings on Suse
+ 9.3 is known to fail due to a Libtool compatibility issue.
+
- Open MPI will build bindings suitable for all common forms of
Fortran 77 compiler symbol mangling on platforms that support it
(e.g., Linux). On platforms that do not support weak symbols (e.g.,
@@ -349,43 +709,6 @@
You can use the ompi_info command to see the Fortran compiler that
Open MPI was configured with.
-- Running on nodes with different endian and/or different datatype
- sizes within a single parallel job is supported in this release.
- However, Open MPI does not resize data when datatypes differ in size
- (for example, sending a 4 byte MPI_DOUBLE and receiving an 8 byte
- MPI_DOUBLE will fail).
-
-- MPI_THREAD_MULTIPLE support is included, but is only lightly tested.
- It likely does not work for thread-intensive applications.
-
-- Asynchronous message passing progress using threads can be turned on
- with the --enable-progress-threads option to configure.
- Asynchronous message passing progress is only supported for TCP,
- shared memory, and Myrinet/GM. Myrinet/GM has only been lightly
- tested.
-
-- The XGrid support is experimental - see the Open MPI FAQ and this
- post on the Open MPI user's mailing list for more information:
-
- http://www.open-mpi.org/community/lists/users/2006/01/0539.php
-
-- The OpenFabrics Enterprise Distribution (OFED) software package v1.0
- will not work properly with Open MPI v1.2 (and later) due to how its
- Mellanox InfiniBand plugin driver is created. The problem is fixed
- OFED v1.1 (and later).
-
-- The use of the mvapi BTL is deprecated. All new InfiniBand work is
- being done in the openib BTL (i.e., the OpenFabrics driver stack).
-
-- The use of fork() with the openib BTL is only partially supported,
- and only on Linux kernels >= v2.6.15 with libibverbs v1.1 or later
- (first released as part of OFED v1.2). More complete support will
- be included in a future release of Open MPI (see the OFED 1.2
- distribution for details).
-
-- iWARP support is not yet included in the Open MPI OpenFabrics
- support.
-
- The Fortran 90 MPI bindings can now be built in one of three sizes
using --with-mpi-f90-size=SIZE (see description below). These sizes
reflect the number of MPI functions included in the "mpi" Fortran 90
@@ -427,21 +750,129 @@
interface. A "large" size that includes the two choice buffer MPI
functions is possible in future versions of Open MPI.
-- Starting with Open MPI v1.2, there are two MPI network models
- available: "ob1" and "cm". "ob1" uses the familiar BTL components
- for each supported network. "cm" introduces MTL components for
+
+General Run-Time Support Notes
+------------------------------
+
+- The Open MPI installation must be in your PATH on all nodes (and
+ potentially LD_LIBRARY_PATH, if libmpi is a shared library), unless
+ using the --prefix or --enable-mpirun-prefix-by-default
+ functionality (see below).
+
+- LAM/MPI-like mpirun notation of "C" and "N" is not yet supported.
+
+- The XGrid support is experimental - see the Open MPI FAQ and this
+ post on the Open MPI user's mailing list for more information:
+
+ http://www.open-mpi.org/community/lists/users/2006/01/0539.php
+
+- Open MPI's run-time behavior can be customized via MCA ("MPI
+ Component Architecture") parameters (see below for more information
+ on how to get/set MCA parameter values). Some MCA parameters can be
+ set in a way that renders Open MPI inoperable (see notes about MCA
+ parameters later in this file). In particular, some parameters have
+ required options that must be included.
+
+ - If specified, the "btl" parameter must include the "self"
+ component, or Open MPI will not be able to deliver messages to the
+ same rank as the sender. For example: "mpirun --mca btl tcp,self
+ ..."
+ - If specified, the "btl_tcp_if_exclude" paramater must include the
+ loopback device ("lo" on many Linux platforms), or Open MPI will
+ not be able to route MPI messages using the TCP BTL. For example:
+ "mpirun --mca btl_tcp_if_exclude lo,eth1 ..."
+
+- Running on nodes with different endian and/or different datatype
+ sizes within a single parallel job is supported in this release.
+ However, Open MPI does not resize data when datatypes differ in size
+ (for example, sending a 4 byte MPI_DOUBLE and receiving an 8 byte
+ MPI_DOUBLE will fail).
+
+
+MPI Functionality and Features
+------------------------------
+
+- All MPI-2.1 functionality is supported.
+
+- MPI_THREAD_MULTIPLE support is included, but is only lightly tested.
+ It likely does not work for thread-intensive applications. Note
+ that *only* the MPI point-to-point communication functions for the
+ BTL's listed above are considered thread safe. Other support
+ functions (e.g., MPI attributes) have not been certified as safe
+ when simultaneously used by multiple threads.
+
+ Note that Open MPI's thread support is in a fairly early stage; the
+ above devices are likely to *work*, but the latency is likely to be
+ fairly high. Specifically, efforts so far have concentrated on
+ *correctness*, not *performance* (yet).
+
+- MPI_REAL16 and MPI_COMPLEX32 are only supported on platforms where a
+ portable C datatype can be found that matches the Fortran type
+ REAL*16, both in size and bit representation.
+
+- Asynchronous message passing progress using threads can be turned on
+ with the --enable-progress-threads option to configure.
+ Asynchronous message passing progress is only supported with devices
+ that support MPI_THREAD_MULTIPLE, but is only very lightly tested
+ (and may not provide very much performance benefit).
+
+
+Collectives
+-----------
+
+- The "hierarch" coll component (i.e., an implementation of MPI
+ collective operations) attempts to discover network layers of
+ latency in order to segregate individual "local" and "global"
+ operations as part of the overall collective operation. In this
+ way, network traffic can be reduced -- or possibly even minimized
+ (similar to MagPIe). The current "hierarch" component only
+ separates MPI processes into on- and off-node groups.
+
+ Hierarch has had sufficient correctness testing, but has not
+ received much performance tuning. As such, hierarch is not
+ activated by default -- it must be enabled manually by setting its
+ priority level to 100:
+
+ mpirun --mca coll_hierarch_priority 100 ...
+
+ We would appreciate feedback from the user community about how well
+ hierarch works for your applications.
+
+
+Network Support
+---------------
+
+- The OpenFabrics Enterprise Distribution (OFED) software package v1.0
+ will not work properly with Open MPI v1.2 (and later) due to how its
+ Mellanox InfiniBand plugin driver is created. The problem is fixed
+ OFED v1.1 (and later).
+
+- Older mVAPI-based InfiniBand drivers (Mellanox VAPI) are no longer
+ supported. Please use an older version of Open MPI (1.2 series or
+ earlier) if you need mVAPI support.
+
+- The use of fork() with the openib BTL is only partially supported,
+ and only on Linux kernels >= v2.6.15 with libibverbs v1.1 or later
+ (first released as part of OFED v1.2), per restrictions imposed by
+ the OFED network stack.
+
+- There are two MPI network models available: "ob1" and "cm". "ob1"
+ uses BTL ("Byte Transfer Layer") components for each supported
+ network. "cm" uses MTL ("Matching Tranport Layer") components for
each supported network.
- "ob1" supports a variety of networks that can be used in
combination with each other (per OS constraints; e.g., there are
reports that the GM and OpenFabrics kernel drivers do not operate
well together):
- - InfiniBand: mVAPI and the OpenFabrics stack
+ - OpenFabrics: InfiniBand and iWARP
- Loopback (send-to-self)
- Myrinet: GM and MX
- Portals
+ - Quadrics Elan
- Shared memory
- TCP
+ - SCTP
- uDAPL
- "cm" supports a smaller number of networks (and they cannot be
@@ -451,44 +882,47 @@
- InfiniPath PSM
- Portals
- Open MPI will, by default, choose to use "cm" if it finds a
- cm-supported network at run-time. Users can force the use of ob1 if
- desired by setting the "pml" MCA parameter at run-time:
+ Open MPI will, by default, choose to use "cm" when the InfiniPath
+ PSM MTL can be used. Otherwise, OB1 will be used and the
+ corresponding BTLs will be selected. Users can force the use of ob1
+ or cm if desired by setting the "pml" MCA parameter at run-time:
shell$ mpirun --mca pml ob1 ...
+ or
+ shell$ mpirun --mca pml cm ...
-- The MX support is shared between the 2 internal devices, the MTL
- and the BTL. MTL stands for Message Transport Layer, while BTL
- stands for Byte Transport Layer. The design of the BTL interface
- in Open MPI assumes that only naive one-sided communication
- capabilities are provided by the low level communication layers.
- However, modern communication layers such as MX, PSM or Portals,
- natively implement highly-optimized two-sided communication
- semantics. To leverage these capabilities, Open MPI provides the
- MTL interface to transfer messages rather than bytes.
+- Myrinet MX support is shared between the 2 internal devices, the MTL
+ and the BTL. The design of the BTL interface in Open MPI assumes
+ that only naive one-sided communication capabilities are provided by
+ the low level communication layers. However, modern communication
+ layers such as Myrinet MX, InfiniPath PSM, or Portals, natively
+ implement highly-optimized two-sided communication semantics. To
+ leverage these capabilities, Open MPI provides the "cm" PML and
+ corresponding MTL components to transfer messages rather than bytes.
The MTL interface implements a shorter code path and lets the
- low-level network library decide which protocol to use, depending
- on message length, internal resources and other parameters
- specific to the interconnect used. However, Open MPI cannot
- currently use multiple MTL modules at once. In the case of the
- MX MTL, self and shared memory communications are provided by the
- MX library. Moreover, the current MX MTL does not support message
- pipelining resulting in lower performances in case of non-contiguous
- data-types.
- In the case of the BTL, MCA parameters allow Open MPI to use our own
- shared memory and self device for increased performance.
+ low-level network library decide which protocol to use (depending on
+ issues such as message length, internal resources and other
+ parameters specific to the underlying interconnect). However, Open
+ MPI cannot currently use multiple MTL modules at once. In the case
+ of the MX MTL, process loopback and on-node shared memory
+ communications are provided by the MX library. Moreover, the
+ current MX MTL does not support message pipelining resulting in
+ lower performances in case of non-contiguous data-types.
+
+The "ob1" PML and BTL components use Open MPI's internal on-node
+ shared memory and process loopback devices for high performance.
The BTL interface allows multiple devices to be used simultaneously.
- For the MX BTL it is recommended that the first segment (which is
- as a threshold between the eager and the rendezvous protocol) should
- always be at most 4KB, but there is no further restriction on
- the size of subsequent fragments.
- The MX MTL is recommended in the common case for best performance
- on 10G hardware, when most of the data transfers cover contiguous
- memory layouts. The MX BTL is recommended in all other cases, more
- specifically when using multiple interconnects at the same time
- (including TCP), transferring non contiguous data-types or when
- using the DR PML.
+ For the MX BTL it is recommended that the first segment (which is as
+ a threshold between the eager and the rendezvous protocol) should
+ always be at most 4KB, but there is no further restriction on the
+ size of subsequent fragments.
+ The MX MTL is recommended in the common case for best performance on
+ 10G hardware when most of the data transfers cover contiguous memory
+ layouts. The MX BTL is recommended in all other cases, such as when
+ using multiple interconnects at the same time (including TCP), or
+ transferring non contiguous data-types.
+
===========================================================================
Building Open MPI
@@ -510,10 +944,28 @@
Open MPI will place its executables in <directory>/bin, its header
files in <directory>/include, its libraries in <directory>/lib, etc.
+--with-elan=<directory>
+ Specify the directory where the Quadrics Elan library and header
+ files are located. This option is generally only necessary if the
+ Elan headers and libraries are not in default compiler/linker
+ search paths.
+
+ Elan is the support library for Quadrics-based networks.
+
+--with-elan-libdir=<directory>
+ Look in directory for the Quadrics Elan libraries. By default, Open
+ MPI will look in <elan directory>/lib and <elan directory>/lib64,
+ which covers most cases. This option is only needed for special
+ configurations.
+
--with-gm=<directory>
Specify the directory where the GM libraries and header files are
- located. This enables GM support in Open MPI.
+ located. This option is generally only necessary if the GM headers
+ and libraries are not in default compiler/linker search paths.
+ GM is the support library for older Myrinet-based networks (GM has
+ been obsoleted by MX).
+
--with-gm-libdir=<directory>
Look in directory for the GM libraries. By default, Open MPI will
look in <gm directory>/lib and <gm directory>/lib64, which covers
@@ -521,59 +973,121 @@
--with-mx=<directory>
Specify the directory where the MX libraries and header files are
- located. This enables MX support in Open MPI.
+ located. This option is generally only necessary if the MX headers
+ and libraries are not in default compiler/linker search paths.
+ MX is the support library for Myrinet-based networks.
+
--with-mx-libdir=<directory>
Look in directory for the MX libraries. By default, Open MPI will
look in <mx directory>/lib and <mx directory>/lib64, which covers
most cases. This option is only needed for special configurations.
---with-mvapi=<directory>
- Specify the directory where the mVAPI libraries and header files are
- located. This enables mVAPI support in Open MPI (although it is
- deprecated).
-
---with-mvapi-libdir=<directory>
- Look in directory for the MVAPI libraries. By default, Open MPI will
- look in <mvapi directory>/lib and <mvapi directory>/lib64, which covers
- most cases. This option is only needed for special configurations.
-
--with-openib=<directory>
Specify the directory where the OpenFabrics (previously known as
- OpenIB) libraries and header files are located. This enables
- OpenFabrics support in Open MPI.
+ OpenIB) libraries and header files are located. This option is
+ generally only necessary if the OpenFabrics headers and libraries
+ are not in default compiler/linker search paths.
+ "OpenFabrics" refers to iWARP- and InifiniBand-based networks.
+
--with-openib-libdir=<directory>
Look in directory for the OpenFabrics libraries. By default, Open
MPI will look in <openib directory>/lib and <openib
directory>/lib64, which covers most cases. This option is only
needed for special configurations.
+--with-portals=<directory>
+ Specify the directory where the Portals libraries and header files
+ are located. This option is generally only necessary if the Portals
+ headers and libraries are not in default compiler/linker search
+ paths.
+
+ Portals is the support library for Cray interconnects, but is also
+ available on other platforms (e.g., there is a Portals library
+ implemented over regular TCP).
+
+--with-portals-config=<type>
+ Configuration to use for Portals support. The following <type>
+ values are possible: "utcp", "xt3", "xt3-modex" (default: utcp).
+
+--with-portals-libs=<libs>
+ Additional libraries to link with for Portals support.
+
--with-psm=<directory>
- Specify the directory where the QLogic PSM library and header files
- are located. This enables InfiniPath support in Open MPI.
+ Specify the directory where the QLogic InfiniPath PSM library and
+ header files are located. This option is generally only necessary
+ if the InfiniPath headers and libraries are not in default
+ compiler/linker search paths.
+ PSM is the support library for QLogic InfiniPath network adapters.
+
--with-psm-libdir=<directory>
Look in directory for the PSM libraries. By default, Open MPI will
look in <psm directory>/lib and <psm directory>/lib64, which covers
most cases. This option is only needed for special configurations.
+--with-sctp=<directory>
+ Specify the directory where the SCTP libraries and header files are
+ located. This option is generally only necessary if the SCTP headers
+ and libraries are not in default compiler/linker search paths.
+
+ SCTP is a special network stack over ethernet networks.
+
+--with-sctp-libdir=<directory>
+ Look in directory for the SCTP libraries. By default, Open MPI will
+ look in <sctp directory>/lib and <sctp directory>/lib64, which covers
+ most cases. This option is only needed for special configurations.
+
--with-udapl=<directory>
Specify the directory where the UDAPL libraries and header files are
- located. This enables UDAPL support in Open MPI. Note that UDAPL
- support is disabled by default on Linux; the --with-udapl flag must
- be specified in order to enable it.
+ located. Note that UDAPL support is disabled by default on Linux;
+ the --with-udapl flag must be specified in order to enable it.
+ Specifying the directory argument is generally only necessary if the
+ UDAPL headers and libraries are not in default compiler/linker
+ search paths.
+ UDAPL is the support library for high performance networks in Sun
+ HPC ClusterTools and on Linux OpenFabrics networks (although the
+ "openib" options are preferred for Linux OpenFabrics networks, not
+ UDAPL).
+
--with-udapl-libdir=<directory>
Look in directory for the UDAPL libraries. By default, Open MPI
will look in <udapl directory>/lib and <udapl directory>/lib64,
which covers most cases. This option is only needed for special
configurations.
+--with-lsf=<directory>
+ Specify the directory where the LSF libraries and header files are
+ located. This option is generally only necessary if the LSF headers
+ and libraries are not in default compiler/linker search paths.
+
+ LSF is a resource manager system, frequently used as a batch
+ scheduler in HPC systems.
+
+--with-lsf-libdir=<directory>
+ Look in directory for the LSF libraries. By default, Open MPI will
+ look in <lsf directory>/lib and <lsf directory>/lib64, which covers
+ most cases. This option is only needed for special configurations.
+
--with-tm=<directory>
Specify the directory where the TM libraries and header files are
- located. This enables PBS / Torque support in Open MPI.
+ located. This option is generally only necessary if the TM headers
+ and libraries are not in default compiler/linker search paths.
+ TM is the support library for the Torque and PBS Pro resource
+ manager systems, both of which are frequently used as a batch
+ scheduler in HPC systems.
+
+--with-sge
+ Specify to build support for the Sun Grid Engine (SGE) resource
+ manager. SGE support is disabled by default; this option must be
+ specified to build OMPI's SGE support.
+
+ The Sun Grid Engine (SGE) is a resource manager system, frequently
+ used as a batch scheduler in HPC systems.
+
--with-mpi-param_check(=value)
"value" can be one of: always, never, runtime. If --with-mpi-param
is not specified, "runtime" is the default. If --with-mpi-param
@@ -601,7 +1115,8 @@
--enable-progress-threads
Allows asynchronous progress in some transports. See
- --with-threads; this is currently disabled by default.
+ --with-threads; this is currently disabled by default. See the
+ above note about asynchronous progress.
--disable-mpi-cxx
Disable building the C++ MPI bindings. Note that this does *not*
@@ -654,7 +1169,7 @@
are built as dynamic shared objects (DSOs). This switch disables
this default; it is really only useful when used with
--enable-static. Specifically, this option does *not* imply
- --disable-shared; enabling static libraries and disabling shared
+ --enable-static; enabling static libraries and disabling shared
libraries are two independent options.
--enable-static
@@ -663,6 +1178,80 @@
--disable-shared; enabling static libraries and disabling shared
libraries are two independent options.
+--enable-sparse-groups
+ Enable the usage of sparse groups. This would save memory
+ significantly especially if you are creating large
+ communicators. (Disabled by default)
+
+--enable-peruse
+ Enable the PERUSE MPI data analysis interface.
+
+--enable-dlopen
+ Build all of Open MPI's components as standalone Dynamic Shared
+ Objects (DSO's) that are loaded at run-time. The opposite of this
+ option, --disable-dlopen, causes two things:
+
+ 1. All of Open MPI's components will be built as part of Open MPI's
+ normal libraries (e.g., libmpi).
+ 2. Open MPI will not attempt to open any DSO's at run-time.
+
+ Note that this option does *not* imply that OMPI's libraries will be
+ built as static objects (e.g., libmpi.a). It only specifies the
+ location of OMPI's components: standalone DSOs or folded into the
+ Open MPI libraries. You can control whenther Open MPI's libraries
+ are build as static or dynamic via --enable|disable-static and
+ --enable|disable-shared.
+
+--enable-heterogeneous
+ Enable support for running on heterogeneous clusters (e.g., machines
+ with different endian representations). Heterogeneous support is
+ disabled by default because it imposes a minor performance penalty.
+
+--enable-ptmalloc2-internal
+ ***NOTE: This option no longer exists.
+
+ This option was introduced in Open MPI v1.3 and was then removed in
+ Open MPI v1.3.2. Open MPI fundamentally changed how it uses
+ ptmalloc2 support in v1.3.2 such that the
+ --enable-ptmalloc2-internal flag was no longer necessary. It can
+ still harmlessly be supplied to Open MPI's configure script, but a
+ warning will appear about how it is an unrecognized option.
+
+ In v1.3 and v1.3.1, Open MPI built the ptmalloc2 library as a
+ standalone library that users could choose to link in or not (by
+ adding -lopenmpi-malloc to their link command). Using this option
+ restored pre-v1.3 behavior of *always* forcing the user to use the
+ ptmalloc2 memory manager (because it is part of libmpi).
+
+ Starting with v1.3.2, ptmalloc2 is always built into Open MPI, but
+ is only activated in certain scenarios.
+
+--with-wrapper-cflags=<cflags>
+--with-wrapper-cxxflags=<cxxflags>
+--with-wrapper-fflags=<fflags>
+--with-wrapper-fcflags=<fcflags>
+--with-wrapper-ldflags=<ldflags>
+--with-wrapper-libs=<libs>
+ Add the specified flags to the default flags that used are in Open
+ MPI's "wrapper" compilers (e.g., mpicc -- see below for more
+ information about Open MPI's wrapper compilers). By default, Open
+ MPI's wrapper compilers use the same compilers used to build Open
+ MPI and specify an absolute minimum set of additional flags that are
+ necessary to compile/link MPI applications. These configure options
+ give system administrators the ability to embed additional flags in
+ OMPI's wrapper compilers (which is a local policy decision). The
+ meanings of the different flags are:
+
+ <cflags>: Flags passed by the mpicc wrapper to the C compiler
+ <cxxflags>: Flags passed by the mpic++ wrapper to the C++ compiler
+ <fflags>: Flags passed by the mpif77 wrapper to the F77 compiler
+ <fcflags>: Flags passed by the mpif90 wrapper to the F90 compiler
+ <ldflags>: Flags passed by all the wrappers to the linker
+ <libs>: Flags passed by all the wrappers to the linker
+
+ There are other ways to configure Open MPI's wrapper compiler
+ behavior; see the Open MPI FAQ for more information.
+
There are many other options available -- see "./configure --help".
Changing the compilers that Open MPI uses to build itself uses the
@@ -692,6 +1281,12 @@
shell$ ./configure CC=mycc CXX=myc++ F77=myf77 F90=myf90 ...
+***Note: We generally suggest using the above command line form for
+ setting different compilers (vs. setting environment variables and
+ then invoking "./configure"). The above form will save all
+ variables and values in the config.log file, which makes
+ post-mortem analysis easier when problems occur.
+
It is required that the compilers specified be compile and link
compatible, meaning that object files created by one compiler must be
able to be linked with object files from the other compilers and
@@ -708,14 +1303,14 @@
Once Open MPI has been built and installed, it is safe to run "make
clean" and/or remove the entire build tree.
-VPATH builds are fully supported.
+VPATH and parallel builds are fully supported.
Generally speaking, the only thing that users need to do to use Open
MPI is ensure that <prefix>/bin is in their PATH and <prefix>/lib is
in their LD_LIBRARY_PATH. Users may need to ensure to set the PATH
and LD_LIBRARY_PATH in their shell setup files (e.g., .bashrc, .cshrc)
-so that rsh/ssh-based logins will be able to find the Open MPI
-executables.
+so that non-interactive rsh/ssh-based logins will be able to find the
+Open MPI executables.
===========================================================================
@@ -774,6 +1369,10 @@
actual compilation or linking of programs. The end result is an MPI
executable that is properly linked to all the relevant libraries.
+Customizing the behavior of the wrapper compilers is possible (e.g.,
+changing the compiler [not recommended] or specifying additional
+compiler/linker flags); see the Open MPI FAQ for more information.
+
===========================================================================
Running Open MPI Applications
@@ -783,9 +1382,7 @@
equivalent). For example:
shell$ mpirun -np 2 hello_world_mpi
-
or
-
shell$ mpiexec -np 1 hello_world_mpi : -np 1 hello_world_mpi
are equivalent. Some of mpiexec's switches (such as -host and -arch)
@@ -814,16 +1411,16 @@
will launch MPI_COMM_WORLD rank 0 on node1, rank 1 on node2, ranks 2
and 3 on node3, and ranks 4 through 7 on node4.
-Other starters, such as the batch scheduling environments, do not
-require hostfiles (and will ignore the hostfile if it is supplied).
-They will also launch as many processes as slots have been allocated
-by the scheduler if no "-np" argument has been provided. For example,
-running an interactive SLURM job with 8 processors:
+Other starters, such as the resource manager / batch scheduling
+environments, do not require hostfiles (and will ignore the hostfile
+if it is supplied). They will also launch as many processes as slots
+have been allocated by the scheduler if no "-np" argument has been
+provided. For example, running a SLURM job with 8 processors:
-shell$ srun -n 8 -A
-shell$ mpirun a.out
+shell$ salloc -n 8 mpirun a.out
-The above command will launch 8 copies of a.out in a single
+The above command will reserve 8 processors and run 1 copy of mpirun,
+which will, in turn, launch 8 copies of a.out in a single
MPI_COMM_WORLD on the processors that were allocated by SLURM.
Note that the values of component parameters can be changed on the
@@ -839,20 +1436,24 @@
component frameworks in Open MPI:
---------------------------------------------------------------------------
+
MPI component frameworks:
-------------------------
allocator - Memory allocator
bml - BTL management layer
-btl - MPI point-to-point byte transfer layer, used for MPI
+btl - MPI point-to-point Byte Transfer Layer, used for MPI
point-to-point messages on some types of networks
coll - MPI collective algorithms
+crcp - Checkpoint/restart coordination protocol
+dpm - MPI-2 dynamic process management
io - MPI-2 I/O
mpool - Memory pooling
mtl - Matching transport layer, used for MPI point-to-point
messages on some types of networks
osc - MPI-2 one-sided communications
pml - MPI point-to-point management layer
+pubsub - MPI-2 publish/subscribe management
rcache - Memory registration cache
topo - MPI topology routines
@@ -860,39 +1461,42 @@
---------------------------------------------------
errmgr - RTE error manager
-gpr - General purpose registry
+ess - RTE environment-specfic services
+filem - Remote file management
+grpcomm - RTE group communications
iof - I/O forwarding
-ns - Name server
+notifier - System/network administrator noficiation system
odls - OpenRTE daemon local launch subsystem
oob - Out of band messaging
-pls - Process launch system
+plm - Process lifecycle management
ras - Resource allocation system
-rds - Resource discovery system
rmaps - Resource mapping system
-rmgr - Resource manager
rml - RTE message layer
-schema - Name schemas
-sds - Startup / discovery service
-smr - State-of-health monitoring subsystem
+routed - Routing table for the RML
+snapc - Snapshot coordination
Miscellaneous frameworks:
-------------------------
-backtrace - Debugging call stack backtrace support
-maffinity - Memory affinity
-memory - Memory subsystem hooks
-memcpy - Memopy copy support
-memory - Memory management hooks
-paffinity - Processor affinity
-timer - High-resolution timers
+backtrace - Debugging call stack backtrace support
+carto - Cartography (host/network mapping) support
+crs - Checkpoint and restart service
+installdirs - Installation directory relocation services
+maffinity - Memory affinity
+memchecker - Run-time memory checking
+memcpy - Memopy copy support
+memory - Memory management hooks
+paffinity - Processor affinity
+timer - High-resolution timers
---------------------------------------------------------------------------
Each framework typically has one or more components that are used at
-run-time. For example, the btl framework is used by MPI to send bytes
-across underlying networks. The tcp btl, for example, sends messages
-across TCP-based networks; the gm btl sends messages across GM
-Myrinet-based networks.
+run-time. For example, the btl framework is used by the MPI layer to
+send bytes across different types underlying networks. The tcp btl,
+for example, sends messages across TCP-based networks; the openib btl
+sends messages across OpenFabrics-based networks; the MX btl sends
+messages across Myrinet networks.
Each component typically has some tunable parameters that can be
changed at run-time. Use the ompi_info command to check a component
@@ -959,6 +1563,12 @@
Found a bug? Got a question? Want to make a suggestion? Want to
contribute to Open MPI? Please let us know!
+When submitting questions and problems, be sure to include as much
+extra information as possible. This web page details all the
+information that we request in order to provide assistance:
+
+ http://www.open-mpi.org/community/help/
+
User-level questions and comments should generally be sent to the
user's mailing list (users at open-mpi.org). Because of spam, only
subscribers are allowed to post to this list (ensure that you
@@ -977,10 +1587,4 @@
http://www.open-mpi.org/mailman/listinfo.cgi/devel
-When submitting bug reports to either list, be sure to include as much
-extra information as possible. This web page details all the
-information that we request in order to provide assistance:
-
- http://www.open-mpi.org/community/help/
-
Make today an Open MPI day!
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/opensm_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/opensm_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/opensm_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -3,7 +3,7 @@
Version: OpenSM 3.2.x
Repo: git://git.openfabrics.org/~sashak/management.git
-Date: Dec 2008
+Date: May 2009
1 Overview
----------
@@ -239,6 +239,8 @@
OpenSM prints list of "Invalid Cached Option" error messages.
This does not affect OpenSM functionality.
+* SMs do not hand-over when running on ConnectX in a switch-based topology.
+
3 Unsupported IB Compliance Statements
--------------------------------------
The following section lists all the IB compliance statements which
@@ -320,6 +322,8 @@
* Don't startup automatically on SuSE based systems
+* Discovery bug, where some ports were leaved unlinked (without remote side).
+
4.2 Other Bug Fixes
* opensm/osm_console.c: fix seg fault when running "portstatus ca" in
@@ -402,6 +406,13 @@
* Other less critical or visible bugs were also fixed.
+* opensm: update LFTs when entering master
+
+* opensm: invalidate routing cache when entering master state
+
+* opensm/osm_port_info_rcv.c: don't clear sw->need_update if port 0 is active
+
+
5 Main Verification Flows
-------------------------
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/sdp_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/sdp_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/sdp_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,7 +1,7 @@
Open Fabrics Enterprise Distribution (OFED)
- SDP in OFED 1.4 Release Notes
+ SDP in OFED 1.4.1 Release Notes
- December 2008
+ May 2009
@@ -9,18 +9,19 @@
Table of Contents
===============================================================================
1. Overview
-2. Bug Fixes and Enhancements
-3. Known Issues
-4. Verification Applications/Flows/Tests
+2. Bug Fixes and Enhancements since OFED 1.3
+3. Bug Fixes and Enhancements since OFED 1.4
+4. Known Issues
+5. Verification Applications/Flows/Tests
===============================================================================
1. Overview
===============================================================================
-SDP in OFED is at GA level for OFED 1.4.
+SDP in OFED is at GA level for OFED 1.4.1
===============================================================================
-2. Bug Fixes and Enhancements
+2. Bug Fixes and Enhancements since OFED 1.3
===============================================================================
* Cleanup
- Compilation warnings
@@ -38,6 +39,16 @@
===============================================================================
+2. Bug Fixes and Enhancements since OFED 1.4
+===============================================================================
+SDP:
+- BUG1311 Netpipe fails with a IB_WC_LOC_LEN_ERR.
+- BUG1472 - clean socket timeouts and refcount when device is removed
+- BUG1502 - scheduling while atomic
+- BUG1309 - SDP close is slow + fix recv buffer initial size setting
+- BUG1087 - fixed recovery from failing rdma_create_qp()
+
+===============================================================================
3. Known Issues
===============================================================================
- BUG1444 - setsockopt(SO_RCVBUF) is not working in sdp socket. To limit top
@@ -49,12 +60,6 @@
- TCP allows connecting to IP_ANY - 0.0.0.0 (as a destination address!). SDP
does not allow - and will reject the connection.
-- BUG1309 - sometimes SDP close connection takes longer than TCP close.
-
-- BUG1256 - libsdp does not support epoll
-
-- BUG1087 - sometimes libsdp does not recover well when host is running out of QPs.
-
- Each SDP socket currently consumes up to 2 MBytes of memory. If this value
is high for your installation, it is possible to trade off performance
for lower memory utilization per socket by reducing the value of the
@@ -116,124 +121,12 @@
- Various Java client server applications (SUN:jre, BEA:jrockit/WebLogic, GNU:gij/gcj)
- Many UNIX utilities to verify that pre-load did not harm the applications
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Open Fabrics Enterprise Distribution (OFED)
- SDP in OFED 1.4 Release Notes
- December 2008
-
-
-
-===============================================================================
-Table of Contents
-===============================================================================
-1. Overview
-2. Bug Fixes and Enhancements
-3. Known Issues
-4. Verification Applications/Flows/Tests
-
-===============================================================================
-1. Overview
-===============================================================================
-SDP in OFED is at GA level for OFED 1.3.
-
-
-===============================================================================
-2. Bug Fixes and Enhancements
-===============================================================================
-* Fixes for SDP specification compliance
- - OOB data not marked as solicited (bug 596)
- - DisConn, ChRcvBuf, ChRcvBufAck marked solicited (bug 644)
- - Do not send DisConn if only 1 credit (bug 646)
- - Validate ChRcvBuf range (bug 647)
-
-* Cleanup
- - Compilation warnings
- - New kernel support
-
-* New function
- - SIOCOUTQ ioctl support
- - Add keepalive support
- - New /sys options: sdp_keepalive_probes_sent, sdp_keepalive_time
- - New options: SOCK_KEEPALIVE, TCP_KEEPIDLE
- - Add Zero copy bcopy support (bzcopy)
- - New /sys option: sdp_zcopy_thresh
-
-* Bugs fixed
- - Resize buffers if out of credits (bug 556)
- - Resize using skb_put (bug 620)
- - Move to accept queue on RTU drop and DREQ (bug 645)
- - Modify memory allocation to support in kernel users
- - Fix reference count but that prevents driver unload
- - connect() now allows AF_INET_SDP and AF_INET (bug 294)
- - poll() always returns POLLOUT on non-blocking socket (bug 829)
- - Executing netperf with TCP_CORK never ends (bug 837)
-
-
-===============================================================================
-3. Known Issues
-===============================================================================
-- Each SDP socket currently consumes up to 2 MBytes of memory. If this value
- is high for your installation, it is possible to trade off performance
- for lower memory utilization per socket by reducing the value of the
- "rcvbuf_scale" module parameter (default: 16).
-
- Note: the minimum legal value for this parameter is 1.
- At this parameter value, each socket will consume approximately 128 KBytes.
-
-- Small message size performance is low when messages are sent by client
- at a rate lower than the rate at which they are consumed by server,
- and when TCP_CORK is not set. This is observed, for example, with iperf
- benchmark. As a workaround, set the TCP_CORK socket option
- to ensure data is sent in at least 32K byte chunks.
-
-- Performance is low on 32-bit kernels, as SDP utilizes high memory
- to ease memory pressure. Moving to a 64-bit kernel solves this
- problem even if the application remains a 32-bit one.
-
-- By default, SDP utilizes a 2 Kbyte MTU size. This may cause PCI-X cards
- using Mellanox Technologies "Infinihost" HCAs to experience low bandwidth.
- Workaround: reset the MTU size to 1K in this situation, using either of
- the two methods below:
-
- 1. Activate the "tavor quirk" workaround in opensm:
- a. Create an opensm options cache file (/var/cache/osm/opensm.opts):
- > opensm --cache-options -o
- b. Add the following line to /var/cache/osm/opensm.opts:
- enable_quirks TRUE
- c. Rerun opensm using your usual command line options to activate
- the opensm quirk option.
-
- 2. Activate the "tavor quirk" workaround in cma:
- set the tavor_quirk module parameter of the rdma_cm module to value 1
- (default: 0).
-
-- The new BZCOPY mode is only effective for large block transfers.
- By setting the /sys parameter 'sdp_zcopy_thresh' to a non-zero value, a
- non-standard SDP speedup is enabled. All messages longer than
- 'sdp_zcopy_thresh' bytes in length will cause the user space buffer to
- be pinned and the data sent directly from the original buffer. This
- results in less CPU use and, on many systems, much better bandwidth.
- The default 64K value for 'sdp_zcopy_thresh' is sometimes too low for
- some systems. You must experiment with your hardware to select the
- best value.
-
-- Windows interoperability
- The Windows version of SDP does not support resizing buffers using the
- standard protocol messages. There will sometimes be inter-operability
- problems for this reason.
-
-===============================================================================
-4. Verification Applications/Flows/Tests
-===============================================================================
-See the corresponding section in the SDP release notes above.
-
-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Open Fabrics Enterprise Distribution (OFED)
- libsdp v. 9382 in OFED 1.4 Release Notes
+ libsdp v. 9382 in OFED 1.4.1 Release Notes
- December 2008
+ May 2009
===============================================================================
@@ -242,8 +135,9 @@
1. Overview
2. New Features
3. Bug Fixes
-4. Known Issues
-5. Verification Applications/Flows/Tests
+4. Bug Fixes and Enhancements since OFED 1.4
+5. Known Issues
+6. Verification Applications/Flows/Tests
===============================================================================
1. Overview
@@ -266,9 +160,8 @@
* Add libsdp-devel sub-package
-
===============================================================================
-3 Bug Fixes
+3. Bug Fixes
===============================================================================
The following list of bugs were fixed. Note that other less critical
or visible bugs were also fixed.
@@ -286,8 +179,18 @@
returning -1.
===============================================================================
-4. Known Issues
+4. Bug Fixes and Enhancements since OFED 1.4
===============================================================================
+libsdp:
+* Enable building libsdp on Solaris
+* BUG1256 - Add epoll support
+
+sdpnetstat:
+* BUF1513 - sdpnetstat is not showing all the listening processes on ipv6 sockets.
+
+===============================================================================
+5. Known Issues
+===============================================================================
* libsdp cannot provide its socket switch functionality for executables
statically linked with libc.
@@ -296,7 +199,7 @@
===============================================================================
-5. Verification Applications/Flows/Tests
+6. Verification Applications/Flows/Tests
===============================================================================
See the corresponding section in the SDP release notes above.
Modified: branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/uDAPL_release_notes.txt
===================================================================
--- branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/uDAPL_release_notes.txt 2009-05-29 14:04:56 UTC (rev 294)
+++ branches/ofed-1.4.1upgrade/ofed-docs/branches/upstream/current/uDAPL_release_notes.txt 2009-05-29 14:22:31 UTC (rev 295)
@@ -1,73 +1,165 @@
- Release Notes for
- OFED 1.4 DAPL Release
- December 2008
+ Release Notes for
+ OFED 1.4.1 DAPL Release
+ May 2009
+ OFED 1.4.1 RELEASE NOTES
- OFED 1.4 RELEASE NOTES
+ This release of the uDAPL reference implementation package for both
+ DAT 1.2 and 2.0 specification is timed to coincide with OFED release
+ of the Open Fabrics (www.openfabrics.org) software stack.
- This release of the DAPL reference implementation
- is timed to coincide with OFED release 1.3.1 of the
- Open Fabrics (www.openfabrics.org) software stack.
+ NEW SINCE OFED 1.4 - new versions of uDAPL v1 (1.2.14-1) and v2 (2.0.19-1)
- NEW SINCE OFED 1.3.1
-
- OFED 1.4 includes new versions compat-dapl-1.2.12-1, dapl-2.0.15-1
+ * New Features - optional counters, must be configured/built with -DDAPL_COUNTERS
- Summary of changes since OFED 1.3.1 release:
+ * Bug Fixes
- * New Features (scalability improvements - socket cm and UD support)
+ v2 - scm, cma: dat max_lmr_block_size is 32 bit, verbs max_mr_size is 64 bit
+ v2 - scm, cma: use direct SGE mappings from dat_lmr_triplet to ibv_sge
+ v2 - dtest: add flush EVD call after data transfer errors
+ v2 - scm: increase default MTU size from 1024 to 2048
+ v2 - dapltest: reset server listen ports to avoid collisions during long runs
+ v2 - dapltest: avoid duplicating ports, increment based on ep/thread count
+ v2 - dapltest: fix assumptions that multiple EP's will connect in order
+ v2 - common: sync missing with when removing items off of EVD pending queue
+ v2 - scm: reduce open time with thread start up
+ v2 - scm: getsockopt optlen needs initialized to size of optval
+ v2 - scm: cr_thread cleanup
+ v2 - OFED and WinOF code sync
+ v2 - scm: remove unnecessary query gid/lid from connection phase code.
+ v2 - scm: add optional 64-bit counters, build with -DDAPL_COUNTERS.
+ v1,v2 - spec files missing Requires(post) statements for sed/coreutils
+ v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test builds
+ v1,v2 - scm: remove unecessary thread when using direct objects
+ v1,v2 - Fix SuSE 11 build issues, asm/atomic.h no longer exists
- 1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages,
- assumes homogeneous cluster and will setup the QP's based on local
- HCA port attributes and exchanges QP information via socket's using
- the hostname of each node. IPoIB and rdma_cm are NOT required for
- this provider. QP attributes can be adjusted via the following
- environment parameters:
+ * Build Notes:
- DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 =268ms)
- DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds)
- DAPL_RNR_TIMER (default=12 5 bits, 12 = 64ms, 28 = 163ms, 31 = 491ms)
- DAPL_RNR_RETRY (default=7 3 bits, 7 = infinite)
- DAPL_IB_MTU (default=1024, limited to active MTU max)
+ # NON_DEBUG build/install example for x86_64, OFED targets
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make install
- The new socket cm entries in /etc/dat.conf provide a link to the actual
- HCA device and port. Example v1 and v2 entries for a Mellanox connectx
- device, port 1:
- - OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
- - ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+ # DEBUG build/install example for x86_64, using OFED targets
+ ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make install
+
+ # COUNTERS build/install example for x86_64, using OFED targets
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS"
+ make install
+
+ * BKM for running new DAPL library on your cluster without any impact on existing OFED installation:
+
+ Note: example for user /home/ardavis, (assumes /home/ardavis is exported) and MLX4 adapter, port 1
+
+ Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.19.tar.gz
+
+ untar in /home/ardavis
+ cd /home/ardavis/dapl-2.0.19
+ ./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries)
+
+ create /home/ardavis/dat.conf with following 2 lines. (entries with path to new libraries):
+
+ ofa-v2-ib0 u2.0 nonthreadsafe default /home/ardavis/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
+ ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/ardavis/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+
+ Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following:
+
+ setenv DAT_OVERRIDE=/home/ardavis/dat.conf
+
+ If running Intel MPI and uDAPL socket cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1
+
+ or if running Intel MPI and uDAPL rdma_cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0
+
+-------------------------
+
+ OFED 1.4 RELEASE NOTES
+
+ NEW SINCE OFED 1.3.1 - new versions of uDAPL v1 (1.2.12-1) and v2 (2.0.15-1)
+
+ * New Features
+
+ 1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages,
+ assumes homogeneous cluster and will setup the QP's based on local HCA port
+ attributes and exchanges QP information via socket's using the hostname of
+ each node. IPoIB and rdma_cm are NOT required for this provider. QP attributes
+ can be adjusted via the following environment parameters:
+
+ DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 == 268ms)
+ DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds)
+ DAPL_RNR_TIMER (default=12 5 bits, 12 == 64ms, 28 == 163ms, 31 == 491ms)
+ DAPL_RNR_RETRY (default=7 3 bits, 7 == infinite)
+ DAPL_IB_MTU (default=1024 limited to active MTU max)
+
+ The new socket cm entries in /etc/dat.conf provide a link to the actual HCA
+ device and port. Example v1 and v2 entries for a Mellanox connectx device, port 1:
+
+ OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
+ ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+
+ This new socket cm provider, was successfully tested on the TATA CRL cluster
+ (#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on
+ 1798 nodes, 14384 cores at ~76.9% of peak. DAPL_ACK_TIMER was increased to 21
+ for this scale.
+
+ 2. New v2 definitions for IB unreliable datagram extension (only supported in
+ scm provider, libdaploscm.so.2)
+
+ Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD
+ Add IB extension call dat_ib_post_send_ud().
+ Add address handle definition for UD calls.
+ Add IB event definitions to provide remote AH via connect and connect requests
+ See dtestx (-d) source for example usage model
+
+ * Bug Fixes
+
+ v1,v2 - dapltest: trans test moves to cleanup stage before rdma_read processing is complete
+ v1,v2 - Fix static registration (dat.conf) to include sysconfdir override
+ v1,v2 - dat.conf: add default iwarp entry for eth2
+ v1,v2 - dapl: adjust max_rdma_read_iov to 1 for iWARP devices
+ v1,v2 - dtest: reduce default IOV's for ep_create to support iWARP
+ v1,v2 - dtest: fix 32-bit build issues
+ v1,v2 - build: $(DESTDIR) prepend needed on install hooks for dat.conf
+ v2 - scm: UD shares EP;s which requires serialization
+ v2 - dapl: fixes for IB UD extensions in common code and socket cm provider.
+ v2 - dapl: add provider specific attribute query option for IB UD MTU size
+ v2 - dapl build: add correct CFLAGS, set non-debug build by default for v2
+ v2 - dtestx: fix stack corruption problem with hostname strcpy
+ v2 - dapl extension: dapli_post_ext should always allocate cookie for requests.
+ v2 - dapltest: manpage - rdma write example incorrect
+ v1,v2 - dat, dapl, dtest, dapltest, providers: fix compiler warnings in dat common code
+ v1,v2 - dapl cma: debug message during query needs definition for inet_ntoa
+ v1,v2 - dapl scm: fix corner case that delivers duplicate disconnect events
+ v1,v2 - dat: include stddef.h for NULL definition in dat_platform_specific.h
+ v1,v2 - dapl: add debug messages during async and overflow events
+ v1,v2 - dapltest: add check for duplicate disconnect events in transaction test
+ v1,v2 - dapl scm: use correct device attribute for max_rdma_read_out, max_qp_init_rd_atom
+ v1,v2 - dapl scm: change IB RC qp inline and timer defaults.
+ v1,v2 - dapl scm: add mtu adjustments via environment, default = 1024.
+ v1,v2 - dapl scm: change connect and accept to non-blocking to avoid blocking user thread.
+ v1,v2 - dapl scm: update max_rdma_read_iov, max_rdma_write_iov EP attributes during query
+ v1,v2 - dat: allow TYPE_ERR messages to be turned off with DAT_DBG_TYPE
+ v1,v2 - dapl: remove needless terminating 0 in dto_op_str functions.
+ v1,v2 - dat: remove reference to doc/dat.conf in makefile.am
+ v1,v2 - dapl scm: fix ibv_destroy_cq busy error condition during dat_evd_free.
+ v1,v2 - dapl scm: add stdout logging for uname and gethostbyname errors during open.
+ v1,v2 - dapl scm: support global routing and set mtu based on active_mtu
+ v1,v2 - dapl: add opcode to string function to report opcode during failures.
+ v1,v2 - dapl: remove unused iov buffer allocation on the endpoint
+ v1,v2 - dapl: endpoint pending request count is wrong
- 2. New v2 definitions for IB unreliable datagram extension
- (only supported in v2 scm provider, libdaploscm.so.2)
- - Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD
- - Add IB extension call dat_ib_post_send_ud().
- - Add address handle definition for UD calls.
- - Add IB event definitions to provide remote AH via connect
- and connect requests
- - See dtestx (-d) source for example usage model
-
- * Bug Fixes
+-------------------------
- v1,v2 - allow override of /etc/dat.conf via syscondir option
- v1,v2 - fix dapltest transaction test to avoid cleanup before rdma complete
- v1 - add ipath, ehca socket cm provider entries for v1.2, sync with v2.0
- v1,v2 - iWarp, 1 iov on rdma_reads, reduce iov's in dtest, add dat.conf entry
- v1,v2 - add $(DESTDIR) on install/uninstall hooks
- v2 - add new options to dtestx for UD testing
- v2 - IB UD fixes in common code/socket cm provider to allow multiple EP support
- v1,v2 - iWarp, 1 iov on rdma_reads, reduce iov's in dtest, add dat.conf entry
- v1,v2 - add $(DESTDIR) on install/uninstall hooks
- v2 - add new options to dtestx for UD testing
- v2 - IB UD fixes in common code/socket cm provider to allow multiple EP support
- v2 - fix dtest and dtestx build warnings
- v1,v2 - socket cm fixes, added DAPL_IB_MTU,
- changed default QP timers, include NULL definition.
- v1,v2 - Fix compiler warnings: dat, dapl, dtest, and dapltest
+ OFED 1.3.1 RELEASE NOTES
- NEW SINCE OFED 1.3
+ NEW SINCE OFED 1.3 - new versions of uDAPL v1 (1.2.7-1) and v2 (2.0.9-1)
+
+ * New Features - None
- OFED 1.3.1 includes new versions of uDAPL v1 (1.2.7-1) and v2 (2.0.9-1)
-
- Summary of changes since OFED 1.3 release:
+ * Bug Fixes
v2 - add private data exchange with reject
v1,v2 - better error reporting in non-debug builds
v1,v2 - update only OFA entries in dat.conf, cooperate with non-ofa providers
@@ -78,9 +170,14 @@
v1,v2 - long delay during dat_ia_open when DNS not configured
v1,v2 - use rdma_read_in/out from ep_attr per consumer instead of HCA max
- NEW SINCE OFED 1.2
+-------------------------
+ OFED 1.3 RELEASE NOTES
+
+ NEW SINCE OFED 1.2
+
* New Features
+
1. Add v2.0 library support for new 2.0 API Specification
2. Separate v1.2 library release to co-exist with v2.0 libraries.
3. New dat.conf with both 1.2 and 2.0 support
@@ -117,10 +214,10 @@
- dtest: typo in memset
- BUILD: v1 and v2 uDAPL source install/build instructions (redhat example):
+ BUILD: v1 and v2 uDAPL source install/build instructions (redhat example):
- # cd to distribution SRPMS directory
- cd /tmp/OFED-1.3/SRPMS
+ # cd to distribution SRPMS directory
+ cd /tmp/OFED-1.3/SRPMS
rpm -i dapl-1.2*.rpm
rpm -i dapl-2.0*.rpm
cd /usr/src/redhat/SOURCES
@@ -164,9 +261,12 @@
DAPL_DBG_TYPE_SRQ = 0x0800,
DAPL_DBG_TYPE_CNTR = 0x1000
+-------------------------
- NEW SINCE Gamma 3.2 and OFED 1.1
+ OFED 1.2 RELEASE NOTES
+ NEW SINCE Gamma 3.2 and OFED 1.1
+
* New Features
1. Added dtest and dapltest to the openfabrics build and utils rpm.
More information about the Pkg-ofed-commits
mailing list