[Pkg-ofed-commits] r437 - in /branches/ofed-1.4.2/ofed-docs/trunk: DEBIAN-HOWTO/...
gmpc-guest at alioth.debian.org
gmpc-guest at alioth.debian.org
Fri Aug 7 17:11:42 UTC 2009
Author: gmpc-guest
Date: Fri Aug 7 17:11:42 2009
New Revision: 437
URL: http://svn.debian.org/wsvn/pkg-ofed/?sc=1&rev=437
Log:
Add howto to the docs
Added:
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-1.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-10.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-11.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-2.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-3.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-4.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-5.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-6.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-7.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-8.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-9.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.html
branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.txt
Removed:
branches/ofed-1.4.2/ofed-docs/trunk/debian/dirs
Modified:
branches/ofed-1.4.2/ofed-docs/trunk/debian/docs
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-1.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-1.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-1.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-1.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,46 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Introduction</TITLE>
+ <LINK HREF="infiniband-howto-2.html" REL=next>
+
+ <LINK HREF="infiniband-howto.html#toc1" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-2.html">Next</A>
+Previous
+<A HREF="infiniband-howto.html#toc1">Contents</A>
+<HR>
+<H2><A NAME="s1">1.</A> <A HREF="infiniband-howto.html#toc1">Introduction</A></H2>
+
+<P>This document describes how to install and configure the OFED infiniband software on Debian. This document is intended
+to show you how to configure a simple Infiniband network as quickly as possible. It is not a replacement
+for the details documentation provided in the ofed-docs package!</P>
+
+<H2><A NAME="ss1.1">1.1</A> <A HREF="infiniband-howto.html#toc1.1">The latest version</A>
+</H2>
+
+<P>The latest version of the howto can be found on the pkg-ofed alioth webite:</P>
+<P>
+<A HREF="http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html">http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html</A></P>
+<P>Source is kept in the SVN repository:</P>
+<P>
+<A HREF="http://svn.debian.org/wsvn/pkg-ofed/">http://svn.debian.org/wsvn/pkg-ofed/</A></P>
+
+
+<H2><A NAME="ss1.2">1.2</A> <A HREF="infiniband-howto.html#toc1.2">What is OFED?</A>
+</H2>
+
+<P>OFED (OpenFabric's Enterprise Distribution) is the defacto Infiniband software stack on Linux. OFED
+provides a consistent set of kernel modules and userspace libraries which have been tested together.</P>
+<P>Further details of the Openfabrics Alliance and OFED can be found here
+<A HREF="http://www.openfabrics.org/">http://www.openfabrics.org</A></P>
+
+
+<HR>
+<A HREF="infiniband-howto-2.html">Next</A>
+Previous
+<A HREF="infiniband-howto.html#toc1">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-10.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-10.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-10.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-10.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,33 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Network Troubleshooting</TITLE>
+ <LINK HREF="infiniband-howto-11.html" REL=next>
+ <LINK HREF="infiniband-howto-9.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc10" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-11.html">Next</A>
+<A HREF="infiniband-howto-9.html">Previous</A>
+<A HREF="infiniband-howto.html#toc10">Contents</A>
+<HR>
+<H2><A NAME="s10">10.</A> <A HREF="infiniband-howto.html#toc10">Network Troubleshooting</A></H2>
+
+
+<H2><A NAME="ss10.1">10.1</A> <A HREF="infiniband-howto.html#toc10.1">ibdiagnet</A>
+</H2>
+
+<P>The ibdiagnet program can be used to troubleshoot potential issues with your infiniband fabric.</P>
+<P>
+<BLOCKQUOTE><CODE>
+ibdiagnet -r
+</CODE></BLOCKQUOTE>
+</P>
+
+<HR>
+<A HREF="infiniband-howto-11.html">Next</A>
+<A HREF="infiniband-howto-9.html">Previous</A>
+<A HREF="infiniband-howto.html#toc10">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-11.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-11.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-11.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-11.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,43 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Further Information</TITLE>
+ <LINK HREF="infiniband-howto-10.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc11" REL=contents>
+</HEAD>
+<BODY>
+Next
+<A HREF="infiniband-howto-10.html">Previous</A>
+<A HREF="infiniband-howto.html#toc11">Contents</A>
+<HR>
+<H2><A NAME="s11">11.</A> <A HREF="infiniband-howto.html#toc11">Further Information</A></H2>
+
+<P>Extensive documentation on the OFED software is present in the ofed-docs package.</P>
+<P>The openfabrics alliance webpage can be found here:</P>
+<P>
+<A HREF="http://www.openfabrics.org/">http://www.openfabrics.org/</A></P>
+
+<P>The following mailing lists are also useful:</P>
+<P>
+<A HREF="http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel">http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel</A>:
+pkg-ofed-devel: Discussion of debian specific problem or issues.</P>
+
+<P>
+<A HREF="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</A>:
+ofa-general: General discussion of the OFED software.</P>
+<P>Books:
+<PRE>
+Infiniband Network Architecture
+by MindShare, Inc.; Tom Shanley
+Publisher: Addison-Wesley Professional
+Pub Date: October 31, 2002
+Print ISBN-10: 0-321-11765-4
+</PRE>
+</P>
+<HR>
+Next
+<A HREF="infiniband-howto-10.html">Previous</A>
+<A HREF="infiniband-howto.html#toc11">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-2.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-2.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-2.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-2.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,136 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Installing the OFED Software</TITLE>
+ <LINK HREF="infiniband-howto-3.html" REL=next>
+ <LINK HREF="infiniband-howto-1.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc2" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-3.html">Next</A>
+<A HREF="infiniband-howto-1.html">Previous</A>
+<A HREF="infiniband-howto.html#toc2">Contents</A>
+<HR>
+<H2><A NAME="s2">2.</A> <A HREF="infiniband-howto.html#toc2">Installing the OFED Software</A></H2>
+
+<P>Before you can use your infiniband network you will need to install the OFED software on your infiniband client machines.
+You can choose to use the pre-build packages on alioth, or build your own packages straight from the alioth SVN repository.</P>
+<H2><A NAME="ss2.1">2.1</A> <A HREF="infiniband-howto.html#toc2.1">Installing prebuilt packages</A>
+</H2>
+
+
+<P>Add the following lines to your sources.list file:
+<BLOCKQUOTE><CODE>
+<PRE>
+deb http://pkg-ofed.alioth.debian.org/apt/ofed ./
+deb-src http://pkg-ofed.alioth.debian.org/apt/ofed ./
+</PRE>
+</CODE></BLOCKQUOTE>
+
+and run:
+<BLOCKQUOTE><CODE>
+<PRE>
+aptitude update
+aptitude install ofed
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss2.2">2.2</A> <A HREF="infiniband-howto.html#toc2.2">Building packages from source</A>
+</H2>
+
+<P>If you wish to build the OFED packages from the alioth svn repository, use the following procedure.</P>
+
+<H3>Install the prerequisites development packages</H3>
+
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+aptitude install svn-buildpackage build-essential devscripts
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<H3>Checkout the svn tree</H3>
+
+<P>
+<BLOCKQUOTE><CODE>
+svn co svn://svn.debian.org/pkg-ofed/
+</CODE></BLOCKQUOTE>
+</P>
+
+<H3>Install the upstream source (optional)</H3>
+
+<P>The upstream source tarballs need to be available if you
+want to build pukka debian packages suitable for inclusion
+upstream. If you are simply building packages for your own use,
+you can ignore this step.
+<BLOCKQUOTE><CODE>
+<PRE>
+cd pkg-ofed
+mkdir tarballs
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>Populate the tarballs with the *.orig.tar.gz files available form
+the "upstream source" release on
+<A HREF="https://alioth.debian.org/frs/?group_id=100311">https://alioth.debian.org/frs/?group_id=100311</A></P>
+
+<H3>Build the packages.</H3>
+
+<P>cd into the package you wish to build. eg for libibcommon,
+<BLOCKQUOTE><CODE>
+cd pkg-ofed/libibcommon
+</CODE></BLOCKQUOTE>
+
+Link in the upstream tarballs directory (optional)
+<BLOCKQUOTE><CODE>
+ln -s -f ../tarballs .
+</CODE></BLOCKQUOTE>
+
+Run svn-buildpackage from within the trunk directory.
+<BLOCKQUOTE><CODE>
+<PRE>
+ cd pkg-ofed/libibcommon/trunk
+ svn-buildpackage -uc -us -rfakeroot
+</PRE>
+</CODE></BLOCKQUOTE>
+
+The build process will generate a deb in the build-area directory. </P>
+<P>Repeat the process for the rest of the packages. Note that some packages have build dependancies on other OFED packages. The suggested build order is:
+<BLOCKQUOTE><CODE>
+<PRE>
+ libibcm
+ libibcommon
+ libibumad
+ libibmad
+ libnes
+ libsdp
+ dapl
+ opensm
+ infiniband-diags
+ ibutils
+ mstflint
+ perftest
+ qlvnictools
+ qpert
+ rds-tools
+ sdpnetstat
+ srptools
+ tvflash
+ ibsim
+ ofed-docs
+ ofa_kernel
+ ofed
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+
+
+<HR>
+<A HREF="infiniband-howto-3.html">Next</A>
+<A HREF="infiniband-howto-1.html">Previous</A>
+<A HREF="infiniband-howto.html#toc2">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-3.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-3.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-3.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-3.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,80 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Install the kernel modules</TITLE>
+ <LINK HREF="infiniband-howto-4.html" REL=next>
+ <LINK HREF="infiniband-howto-2.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc3" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-4.html">Next</A>
+<A HREF="infiniband-howto-2.html">Previous</A>
+<A HREF="infiniband-howto.html#toc3">Contents</A>
+<HR>
+<H2><A NAME="s3">3.</A> <A HREF="infiniband-howto.html#toc3">Install the kernel modules</A></H2>
+
+<P>You now need to build a set of OFED kernel modules which match the version of the OFED software you have installed.</P>
+<P>The Debian kernel contains a set of OFED infiniband drivers, but they may not match the OFED userspace version have installed.
+Consult the table below to determine what OFED version the Debian kernel contains. </P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+Debian Kernel Version OFED Version
+<=2.6.26 1.3
+>=2.6.27 1.4
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<P>If the debian kernel modules are the incorrect version, you can build a new set of modules using the ofa-kernel-source package.
+If your kernel already includes the correct OFED kernel modules you can skip the rest of this section. If you are in doubt, you should
+build a new set of modules rather than relying on the modules shipped with the kernel.</P>
+
+<H2><A NAME="ss3.1">3.1</A> <A HREF="infiniband-howto.html#toc3.1">Building new kernel modules</A>
+</H2>
+
+<P>You can build new kernel modules using module-assistant.
+<BLOCKQUOTE><CODE>
+<PRE>
+aptitude install module-assistant
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>Ensure you have the ofa-kernel-source package installed, and then run:
+<BLOCKQUOTE><CODE>
+<PRE>
+ module-assistant prepare
+ module-assistant clean ofa-kernel
+ module-assistant build ofa-kernel
+</PRE>
+</CODE></BLOCKQUOTE>
+
+This will create a deb which you can then install. As the deb contains replacements for existing kernel modules you will need to either manually remove
+any infiniband modules which have already been loaded, or reboot the machine, before you can use the new modules. </P>
+<P>The new kernel modules will be installed into /usr/lib/<kernel-version>/updates. They will not overwrite the original kernel modules, but the module
+loader will pick up the modules from the updates directory in preference. You can verify that the system is using the new kernel modules by running the
+modinfo command.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+# modinfo ib_core
+filename: /lib/modules/2.6.22.19/updates/kernel/drivers/infiniband/core/ib_core.ko
+author: Roland Dreier
+description: core kernel InfiniBand API
+license: Dual BSD/GPL
+vermagic: 2.6.22.19 SMP mod_unload
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<P>Note that if you wish to rebuild the kernel modules (eg for a new kernel version) then you must issue
+the module-assistant clean command before trying a new build.</P>
+
+
+<HR>
+<A HREF="infiniband-howto-4.html">Next</A>
+<A HREF="infiniband-howto-2.html">Previous</A>
+<A HREF="infiniband-howto.html#toc3">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-4.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-4.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-4.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-4.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,279 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Setting up a basic infiniband network </TITLE>
+ <LINK HREF="infiniband-howto-5.html" REL=next>
+ <LINK HREF="infiniband-howto-3.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc4" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-5.html">Next</A>
+<A HREF="infiniband-howto-3.html">Previous</A>
+<A HREF="infiniband-howto.html#toc4">Contents</A>
+<HR>
+<H2><A NAME="s4">4.</A> <A HREF="infiniband-howto.html#toc4">Setting up a basic infiniband network </A></H2>
+
+<P>This sections describes how to set up a basic infiniband network and test its functionality.</P>
+
+<H2><A NAME="ss4.1">4.1</A> <A HREF="infiniband-howto.html#toc4.1">Upgrade your Infiniband card and switch firmware</A>
+</H2>
+
+<P>Before proceeding you should ensure that the firmware in your switches and infiniband cards is at the latest release.
+Older firmware versions may cause interoperability and fabric stability issues. Do not assume that just because your
+hardware has come fresh from the factory that it has the latest firmware on it. </P>
+<P>You should follow the documentation from your vendor as to how the firmware should be updated.</P>
+
+<H2><A NAME="ss4.2">4.2</A> <A HREF="infiniband-howto.html#toc4.2">Physically Connect the network</A>
+</H2>
+
+<P>Connect up to your hosts and switches.</P>
+
+<H2><A NAME="ss4.3">4.3</A> <A HREF="infiniband-howto.html#toc4.3">Choose a Subnet Manager</A>
+</H2>
+
+<P>Each infiniband network requires a subnet manager. You can choose to run the OFED opensm subnet manager on one of the
+Linux clients, or you may choose to use an embedded subnet manager running on one of the switches in your fabric. Note
+that not all switches come with a subnet manager; check your switch documentation.</P>
+
+
+<H2><A NAME="ss4.4">4.4</A> <A HREF="infiniband-howto.html#toc4.4">Load the kernel modules</A>
+</H2>
+
+<P>Infiniband kernel modules are not loaded automatically. You should adding them to /etc/modules so that they are automatically loaded on machine
+bootup. You will need to include the hardware specific modules and the protocol modules.</P>
+
+<P>/etc/modules:
+<PRE>
+# Hardware drivers
+# Choose the apropriate modules from
+# /lib/modules/<kernel-version>/updates/kernel/drivers/infiniband/hw
+#
+#mlx4_ib # Mellanox ConnectX cards
+#ib_mthca # some mellanox cards
+#iw_cxgb3 # Chelsio T3 cards
+#iw_nes # NetEffect cards
+#
+# Protocol modules
+# Common modules
+rdma_ucm
+ib_umad
+ib_uverbs
+# IP over IB
+ib_ipoib
+# scsi over IB
+ib_srp
+# IB SDP protocol
+ib_sdp
+</PRE>
+</P>
+
+
+<H2><A NAME="ss4.5">4.5</A> <A HREF="infiniband-howto.html#toc4.5">(optional) Start opensm</A>
+</H2>
+
+<P>If you are going to use the opensm suetnet manager, edit /etc/default/opensm and add the port
+GUIDs of the interfaces on which you wish to start opensm. </P>
+<P>You can find the port GUIDs of your cards with the ibstat -p command:
+<BLOCKQUOTE><CODE>
+<PRE>
+# ibstat -p
+0x0002c9030002fb05
+0x0002c9030002fb06
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>/etc/default/opensm:
+<BLOCKQUOTE><CODE>
+<PRE>
+PORTS="0x0002c9030002fb05 0x0002c9030002fb06"
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>Note if you want to start opensm on all ports you can use the PORTS="ALL" keyword.</P>
+<P>Start opensm:</P>
+<P>
+<PRE>
+#/etc/init.d/opensm start
+</PRE>
+</P>
+<P>If opensm has started correctly you should see SUBNET UP messages in the opensm logfile (/var/log/opensm.<PORTID>.log).</P>
+<P>
+<PRE>
+Mar 04 14:56:06 600685 [4580A960] 0x02 -> SUBNET UP
+</PRE>
+</P>
+<P>Note that you can start opensm on multiple nodes; one node will be the active subnet manager and the others will put themselves into standby.</P>
+
+
+<H2><A NAME="ss4.6">4.6</A> <A HREF="infiniband-howto.html#toc4.6">Check network health</A>
+</H2>
+
+<P>You can now check the status of the local IB link with the ibstat command. Connected links should be in the "LinkUp" state. The following
+output is from a dual ported card, only one of which (port1) is connected.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+# ibstat
+CA 'mlx4_0'
+ CA type: MT25418
+ Number of ports: 2
+ Firmware version: 2.3.0
+ Hardware version: a0
+ Node GUID: 0x0002c9030002fb04
+ System image GUID: 0x0002c9030002fb07
+ Port 1:
+ State: Active
+ Physical state: LinkUp
+ Rate: 20
+ Base lid: 2
+ LMC: 0
+ SM lid: 1
+ Capability mask: 0x02510868
+ Port GUID: 0x0002c9030002fb05
+ Port 2:
+ State: Down
+ Physical state: Polling
+ Rate: 10
+ Base lid: 0
+ LMC: 0
+ SM lid: 0
+ Capability mask: 0x02510868
+ Port GUID: 0x0002c9030002fb06
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss4.7">4.7</A> <A HREF="infiniband-howto.html#toc4.7">Check the extended network connectivity</A>
+</H2>
+
+<P>Once the host is connected to the infiniband network you can check the health of all of the other network components with the ibhosts, ibswitches and iblinkinfo commands.</P>
+<P>ibhosts displays all of the hosts visible on the network.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+# ibhosts
+Ca : 0x0008f1040399d3d0 ports 2 "Voltaire HCA400Ex-D"
+Ca : 0x0008f1040399d370 ports 2 "Voltaire HCA400Ex-D"
+Ca : 0x0008f1040399d3fc ports 2 "Voltaire HCA400Ex-D"
+Ca : 0x0008f1040399d3f4 ports 2 "Voltaire HCA400Ex-D"
+Ca : 0x0002c9030002faf4 ports 2 "MT25408 ConnectX Mellanox Technologies"
+Ca : 0x0002c9030002fc0c ports 2 "MT25408 ConnectX Mellanox Technologies"
+Ca : 0x0002c9030002fc10 ports 2 "MT25408 ConnectX Mellanox Technologies"
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>ibswitches will display all of the switches in the network.
+<BLOCKQUOTE><CODE>
+<PRE>
+# ibswitches
+Switch : 0x0008f104004121fa ports 24 "ISR9024D-M Voltaire" enhanced port 0 lid 1 lmc 0
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>iblinkinfo will show the status and speed of all of the links in the network.
+<BLOCKQUOTE><CODE>
+<PRE>
+#iblinkinfo.pl
+Switch 0x0008f104004121fa ISR9024D-M Voltaire:
+ 1 1[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 2 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 2[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 13 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 3[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 4 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 4[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 26 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 5[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 27 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 6[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 24 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 7[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 28 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 8[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 25 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 9[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 31 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 10[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 32 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 11[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 33 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 12[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 29 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 13[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 30 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 14[ ] ==( 4X 2.5 Gbps Down / Polling)==> [ ] "" ( )
+ 1 15[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 3 1[ ] "Voltaire HCA400Ex-D" ( )
+ 1 16[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 10 1[ ] "Voltaire HCA400Ex-D" ( )
+ 17[ ] ==( 4X 2.5 Gbps Down / Polling)==> [ ] "" ( )
+ 18[ ] ==( 4X 2.5 Gbps Down / Polling)==> [ ] "" ( )
+ 1 19[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 7 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 20[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 6 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 21[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 5 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 22[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 21 1[ ] "Voltaire HCA400Ex-D" ( )
+ 1 23[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 9 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 24[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 8 1[ ] "Voltaire HCA400Ex-D" ( )
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss4.8">4.8</A> <A HREF="infiniband-howto.html#toc4.8">testing connectivity with ibping</A>
+</H2>
+
+<P>ibping is an infiniband equivalent to the icmp ping command. Choose a node on the fabric and run a ibping server:
+<BLOCKQUOTE><CODE>
+#ibping -S
+</CODE></BLOCKQUOTE>
+</P>
+<P>Choose another node on your network, and then ping the port GUID of the server. (ibstat on the server will list the port GUID).</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+#ibping -G 0x0002c9030002fc1d
+Pong from test.example.com (Lid 13): time 0.072 ms
+Pong from test.example.com (Lid 13): time 0.043 ms
+Pong from test.example.com (Lid 13): time 0.045 ms
+Pong from test.example.com (Lid 13): time 0.045 ms
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss4.9">4.9</A> <A HREF="infiniband-howto.html#toc4.9">Testing RDMA performance</A>
+</H2>
+
+
+<P>You can test the latency and bandwidth of a link with the ib_rdma_lat commands.</P>
+<P>To test the latency, start the server on a node:
+<BLOCKQUOTE><CODE>
+#ib_rdma_lat
+</CODE></BLOCKQUOTE>
+
+and then start a client on another node, giving it the hostname of the server.
+<BLOCKQUOTE><CODE>
+<PRE>
+#ib_rdma_lat hostname-of-server
+ local address: LID 0x0d QPN 0x18004a PSN 0xca58c4 RKey 0xda002824 VAddr 0x00000000509001
+ remote address: LID 0x02 QPN 0x7c004a PSN 0x4b4eba RKey 0x82002466 VAddr 0x00000000509001
+Latency typical: 1.15193 usec
+Latency best : 1.13094 usec
+Latency worst : 5.48519 usec
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>You can test the bandwith of the link using the ib_rdma_bw command.
+<BLOCKQUOTE><CODE>
+#ib_rdma_bw
+</CODE></BLOCKQUOTE>
+
+and then start a client on another node, giving it the hostname of the server.
+<BLOCKQUOTE><CODE>
+<PRE>
+#ib_rdma_bw hostname-of-server
+855: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
+855: Local address: LID 0x0d, QPN 0x1c004a, PSN 0xbf60dd RKey 0xde002824 VAddr 0x002aea4092b000
+855: Remote address: LID 0x02, QPN 0x004a, PSN 0xaad03c, RKey 0x86002466 VAddr 0x002b8a4e191000
+
+
+855: Bandwidth peak (#0 to #955): 1486.85 MB/sec
+855: Bandwidth average: 1486.47 MB/sec
+855: Service Demand peak (#0 to #955): 1970 cycles/KB
+855: Service Demand Avg : 1971 cycles/KB
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>The perftest package contains a number of other similar benchmarking programs to test various aspects of your network.</P>
+
+
+<HR>
+<A HREF="infiniband-howto-5.html">Next</A>
+<A HREF="infiniband-howto-3.html">Previous</A>
+<A HREF="infiniband-howto.html#toc4">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-5.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-5.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-5.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-5.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,124 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: IP over Infiniband (IPoIB)</TITLE>
+ <LINK HREF="infiniband-howto-6.html" REL=next>
+ <LINK HREF="infiniband-howto-4.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc5" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-6.html">Next</A>
+<A HREF="infiniband-howto-4.html">Previous</A>
+<A HREF="infiniband-howto.html#toc5">Contents</A>
+<HR>
+<H2><A NAME="s5">5.</A> <A HREF="infiniband-howto.html#toc5">IP over Infiniband (IPoIB)</A></H2>
+
+<P>The OFED stack allows you to run TCP/IP over your infiniband network, allowing you to run non-infiniband aware applications across
+your network. Several native infiniband applications also use IPoIB for host resolution (eg Lustre and SDP).</P>
+
+<H2><A NAME="ss5.1">5.1</A> <A HREF="infiniband-howto.html#toc5.1">List the network devices</A>
+</H2>
+
+<P>Check that the IBoIP modules is loaded.</P>
+<P>
+<BLOCKQUOTE><CODE>
+#modprobe ib_ipoib
+</CODE></BLOCKQUOTE>
+
+You will now have an "ib" network interface for each of your infiniband cards.
+<BLOCKQUOTE><CODE>
+<PRE>
+#ifconfig -a
+
+<snip>
+ib0 Link encap:UNSPEC HWaddr 80-06-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
+ BROADCAST MULTICAST MTU:2044 Metric:1
+ RX packets:0 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:256
+ RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
+
+ib1 Link encap:UNSPEC HWaddr 80-06-00-49-FE-80-00-00-00-00-00-00-00-00-00-00
+ BROADCAST MULTICAST MTU:2044 Metric:1
+ RX packets:0 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:256
+ RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
+<snip>
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss5.2">5.2</A> <A HREF="infiniband-howto.html#toc5.2">IP Configuration</A>
+</H2>
+
+<P>You can now configure the ib network devices using /etc/network/interfaces.
+<BLOCKQUOTE><CODE>
+<PRE>
+auto ib0
+iface ib0 inet static
+ address 172.31.128.50
+ netmask 255.255.240.0
+ broadcast 172.31.143.255
+</PRE>
+</CODE></BLOCKQUOTE>
+
+Bring the network device up, as normal.
+<BLOCKQUOTE><CODE>
+ifup ib0
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss5.3">5.3</A> <A HREF="infiniband-howto.html#toc5.3">Connected vs Unconnected Mode</A>
+</H2>
+
+<P>IPoIB can run over two infiniband transports, Unreliable Datagram (UD) mode or Connected mode (CM). The difference between
+these two modes are described in:
+<PRE>
+RFC4392 - IP over InfiniBand (IPoIB) Architecture
+RFC4391 - Transmission of IP over InfiniBand (IPoIB) (UD mode)
+RFC4755 - IP over InfiniBand: Connected Mode
+</PRE>
+
+ADDME: Pro/cons of these two methods?</P>
+<P>You can switch between these two mode at runtime with:
+<BLOCKQUOTE><CODE>
+<PRE>
+
+ echo datagram > /sys/class/net/ibX/mode
+ echo connected > /sys/class/net/ibX/mode
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>The default is datagram (UD) mode. If you with to use CM then you can add a script to /etc/network/interfaces/if-up.d to
+automatically set CM mode on your interfaces when they are configured.</P>
+
+
+<H2><A NAME="ss5.4">5.4</A> <A HREF="infiniband-howto.html#toc5.4">TCP tuning</A>
+</H2>
+
+<P>In order to obtain maximum IPoIB throughput you may need to tweak the MTU and various kernel TCP buffer and window settings.
+See the details in the ipoib_release_notes.txt document in the ofed-docs package.</P>
+
+<H2><A NAME="ss5.5">5.5</A> <A HREF="infiniband-howto.html#toc5.5">ARP and dual ported cards.</A>
+</H2>
+
+<P>If you have a dual ported card with both ports on the same IB subnet but a different IP subnet, you
+will need to tweak the ARP settings for the IPoIB interfaces. See ipoib_release_notes.txt in the ofed-docs package for a full
+discussion of this issue.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+ sysctl -w net.ipv4.conf.ib0.arp_ignore=1
+ sysctl -w net.ipv4.conf.ib1.arp_ignore=1
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<HR>
+<A HREF="infiniband-howto-6.html">Next</A>
+<A HREF="infiniband-howto-4.html">Previous</A>
+<A HREF="infiniband-howto.html#toc5">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-6.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-6.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-6.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-6.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,156 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: OpenMPI</TITLE>
+ <LINK HREF="infiniband-howto-7.html" REL=next>
+ <LINK HREF="infiniband-howto-5.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc6" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-7.html">Next</A>
+<A HREF="infiniband-howto-5.html">Previous</A>
+<A HREF="infiniband-howto.html#toc6">Contents</A>
+<HR>
+<H2><A NAME="s6">6.</A> <A HREF="infiniband-howto.html#toc6">OpenMPI</A></H2>
+
+
+<P>The section describes how to configure OpenMPI to use Infiniband.</P>
+
+<H2><A NAME="ss6.1">6.1</A> <A HREF="infiniband-howto.html#toc6.1">Configure IPoIB</A>
+</H2>
+
+<P>OpenMPI uses IPoIB for job startup and tear-down. You should configure IPoIB on all of your hosts.</P>
+
+<H2><A NAME="ss6.2">6.2</A> <A HREF="infiniband-howto.html#toc6.2">Load the modules</A>
+</H2>
+
+<P>Ensure the rdma_ucm module is loaded.
+<BLOCKQUOTE><CODE>
+modprobe rdma_ucm
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss6.3">6.3</A> <A HREF="infiniband-howto.html#toc6.3">Check permissions and limits</A>
+</H2>
+
+<P>Uses who want to run MPI jobs will need to have write permissions for the following devices:
+<BLOCKQUOTE><CODE>
+<PRE>
+ /dev/infiniband/uverbs*
+/dev/infiniband/rdma_cm*
+</PRE>
+</CODE></BLOCKQUOTE>
+
+The simplest way to do this is to add the users to the rdma group. If that is not suitiable for
+your site, you can change the permissions and ownership of these devices by editing the following
+udev rules:
+<BLOCKQUOTE><CODE>
+<PRE>
+/etc/udev/rules.d/50-udev.rules
+/etc/udev/rules.d/91-permissions.rules
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<P>OpenMPI will need to pin memory. Edit /etc/security/limits.conf and add the line:
+<BLOCKQUOTE><CODE>
+* hard memlock unlimited
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss6.4">6.4</A> <A HREF="infiniband-howto.html#toc6.4">Install the mpi test programs</A>
+</H2>
+
+<P>Check the mpitests package is installed.
+<BLOCKQUOTE><CODE>
+aptitude install mpitests
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss6.5">6.5</A> <A HREF="infiniband-howto.html#toc6.5">Configure Hosts</A>
+</H2>
+
+<P>OpenMPI uses ssh to spawn jobs on remote hosts. You should configure a public/private keypair to ensure that you
+can ssh between hosts without entering a password. You should also ensure that your login process is silent.</P>
+<P>Choose two hosts on which to test the program and put their hostnames into a file called hostfile:</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+ hostA slots=1
+ hostB slots=1
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss6.6">6.6</A> <A HREF="infiniband-howto.html#toc6.6">Run the MPI PingPong benchmark.</A>
+</H2>
+
+
+<P>We will use the MPI PingPong benchmark for our testing. By default, openmpi should use inifiniband networks in preference to any tcp networks it finds. However, we will force mpi to be extra-chatty during the test to ensure that we are really using the infiniband interfaces.</P>
+<P>(ADDME: Is there a better way to confirm which networks openmpi is using?)</P>
+
+<P>
+<PRE>
+mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile /path/to/hostfile IMB-MPI1 PingPong
+</PRE>
+</P>
+<P>If all goes well you should see openib debugging messages from both hosts, together with the job output.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+<snip>
+# PingPong
+[HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+[HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+[HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+[HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+
+#---------------------------------------------------
+# Benchmarking PingPong
+# #processes = 2
+#---------------------------------------------------
+ #bytes #repetitions t[usec] Mbytes/sec
+ 0 1000 1.53 0.00
+ 1 1000 1.44 0.66
+ 2 1000 1.42 1.34
+ 4 1000 1.41 2.70
+ 8 1000 1.48 5.15
+ 16 1000 1.50 10.15
+ 32 1000 1.54 19.85
+ 64 1000 1.79 34.05
+ 128 1000 3.01 40.56
+ 256 1000 3.56 68.66
+ 512 1000 4.46 109.41
+ 1024 1000 5.37 181.92
+ 2048 1000 8.13 240.25
+ 4096 1000 10.87 359.48
+ 8192 1000 15.97 489.17
+ 16384 1000 30.54 511.68
+ 32768 1000 55.01 568.12
+ 65536 640 122.20 511.46
+ 131072 320 207.20 603.27
+ 262144 160 377.10 662.96
+ 524288 80 706.21 708.00
+ 1048576 40 1376.93 726.25
+ 2097152 20 1946.00 1027.75
+ 4194304 10 3119.29 1282.34
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>If you encounter any errors read the excellent OpenMPI troubleshooting guide.
+<A HREF="http://www.openmpi.org">http://www.openmpi.org</A></P>
+<P>If you want to compare infiniband performance with your ethernet/TCP networks, you can re-run the tests using flags to tell openmpi to use your ethernet network. (The example below assumes that your test nodes are connected via eth0).</P>
+<P>
+<PRE>
+ mpirun --mca btl ^openib --mca btl_tcp_if_include eth0 --hostfile hostfile -n 2 IMB-MPI1 -benchmark PingPong
+</PRE>
+</P>
+
+
+<HR>
+<A HREF="infiniband-howto-7.html">Next</A>
+<A HREF="infiniband-howto-5.html">Previous</A>
+<A HREF="infiniband-howto.html#toc6">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-7.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-7.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-7.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-7.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,100 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: SDP</TITLE>
+ <LINK HREF="infiniband-howto-8.html" REL=next>
+ <LINK HREF="infiniband-howto-6.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc7" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-8.html">Next</A>
+<A HREF="infiniband-howto-6.html">Previous</A>
+<A HREF="infiniband-howto.html#toc7">Contents</A>
+<HR>
+<H2><A NAME="s7">7.</A> <A HREF="infiniband-howto.html#toc7">SDP</A></H2>
+
+<P>Sockets Direct Protocol (SDP) is a network protocol which provides an RDMA accelerated
+alternative to TCP over infiniband networks. OFED provides an LD_PRELOADable library
+(libsdp.so) which allows programs which use TCP to use the more efficient SDP protocol instead.
+The use of an LD_PRELOADable libary means that the switch in protocol is transparent,
+and does not require the application to be recompiled.</P>
+
+
+<H2><A NAME="ss7.1">7.1</A> <A HREF="infiniband-howto.html#toc7.1">Configuration</A>
+</H2>
+
+<P>SDP used IPoIB for address resolution, so you must configure IPoIB before using SDP. </P>
+<P>You should also ensure the ib_sdp kernel module is installed.
+<PRE>
+modprobe ib_sdp
+</PRE>
+</P>
+
+<P>You can use libsdp in two ways; you can either manually LD_PRELOAD the library whilst invoking your application, or
+create a config file which specifies which applications will use SDP.</P>
+<P>To manually LD_PRELOAD a library, simply set the LD_PRELOAD variable before invoking your application.
+<PRE>
+LD_PRELOAD=libsdp.so ./path/to/your/application ...
+</PRE>
+
+If you which to choose which programs will use SDP you can edit /etc/sdp.conf and specify which programs, ports and
+addresses are eligible for use.</P>
+
+
+<H2><A NAME="ss7.2">7.2</A> <A HREF="infiniband-howto.html#toc7.2">Example Using SDP with Netpipe</A>
+</H2>
+
+<P>The following example shows how to use libsdp to make the TCP benchmarking application, netpipe, use SDP rather than TCP.
+NodeA is the server and NodeB is the client. IPoIB is configured on both nodes, and NodeA's IPoIB address is 10.0.0.1</P>
+<P>Install netpipe on both nodes.
+<PRE>
+aptitude install netpipe-tcp
+</PRE>
+</P>
+<P>First, run the netpipe benchmark over TCP in order to obtain a baseline number.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+nodeA# NPtcp
+nodeB# NPtcp -h 10.0.0.1
+Send and receive buffers are 16384 and 87380 bytes
+(A bug in Linux doubles the requested buffer sizes)
+Now starting the main loop
+ 0: 1 bytes 2778 times --> 0.22 Mbps in 34.04 usec
+ 1: 2 bytes 2937 times --> 0.45 Mbps in 33.65 usec
+ 2: 3 bytes 2971 times --> 0.69 Mbps in 33.41 usec
+<snip>
+121: 8388605 bytes 3 times --> 2951.89 Mbps in 21680.99 usec
+122: 8388608 bytes 3 times --> 3008.08 Mbps in 21276.00 usec
+123: 8388611 bytes 3 times --> 2941.76 Mbps in 21755.66 usec
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>Now repeat the test, but force netpipe to use SDP rather than TCP.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+nodeA# LD_PRELOAD=libsdp.so NPtcp
+nodeB# LD_PRELOAD=libsdp.so NPtcp -h 10.0.0.1
+Send and receive buffers are 16384 and 87380 bytes
+(A bug in Linux doubles the requested buffer sizes)
+Now starting the main loop
+ 0: 1 bytes 9765 times --> 1.45 Mbps in 5.28 usec
+ 1: 2 bytes 18946 times --> 2.80 Mbps in 5.46 usec
+ 2: 3 bytes 18323 times --> 4.06 Mbps in 5.63 usec
+<snip>
+121: 8388605 bytes 5 times --> 7665.51 Mbps in 8349.08 usec
+122: 8388608 bytes 5 times --> 7668.62 Mbps in 8345.70 usec
+123: 8388611 bytes 5 times --> 7629.04 Mbps in 8389.00 usec
+</PRE>
+</CODE></BLOCKQUOTE>
+
+You should see a significant increase in performance when using SDP.</P>
+
+<HR>
+<A HREF="infiniband-howto-8.html">Next</A>
+<A HREF="infiniband-howto-6.html">Previous</A>
+<A HREF="infiniband-howto.html#toc7">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-8.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-8.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-8.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-8.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,101 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: SRP</TITLE>
+ <LINK HREF="infiniband-howto-9.html" REL=next>
+ <LINK HREF="infiniband-howto-7.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc8" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-9.html">Next</A>
+<A HREF="infiniband-howto-7.html">Previous</A>
+<A HREF="infiniband-howto.html#toc8">Contents</A>
+<HR>
+<H2><A NAME="s8">8.</A> <A HREF="infiniband-howto.html#toc8">SRP</A></H2>
+
+<P>SRP (SCSI Remote protocol or SCSI RDMA protocol) is a protocol that allows the use of SCSI devices across
+infiniband. If you have infiniband storage, use can use SRP to acess the devices.</P>
+<H2><A NAME="ss8.1">8.1</A> <A HREF="infiniband-howto.html#toc8.1">Configuration</A>
+</H2>
+
+<P>Ensure that your infiniband storage is presented to the host in question. Check your storage controller documentation.
+Ensure that the ib_srp kernel module is loaded and that the srptools package is installed.</P>
+<P>
+<BLOCKQUOTE><CODE>
+modprobe ib_srp
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss8.2">8.2</A> <A HREF="infiniband-howto.html#toc8.2">SRP daemon configuration</A>
+</H2>
+
+<P>srp_daemon is responsible for discovering and connecting to SRP targets. The default configuration shipped with srp_daemon is to ignore all presented
+devices; this is a failsafe to prevent devices from being mounted by accident on the wrong hosts.</P>
+<P>The srp_daemon config file /etc/srp_daemon.conf has a simply syntax, and is described in the srp_daemon(1) manpage. Each line in this file is a rule which can be either
+to allow connection or to disallow connection according to the first character in the line (a or d accordingly) and ID of the storage device.</P>
+
+<H3>Determine the IDs of presented devices</H3>
+
+<P>You can determine the IDs of SRP devices presented to your hosts by running the ibsrpdm -c command.
+<BLOCKQUOTE><CODE>
+<PRE>
+# ibsrpdm -c
+id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H3>Configure srp_deamon to connect to the devices</H3>
+
+<P>Once we have the IDs of the devices, we can add them to /etc/srp_daemon.conf. You can also specify other srp related
+options for the target, such as max_cmd_per_lun and Max_sect. These are storage specific; check your vendor documentation
+for reccomended values.
+<BLOCKQUOTE><CODE>
+<PRE>
+# This rule allows connection to our target
+a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,max_cmd_per_lun=32,max_sect=65535
+# This rule disallows everything else
+d
+</PRE>
+</CODE></BLOCKQUOTE>
+
+Restart the srp_daemon and the storage target should now become visible; check the kernel log to see if the disk has been detected.</P>
+
+<P>
+<PRE>
+/etc/init.d/srptools restart
+</PRE>
+</P>
+<P>In the example kernel log output the disk has been descovered as scsi device sdb.
+<BLOCKQUOTE><CODE>
+<PRE>
+scsi 3:0:0:1: Direct-Access IBM DCS9900 5.03 PQ: 0 ANSI: 5
+sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
+sd 3:0:0:1: [sdb] Write Protect is off
+sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
+sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
+sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
+sd 3:0:0:1: [sdb] Write Protect is off
+sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
+sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
+ sdb:<6>scsi4 : SRP.T10:50001FF10005052A
+ unknown partition table
+sd 3:0:0:1: [sdb] Attached SCSI disk
+sd 3:0:0:1: Attached scsi generic sg5 type 0
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+<H2><A NAME="ss8.3">8.3</A> <A HREF="infiniband-howto.html#toc8.3">Multipathing, LVM and formatting</A>
+</H2>
+
+<P>The newly detected SRP device can be treated as an other scsi device. If you have multiple infiniband adapters you can use multipath-tools
+on top of the SRP devices to protects against a network failure. If you are not using multipathed IO you can simply format the device as normal.</P>
+
+<HR>
+<A HREF="infiniband-howto-9.html">Next</A>
+<A HREF="infiniband-howto-7.html">Previous</A>
+<A HREF="infiniband-howto.html#toc8">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-9.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-9.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-9.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto-9.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,72 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto: Building Lustre against OFED</TITLE>
+ <LINK HREF="infiniband-howto-10.html" REL=next>
+ <LINK HREF="infiniband-howto-8.html" REL=previous>
+ <LINK HREF="infiniband-howto.html#toc9" REL=contents>
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-10.html">Next</A>
+<A HREF="infiniband-howto-8.html">Previous</A>
+<A HREF="infiniband-howto.html#toc9">Contents</A>
+<HR>
+<H2><A NAME="s9">9.</A> <A HREF="infiniband-howto.html#toc9">Building Lustre against OFED</A></H2>
+
+<P>Lustre is a scalable cluster filesystem popular on high performance compute clusters. See
+<A HREF="http://www.lustre.org">http://www.lustre.org</A>
+for more information. lustre can use infiniband as one of its network transports in order to increase performance. The section describes how to compile lustre
+against the OFED infiniband stack.</P>
+<H2><A NAME="ss9.1">9.1</A> <A HREF="infiniband-howto.html#toc9.1">Check Compatibility</A>
+</H2>
+
+<P>Not all lustre versions are compatible with all OFED or kernel versions. Read the lustre release notes for which versions are supported.</P>
+
+
+<H2><A NAME="ss9.2">9.2</A> <A HREF="infiniband-howto.html#toc9.2">Build a lustre patched kernel</A>
+</H2>
+
+<P>Build a lustre patched kernel as per the instructions on the lustre wiki. Once you have build the kernel keep the configured source tree.
+It is required for the next step.</P>
+
+
+<H2><A NAME="ss9.3">9.3</A> <A HREF="infiniband-howto.html#toc9.3">Build OFED modules for the lustre patched kernel.</A>
+</H2>
+
+<P>Build OFED modules against the newly build lustre patched kernel.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+ module-assistant prepare
+ module-assistant clean ofa-kernel
+ module-assistant -k/path/to/lustre/patched/kernel build ofa-kernel
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+<P>Do not issue a "module-assistant clean" command after the build. The ofa-kernel-module source tree is needed for the
+next step.</P>
+
+
+<H2><A NAME="ss9.4">9.4</A> <A HREF="infiniband-howto.html#toc9.4">Configure lustre</A>
+</H2>
+
+
+<P>You can now configure lustre to build against the lustre patched kernel and the ofa-kernel-module sources.</P>
+<P>
+<BLOCKQUOTE><CODE>
+<PRE>
+ cd lustre-source
+ ./configure --with-o2ib=/usr/src/modules/ofa-kernel --with-linux=/path/to/patched/linux/source \
+ --other-options
+</PRE>
+</CODE></BLOCKQUOTE>
+</P>
+
+
+<HR>
+<A HREF="infiniband-howto-10.html">Next</A>
+<A HREF="infiniband-howto-8.html">Previous</A>
+<A HREF="infiniband-howto.html#toc9">Contents</A>
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.html
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.html?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.html (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.html Fri Aug 7 17:11:42 2009
@@ -1,0 +1,111 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
+<HTML>
+<HEAD>
+ <META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.65">
+ <TITLE>Infiniband Howto</TITLE>
+ <LINK HREF="infiniband-howto-1.html" REL=next>
+
+
+</HEAD>
+<BODY>
+<A HREF="infiniband-howto-1.html">Next</A>
+Previous
+Contents
+<HR>
+<H1>Infiniband Howto</H1>
+
+<H2>Guy Coates </H2>
+<P>
+<H2><A NAME="toc1">1.</A> <A HREF="infiniband-howto-1.html">Introduction</A></H2>
+
+<UL>
+<LI><A NAME="toc1.1">1.1</A> <A HREF="infiniband-howto-1.html#ss1.1">The latest version</A>
+<LI><A NAME="toc1.2">1.2</A> <A HREF="infiniband-howto-1.html#ss1.2">What is OFED?</A>
+</UL>
+<P>
+<H2><A NAME="toc2">2.</A> <A HREF="infiniband-howto-2.html">Installing the OFED Software</A></H2>
+
+<UL>
+<LI><A NAME="toc2.1">2.1</A> <A HREF="infiniband-howto-2.html#ss2.1">Installing prebuilt packages</A>
+<LI><A NAME="toc2.2">2.2</A> <A HREF="infiniband-howto-2.html#ss2.2">Building packages from source</A>
+</UL>
+<P>
+<H2><A NAME="toc3">3.</A> <A HREF="infiniband-howto-3.html">Install the kernel modules</A></H2>
+
+<UL>
+<LI><A NAME="toc3.1">3.1</A> <A HREF="infiniband-howto-3.html#ss3.1">Building new kernel modules</A>
+</UL>
+<P>
+<H2><A NAME="toc4">4.</A> <A HREF="infiniband-howto-4.html">Setting up a basic infiniband network </A></H2>
+
+<UL>
+<LI><A NAME="toc4.1">4.1</A> <A HREF="infiniband-howto-4.html#ss4.1">Upgrade your Infiniband card and switch firmware</A>
+<LI><A NAME="toc4.2">4.2</A> <A HREF="infiniband-howto-4.html#ss4.2">Physically Connect the network</A>
+<LI><A NAME="toc4.3">4.3</A> <A HREF="infiniband-howto-4.html#ss4.3">Choose a Subnet Manager</A>
+<LI><A NAME="toc4.4">4.4</A> <A HREF="infiniband-howto-4.html#ss4.4">Load the kernel modules</A>
+<LI><A NAME="toc4.5">4.5</A> <A HREF="infiniband-howto-4.html#ss4.5">(optional) Start opensm</A>
+<LI><A NAME="toc4.6">4.6</A> <A HREF="infiniband-howto-4.html#ss4.6">Check network health</A>
+<LI><A NAME="toc4.7">4.7</A> <A HREF="infiniband-howto-4.html#ss4.7">Check the extended network connectivity</A>
+<LI><A NAME="toc4.8">4.8</A> <A HREF="infiniband-howto-4.html#ss4.8">testing connectivity with ibping</A>
+<LI><A NAME="toc4.9">4.9</A> <A HREF="infiniband-howto-4.html#ss4.9">Testing RDMA performance</A>
+</UL>
+<P>
+<H2><A NAME="toc5">5.</A> <A HREF="infiniband-howto-5.html">IP over Infiniband (IPoIB)</A></H2>
+
+<UL>
+<LI><A NAME="toc5.1">5.1</A> <A HREF="infiniband-howto-5.html#ss5.1">List the network devices</A>
+<LI><A NAME="toc5.2">5.2</A> <A HREF="infiniband-howto-5.html#ss5.2">IP Configuration</A>
+<LI><A NAME="toc5.3">5.3</A> <A HREF="infiniband-howto-5.html#ss5.3">Connected vs Unconnected Mode</A>
+<LI><A NAME="toc5.4">5.4</A> <A HREF="infiniband-howto-5.html#ss5.4">TCP tuning</A>
+<LI><A NAME="toc5.5">5.5</A> <A HREF="infiniband-howto-5.html#ss5.5">ARP and dual ported cards.</A>
+</UL>
+<P>
+<H2><A NAME="toc6">6.</A> <A HREF="infiniband-howto-6.html">OpenMPI</A></H2>
+
+<UL>
+<LI><A NAME="toc6.1">6.1</A> <A HREF="infiniband-howto-6.html#ss6.1">Configure IPoIB</A>
+<LI><A NAME="toc6.2">6.2</A> <A HREF="infiniband-howto-6.html#ss6.2">Load the modules</A>
+<LI><A NAME="toc6.3">6.3</A> <A HREF="infiniband-howto-6.html#ss6.3">Check permissions and limits</A>
+<LI><A NAME="toc6.4">6.4</A> <A HREF="infiniband-howto-6.html#ss6.4">Install the mpi test programs</A>
+<LI><A NAME="toc6.5">6.5</A> <A HREF="infiniband-howto-6.html#ss6.5">Configure Hosts</A>
+<LI><A NAME="toc6.6">6.6</A> <A HREF="infiniband-howto-6.html#ss6.6">Run the MPI PingPong benchmark.</A>
+</UL>
+<P>
+<H2><A NAME="toc7">7.</A> <A HREF="infiniband-howto-7.html">SDP</A></H2>
+
+<UL>
+<LI><A NAME="toc7.1">7.1</A> <A HREF="infiniband-howto-7.html#ss7.1">Configuration</A>
+<LI><A NAME="toc7.2">7.2</A> <A HREF="infiniband-howto-7.html#ss7.2">Example Using SDP with Netpipe</A>
+</UL>
+<P>
+<H2><A NAME="toc8">8.</A> <A HREF="infiniband-howto-8.html">SRP</A></H2>
+
+<UL>
+<LI><A NAME="toc8.1">8.1</A> <A HREF="infiniband-howto-8.html#ss8.1">Configuration</A>
+<LI><A NAME="toc8.2">8.2</A> <A HREF="infiniband-howto-8.html#ss8.2">SRP daemon configuration</A>
+<LI><A NAME="toc8.3">8.3</A> <A HREF="infiniband-howto-8.html#ss8.3">Multipathing, LVM and formatting</A>
+</UL>
+<P>
+<H2><A NAME="toc9">9.</A> <A HREF="infiniband-howto-9.html">Building Lustre against OFED</A></H2>
+
+<UL>
+<LI><A NAME="toc9.1">9.1</A> <A HREF="infiniband-howto-9.html#ss9.1">Check Compatibility</A>
+<LI><A NAME="toc9.2">9.2</A> <A HREF="infiniband-howto-9.html#ss9.2">Build a lustre patched kernel</A>
+<LI><A NAME="toc9.3">9.3</A> <A HREF="infiniband-howto-9.html#ss9.3">Build OFED modules for the lustre patched kernel.</A>
+<LI><A NAME="toc9.4">9.4</A> <A HREF="infiniband-howto-9.html#ss9.4">Configure lustre</A>
+</UL>
+<P>
+<H2><A NAME="toc10">10.</A> <A HREF="infiniband-howto-10.html">Network Troubleshooting</A></H2>
+
+<UL>
+<LI><A NAME="toc10.1">10.1</A> <A HREF="infiniband-howto-10.html#ss10.1">ibdiagnet</A>
+</UL>
+<P>
+<H2><A NAME="toc11">11.</A> <A HREF="infiniband-howto-11.html">Further Information</A></H2>
+
+<HR>
+<A HREF="infiniband-howto-1.html">Next</A>
+Previous
+Contents
+</BODY>
+</HTML>
Added: branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.txt
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.txt?rev=437&op=file
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.txt (added)
+++ branches/ofed-1.4.2/ofed-docs/trunk/DEBIAN-HOWTO/infiniband-howto.txt Fri Aug 7 17:11:42 2009
@@ -1,0 +1,1074 @@
+ Infiniband Howto
+ Guy Coates
+
+ ____________________________________________________________
+
+ Table of Contents
+
+
+
+ 1. Introduction
+ 1.1 The latest version
+ 1.2 What is OFED?
+
+ 2. Installing the OFED Software
+ 2.1 Installing prebuilt packages
+ 2.2 Building packages from source
+ 2.2.1 Install the prerequisites development packages
+ 2.2.2 Checkout the svn tree
+ 2.2.3 Install the upstream source (optional)
+ 2.2.4 Build the packages.
+
+ 3. Install the kernel modules
+ 3.1 Building new kernel modules
+
+ 4. Setting up a basic infiniband network
+ 4.1 Upgrade your Infiniband card and switch firmware
+ 4.2 Physically Connect the network
+ 4.3 Choose a Subnet Manager
+ 4.4 Load the kernel modules
+ 4.5 (optional) Start opensm
+ 4.6 Check network health
+ 4.7 Check the extended network connectivity
+ 4.8 testing connectivity with ibping
+ 4.9 Testing RDMA performance
+
+ 5. IP over Infiniband (IPoIB)
+ 5.1 List the network devices
+ 5.2 IP Configuration
+ 5.3 Connected vs Unconnected Mode
+ 5.4 TCP tuning
+ 5.5 ARP and dual ported cards.
+
+ 6. OpenMPI
+ 6.1 Configure IPoIB
+ 6.2 Load the modules
+ 6.3 Check permissions and limits
+ 6.4 Install the mpi test programs
+ 6.5 Configure Hosts
+ 6.6 Run the MPI PingPong benchmark.
+
+ 7. SDP
+ 7.1 Configuration
+ 7.2 Example Using SDP with Netpipe
+
+ 8. SRP
+ 8.1 Configuration
+ 8.2 SRP daemon configuration
+ 8.2.1 Determine the IDs of presented devices
+ 8.2.2 Configure srp_deamon to connect to the devices
+ 8.3 Multipathing, LVM and formatting
+
+ 9. Building Lustre against OFED
+ 9.1 Check Compatibility
+ 9.2 Build a lustre patched kernel
+ 9.3 Build OFED modules for the lustre patched kernel.
+ 9.4 Configure lustre
+
+ 10. Network Troubleshooting
+ 10.1 ibdiagnet
+
+ 11. Further Information
+
+
+ ______________________________________________________________________
+
+ 11.. IInnttrroodduuccttiioonn
+
+ This document describes how to install and configure the OFED
+ infiniband software on Debian. This document is intended to show you
+ how to configure a simple Infiniband network as quickly as possible.
+ It is not a replacement for the details documentation provided in the
+ ofed-docs package!
+
+ 11..11.. TThhee llaatteesstt vveerrssiioonn
+
+ The latest version of the howto can be found on the pkg-ofed alioth
+ webite:
+
+ http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html
+ <http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html>
+
+ Source is kept in the SVN repository:
+
+ http://svn.debian.org/wsvn/pkg-ofed/ <http://svn.debian.org/wsvn/pkg-
+ ofed/>
+
+ 11..22.. WWhhaatt iiss OOFFEEDD??
+
+ OFED (OpenFabric's Enterprise Distribution) is the defacto Infiniband
+ software stack on Linux. OFED provides a consistent set of kernel
+ modules and userspace libraries which have been tested together.
+
+ Further details of the Openfabrics Alliance and OFED can be found here
+ http://www.openfabrics.org <http://www.openfabrics.org/>
+
+ 22.. IInnssttaalllliinngg tthhee OOFFEEDD SSooffttwwaarree
+
+ Before you can use your infiniband network you will need to install
+ the OFED software on your infiniband client machines. You can choose
+ to use the pre-build packages on alioth, or build your own packages
+ straight from the alioth SVN repository.
+
+ 22..11.. IInnssttaalllliinngg pprreebbuuiilltt ppaacckkaaggeess
+
+ Add the following lines to your sources.list file:
+
+
+ deb http://pkg-ofed.alioth.debian.org/apt/ofed ./
+ deb-src http://pkg-ofed.alioth.debian.org/apt/ofed ./
+
+
+
+ and run:
+
+
+ aptitude update
+ aptitude install ofed
+
+
+
+ 22..22.. BBuuiillddiinngg ppaacckkaaggeess ffrroomm ssoouurrccee
+
+ If you wish to build the OFED packages from the alioth svn repository,
+ use the following procedure.
+
+ 22..22..11.. IInnssttaallll tthhee pprreerreeqquuiissiitteess ddeevveellooppmmeenntt ppaacckkaaggeess
+
+
+
+ aptitude install svn-buildpackage build-essential devscripts
+
+
+
+ 22..22..22.. CChheecckkoouutt tthhee ssvvnn ttrreeee
+
+
+ svn co svn://svn.debian.org/pkg-ofed/
+
+
+ 22..22..33.. IInnssttaallll tthhee uuppssttrreeaamm ssoouurrccee ((ooppttiioonnaall))
+
+ The upstream source tarballs need to be available if you want to build
+ pukka debian packages suitable for inclusion upstream. If you are
+ simply building packages for your own use, you can ignore this step.
+
+
+ cd pkg-ofed
+ mkdir tarballs
+
+
+
+ Populate the tarballs with the *.orig.tar.gz files available form the
+ "upstream source" release on
+ https://alioth.debian.org/frs/?group_id=100311
+ <https://alioth.debian.org/frs/?group_id=100311>
+
+ 22..22..44.. BBuuiilldd tthhee ppaacckkaaggeess..
+
+ cd into the package you wish to build. eg for libibcommon,
+
+ cd pkg-ofed/libibcommon
+
+
+ Link in the upstream tarballs directory (optional)
+
+ ln -s -f ../tarballs .
+
+
+ Run svn-buildpackage from within the trunk directory.
+
+
+ cd pkg-ofed/libibcommon/trunk
+ svn-buildpackage -uc -us -rfakeroot
+
+
+
+ The build process will generate a deb in the build-area directory.
+
+ Repeat the process for the rest of the packages. Note that some
+ packages have build dependancies on other OFED packages. The suggested
+ build order is:
+
+
+
+ libibcm
+ libibcommon
+ libibumad
+ libibmad
+ libnes
+ libsdp
+ dapl
+ opensm
+ infiniband-diags
+ ibutils
+ mstflint
+ perftest
+ qlvnictools
+ qpert
+ rds-tools
+ sdpnetstat
+ srptools
+ tvflash
+ ibsim
+ ofed-docs
+ ofa_kernel
+ ofed
+
+
+
+ 33.. IInnssttaallll tthhee kkeerrnneell mmoodduulleess
+
+ You now need to build a set of OFED kernel modules which match the
+ version of the OFED software you have installed.
+
+ The Debian kernel contains a set of OFED infiniband drivers, but they
+ may not match the OFED userspace version have installed. Consult the
+ table below to determine what OFED version the Debian kernel contains.
+
+
+
+ Debian Kernel Version OFED Version
+ <=2.6.26 1.3
+ >=2.6.27 1.4
+
+
+
+ If the debian kernel modules are the incorrect version, you can build
+ a new set of modules using the ofa-kernel-source package. If your
+ kernel already includes the correct OFED kernel modules you can skip
+ the rest of this section. If you are in doubt, you should build a new
+ set of modules rather than relying on the modules shipped with the
+ kernel.
+
+ 33..11.. BBuuiillddiinngg nneeww kkeerrnneell mmoodduulleess
+
+ You can build new kernel modules using module-assistant.
+
+
+ aptitude install module-assistant
+
+
+
+ Ensure you have the ofa-kernel-source package installed, and then run:
+
+
+
+ module-assistant prepare
+ module-assistant clean ofa-kernel
+ module-assistant build ofa-kernel
+
+
+
+ This will create a deb which you can then install. As the deb contains
+ replacements for existing kernel modules you will need to either manu-
+ ally remove any infiniband modules which have already been loaded, or
+ reboot the machine, before you can use the new modules.
+
+ The new kernel modules will be installed into /usr/lib/<kernel-
+ version>/updates. They will not overwrite the original kernel modules,
+ but the module loader will pick up the modules from the updates
+ directory in preference. You can verify that the system is using the
+ new kernel modules by running the modinfo command.
+
+
+
+ # modinfo ib_core
+ filename: /lib/modules/2.6.22.19/updates/kernel/drivers/infiniband/core/ib_core.ko
+ author: Roland Dreier
+ description: core kernel InfiniBand API
+ license: Dual BSD/GPL
+ vermagic: 2.6.22.19 SMP mod_unload
+
+
+
+ Note that if you wish to rebuild the kernel modules (eg for a new
+ kernel version) then you must issue the module-assistant clean command
+ before trying a new build.
+
+ 44.. SSeettttiinngg uupp aa bbaassiicc iinnffiinniibbaanndd nneettwwoorrkk
+
+ This sections describes how to set up a basic infiniband network and
+ test its functionality.
+
+ 44..11.. UUppggrraaddee yyoouurr IInnffiinniibbaanndd ccaarrdd aanndd sswwiittcchh ffiirrmmwwaarree
+
+ Before proceeding you should ensure that the firmware in your switches
+ and infiniband cards is at the latest release. Older firmware
+ versions may cause interoperability and fabric stability issues. Do
+ not assume that just because your hardware has come fresh from the
+ factory that it has the latest firmware on it.
+
+ You should follow the documentation from your vendor as to how the
+ firmware should be updated.
+
+ 44..22.. PPhhyyssiiccaallllyy CCoonnnneecctt tthhee nneettwwoorrkk
+
+ Connect up to your hosts and switches.
+
+ 44..33.. CChhoooossee aa SSuubbnneett MMaannaaggeerr
+
+ Each infiniband network requires a subnet manager. You can choose to
+ run the OFED opensm subnet manager on one of the Linux clients, or you
+ may choose to use an embedded subnet manager running on one of the
+ switches in your fabric. Note that not all switches come with a subnet
+ manager; check your switch documentation.
+
+ 44..44.. LLooaadd tthhee kkeerrnneell mmoodduulleess
+
+ Infiniband kernel modules are not loaded automatically. You should
+ adding them to /etc/modules so that they are automatically loaded on
+ machine bootup. You will need to include the hardware specific modules
+ and the protocol modules.
+
+ /etc/modules:
+
+ # Hardware drivers
+ # Choose the apropriate modules from
+ # /lib/modules/<kernel-version>/updates/kernel/drivers/infiniband/hw
+ #
+ #mlx4_ib # Mellanox ConnectX cards
+ #ib_mthca # some mellanox cards
+ #iw_cxgb3 # Chelsio T3 cards
+ #iw_nes # NetEffect cards
+ #
+ # Protocol modules
+ # Common modules
+ rdma_ucm
+ ib_umad
+ ib_uverbs
+ # IP over IB
+ ib_ipoib
+ # scsi over IB
+ ib_srp
+ # IB SDP protocol
+ ib_sdp
+
+
+
+ 44..55.. ((ooppttiioonnaall)) SSttaarrtt ooppeennssmm
+
+ If you are going to use the opensm suetnet manager, edit
+ /etc/default/opensm and add the port GUIDs of the interfaces on which
+ you wish to start opensm.
+
+ You can find the port GUIDs of your cards with the ibstat -p command:
+
+
+ # ibstat -p
+ 0x0002c9030002fb05
+ 0x0002c9030002fb06
+
+
+
+ /etc/default/opensm:
+
+
+ PORTS="0x0002c9030002fb05 0x0002c9030002fb06"
+
+
+
+ Note if you want to start opensm on all ports you can use the
+ PORTS="ALL" keyword.
+
+ Start opensm:
+
+
+ #/etc/init.d/opensm start
+
+
+
+ If opensm has started correctly you should see SUBNET UP messages in
+ the opensm logfile (/var/log/opensm.<PORTID>.log).
+
+
+ Mar 04 14:56:06 600685 [4580A960] 0x02 -> SUBNET UP
+
+
+
+ Note that you can start opensm on multiple nodes; one node will be the
+ active subnet manager and the others will put themselves into standby.
+
+ 44..66.. CChheecckk nneettwwoorrkk hheeaalltthh
+
+ You can now check the status of the local IB link with the ibstat
+ command. Connected links should be in the "LinkUp" state. The
+ following output is from a dual ported card, only one of which (port1)
+ is connected.
+
+
+
+ # ibstat
+ CA 'mlx4_0'
+ CA type: MT25418
+ Number of ports: 2
+ Firmware version: 2.3.0
+ Hardware version: a0
+ Node GUID: 0x0002c9030002fb04
+ System image GUID: 0x0002c9030002fb07
+ Port 1:
+ State: Active
+ Physical state: LinkUp
+ Rate: 20
+ Base lid: 2
+ LMC: 0
+ SM lid: 1
+ Capability mask: 0x02510868
+ Port GUID: 0x0002c9030002fb05
+ Port 2:
+ State: Down
+ Physical state: Polling
+ Rate: 10
+ Base lid: 0
+ LMC: 0
+ SM lid: 0
+ Capability mask: 0x02510868
+ Port GUID: 0x0002c9030002fb06
+
+
+
+ 44..77.. CChheecckk tthhee eexxtteennddeedd nneettwwoorrkk ccoonnnneeccttiivviittyy
+
+ Once the host is connected to the infiniband network you can check the
+ health of all of the other network components with the ibhosts,
+ ibswitches and iblinkinfo commands.
+
+ ibhosts displays all of the hosts visible on the network.
+
+
+
+ # ibhosts
+ Ca : 0x0008f1040399d3d0 ports 2 "Voltaire HCA400Ex-D"
+ Ca : 0x0008f1040399d370 ports 2 "Voltaire HCA400Ex-D"
+ Ca : 0x0008f1040399d3fc ports 2 "Voltaire HCA400Ex-D"
+ Ca : 0x0008f1040399d3f4 ports 2 "Voltaire HCA400Ex-D"
+ Ca : 0x0002c9030002faf4 ports 2 "MT25408 ConnectX Mellanox Technologies"
+ Ca : 0x0002c9030002fc0c ports 2 "MT25408 ConnectX Mellanox Technologies"
+ Ca : 0x0002c9030002fc10 ports 2 "MT25408 ConnectX Mellanox Technologies"
+
+
+ ibswitches will display all of the switches in the network.
+
+
+ # ibswitches
+ Switch : 0x0008f104004121fa ports 24 "ISR9024D-M Voltaire" enhanced port 0 lid 1 lmc 0
+
+
+
+ iblinkinfo will show the status and speed of all of the links in the
+ network.
+
+
+ #iblinkinfo.pl
+ Switch 0x0008f104004121fa ISR9024D-M Voltaire:
+ 1 1[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 2 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 2[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 13 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 3[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 4 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 4[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 26 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 5[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 27 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 6[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 24 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 7[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 28 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 8[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 25 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 9[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 31 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 10[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 32 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 11[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 33 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 12[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 29 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 1 13[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 30 1[ ] "MT25408 ConnectX Mellanox Technologies" ( )
+ 14[ ] ==( 4X 2.5 Gbps Down / Polling)==> [ ] "" ( )
+ 1 15[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 3 1[ ] "Voltaire HCA400Ex-D" ( )
+ 1 16[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 10 1[ ] "Voltaire HCA400Ex-D" ( )
+ 17[ ] ==( 4X 2.5 Gbps Down / Polling)==> [ ] "" ( )
+ 18[ ] ==( 4X 2.5 Gbps Down / Polling)==> [ ] "" ( )
+ 1 19[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 7 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 20[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 6 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 21[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 5 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 22[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 21 1[ ] "Voltaire HCA400Ex-D" ( )
+ 1 23[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 9 2[ ] "Voltaire HCA400Ex-D" ( )
+ 1 24[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 8 1[ ] "Voltaire HCA400Ex-D" ( )
+
+
+
+ 44..88.. tteessttiinngg ccoonnnneeccttiivviittyy wwiitthh iibbppiinngg
+
+ ibping is an infiniband equivalent to the icmp ping command. Choose a
+ node on the fabric and run a ibping server:
+
+ #ibping -S
+
+
+ Choose another node on your network, and then ping the port GUID of
+ the server. (ibstat on the server will list the port GUID).
+
+
+
+ #ibping -G 0x0002c9030002fc1d
+ Pong from test.example.com (Lid 13): time 0.072 ms
+ Pong from test.example.com (Lid 13): time 0.043 ms
+ Pong from test.example.com (Lid 13): time 0.045 ms
+ Pong from test.example.com (Lid 13): time 0.045 ms
+
+
+
+ 44..99.. TTeessttiinngg RRDDMMAA ppeerrffoorrmmaannccee
+
+ You can test the latency and bandwidth of a link with the ib_rdma_lat
+ commands.
+
+ To test the latency, start the server on a node:
+
+ #ib_rdma_lat
+
+
+ and then start a client on another node, giving it the hostname of the
+ server.
+
+
+ #ib_rdma_lat hostname-of-server
+ local address: LID 0x0d QPN 0x18004a PSN 0xca58c4 RKey 0xda002824 VAddr 0x00000000509001
+ remote address: LID 0x02 QPN 0x7c004a PSN 0x4b4eba RKey 0x82002466 VAddr 0x00000000509001
+ Latency typical: 1.15193 usec
+ Latency best : 1.13094 usec
+ Latency worst : 5.48519 usec
+
+
+
+ You can test the bandwith of the link using the ib_rdma_bw command.
+
+ #ib_rdma_bw
+
+
+ and then start a client on another node, giving it the hostname of the
+ server.
+
+
+ #ib_rdma_bw hostname-of-server
+ 855: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
+ 855: Local address: LID 0x0d, QPN 0x1c004a, PSN 0xbf60dd RKey 0xde002824 VAddr 0x002aea4092b000
+ 855: Remote address: LID 0x02, QPN 0x004a, PSN 0xaad03c, RKey 0x86002466 VAddr 0x002b8a4e191000
+
+
+ 855: Bandwidth peak (#0 to #955): 1486.85 MB/sec
+ 855: Bandwidth average: 1486.47 MB/sec
+ 855: Service Demand peak (#0 to #955): 1970 cycles/KB
+ 855: Service Demand Avg : 1971 cycles/KB
+
+
+
+ The perftest package contains a number of other similar benchmarking
+ programs to test various aspects of your network.
+
+ 55.. IIPP oovveerr IInnffiinniibbaanndd ((IIPPooIIBB))
+
+ The OFED stack allows you to run TCP/IP over your infiniband network,
+ allowing you to run non-infiniband aware applications across your
+ network. Several native infiniband applications also use IPoIB for
+ host resolution (eg Lustre and SDP).
+
+ 55..11.. LLiisstt tthhee nneettwwoorrkk ddeevviicceess
+
+ Check that the IBoIP modules is loaded.
+
+
+ #modprobe ib_ipoib
+
+
+ You will now have an "ib" network interface for each of your
+ infiniband cards.
+
+
+ #ifconfig -a
+
+ <snip>
+ ib0 Link encap:UNSPEC HWaddr 80-06-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
+ BROADCAST MULTICAST MTU:2044 Metric:1
+ RX packets:0 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:256
+ RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
+
+ ib1 Link encap:UNSPEC HWaddr 80-06-00-49-FE-80-00-00-00-00-00-00-00-00-00-00
+ BROADCAST MULTICAST MTU:2044 Metric:1
+ RX packets:0 errors:0 dropped:0 overruns:0 frame:0
+ TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
+ collisions:0 txqueuelen:256
+ RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
+ <snip>
+
+
+
+ 55..22.. IIPP CCoonnffiigguurraattiioonn
+
+ You can now configure the ib network devices using
+ /etc/network/interfaces.
+
+
+ auto ib0
+ iface ib0 inet static
+ address 172.31.128.50
+ netmask 255.255.240.0
+ broadcast 172.31.143.255
+
+
+
+ Bring the network device up, as normal.
+
+ ifup ib0
+
+
+ 55..33.. CCoonnnneecctteedd vvss UUnnccoonnnneecctteedd MMooddee
+
+ IPoIB can run over two infiniband transports, Unreliable Datagram (UD)
+ mode or Connected mode (CM). The difference between these two modes
+ are described in:
+
+ RFC4392 - IP over InfiniBand (IPoIB) Architecture
+ RFC4391 - Transmission of IP over InfiniBand (IPoIB) (UD mode)
+ RFC4755 - IP over InfiniBand: Connected Mode
+
+
+ ADDME: Pro/cons of these two methods?
+
+ You can switch between these two mode at runtime with:
+
+
+
+ echo datagram > /sys/class/net/ibX/mode
+ echo connected > /sys/class/net/ibX/mode
+
+
+
+ The default is datagram (UD) mode. If you with to use CM then you can
+ add a script to /etc/network/interfaces/if-up.d to automatically set
+ CM mode on your interfaces when they are configured.
+
+ 55..44.. TTCCPP ttuunniinngg
+
+ In order to obtain maximum IPoIB throughput you may need to tweak the
+ MTU and various kernel TCP buffer and window settings. See the
+ details in the ipoib_release_notes.txt document in the ofed-docs
+ package.
+
+ 55..55.. AARRPP aanndd dduuaall ppoorrtteedd ccaarrddss..
+
+ If you have a dual ported card with both ports on the same IB subnet
+ but a different IP subnet, you will need to tweak the ARP settings for
+ the IPoIB interfaces. See ipoib_release_notes.txt in the ofed-docs
+ package for a full discussion of this issue.
+
+
+
+ sysctl -w net.ipv4.conf.ib0.arp_ignore=1
+ sysctl -w net.ipv4.conf.ib1.arp_ignore=1
+
+
+
+ 66.. OOppeennMMPPII
+
+ The section describes how to configure OpenMPI to use Infiniband.
+
+ 66..11.. CCoonnffiigguurree IIPPooIIBB
+
+ OpenMPI uses IPoIB for job startup and tear-down. You should configure
+ IPoIB on all of your hosts.
+
+ 66..22.. LLooaadd tthhee mmoodduulleess
+
+ Ensure the rdma_ucm module is loaded.
+
+ modprobe rdma_ucm
+
+
+ 66..33.. CChheecckk ppeerrmmiissssiioonnss aanndd lliimmiittss
+
+ Uses who want to run MPI jobs will need to have write permissions for
+ the following devices:
+
+
+ /dev/infiniband/uverbs*
+ /dev/infiniband/rdma_cm*
+
+
+
+ The simplest way to do this is to add the users to the rdma group. If
+ that is not suitiable for your site, you can change the permissions
+ and ownership of these devices by editing the following udev rules:
+
+
+ /etc/udev/rules.d/50-udev.rules
+ /etc/udev/rules.d/91-permissions.rules
+
+
+
+ OpenMPI will need to pin memory. Edit /etc/security/limits.conf and
+ add the line:
+
+ * hard memlock unlimited
+
+
+ 66..44.. IInnssttaallll tthhee mmppii tteesstt pprrooggrraammss
+
+ Check the mpitests package is installed.
+
+ aptitude install mpitests
+
+
+ 66..55.. CCoonnffiigguurree HHoossttss
+
+ OpenMPI uses ssh to spawn jobs on remote hosts. You should configure a
+ public/private keypair to ensure that you can ssh between hosts
+ without entering a password. You should also ensure that your login
+ process is silent.
+
+ Choose two hosts on which to test the program and put their hostnames
+ into a file called hostfile:
+
+
+
+ hostA slots=1
+ hostB slots=1
+
+
+
+ 66..66.. RRuunn tthhee MMPPII PPiinnggPPoonngg bbeenncchhmmaarrkk..
+
+ We will use the MPI PingPong benchmark for our testing. By default,
+ openmpi should use inifiniband networks in preference to any tcp
+ networks it finds. However, we will force mpi to be extra-chatty
+ during the test to ensure that we are really using the infiniband
+ interfaces.
+
+ (ADDME: Is there a better way to confirm which networks openmpi is
+ using?)
+
+
+ mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile /path/to/hostfile IMB-MPI1 PingPong
+
+
+
+ If all goes well you should see openib debugging messages from both
+ hosts, together with the job output.
+
+
+
+ <snip>
+ # PingPong
+ [HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+ [HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+ [HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+ [HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+
+ #---------------------------------------------------
+ # Benchmarking PingPong
+ # #processes = 2
+ #---------------------------------------------------
+ #bytes #repetitions t[usec] Mbytes/sec
+ 0 1000 1.53 0.00
+ 1 1000 1.44 0.66
+ 2 1000 1.42 1.34
+ 4 1000 1.41 2.70
+ 8 1000 1.48 5.15
+ 16 1000 1.50 10.15
+ 32 1000 1.54 19.85
+ 64 1000 1.79 34.05
+ 128 1000 3.01 40.56
+ 256 1000 3.56 68.66
+ 512 1000 4.46 109.41
+ 1024 1000 5.37 181.92
+ 2048 1000 8.13 240.25
+ 4096 1000 10.87 359.48
+ 8192 1000 15.97 489.17
+ 16384 1000 30.54 511.68
+ 32768 1000 55.01 568.12
+ 65536 640 122.20 511.46
+ 131072 320 207.20 603.27
+ 262144 160 377.10 662.96
+ 524288 80 706.21 708.00
+ 1048576 40 1376.93 726.25
+ 2097152 20 1946.00 1027.75
+ 4194304 10 3119.29 1282.34
+
+
+
+ If you encounter any errors read the excellent OpenMPI troubleshooting
+ guide. http://www.openmpi.org <http://www.openmpi.org>
+
+ If you want to compare infiniband performance with your ethernet/TCP
+ networks, you can re-run the tests using flags to tell openmpi to use
+ your ethernet network. (The example below assumes that your test nodes
+ are connected via eth0).
+
+
+ mpirun --mca btl ^openib --mca btl_tcp_if_include eth0 --hostfile hostfile -n 2 IMB-MPI1 -benchmark PingPong
+
+
+
+ 77.. SSDDPP
+
+ Sockets Direct Protocol (SDP) is a network protocol which provides an
+ RDMA accelerated alternative to TCP over infiniband networks. OFED
+ provides an LD_PRELOADable library (libsdp.so) which allows programs
+ which use TCP to use the more efficient SDP protocol instead. The use
+ of an LD_PRELOADable libary means that the switch in protocol is
+ transparent, and does not require the application to be recompiled.
+
+ 77..11.. CCoonnffiigguurraattiioonn
+
+ SDP used IPoIB for address resolution, so you must configure IPoIB
+ before using SDP.
+ You should also ensure the ib_sdp kernel module is installed.
+
+ modprobe ib_sdp
+
+
+
+ You can use libsdp in two ways; you can either manually LD_PRELOAD the
+ library whilst invoking your application, or create a config file
+ which specifies which applications will use SDP.
+
+ To manually LD_PRELOAD a library, simply set the LD_PRELOAD variable
+ before invoking your application.
+
+ LD_PRELOAD=libsdp.so ./path/to/your/application ...
+
+
+ If you which to choose which programs will use SDP you can edit
+ /etc/sdp.conf and specify which programs, ports and addresses are eli-
+ gible for use.
+
+ 77..22.. EExxaammppllee UUssiinngg SSDDPP wwiitthh NNeettppiippee
+
+ The following example shows how to use libsdp to make the TCP
+ benchmarking application, netpipe, use SDP rather than TCP. NodeA is
+ the server and NodeB is the client. IPoIB is configured on both nodes,
+ and NodeA's IPoIB address is 10.0.0.1
+
+ Install netpipe on both nodes.
+
+ aptitude install netpipe-tcp
+
+
+
+ First, run the netpipe benchmark over TCP in order to obtain a
+ baseline number.
+
+
+
+ nodeA# NPtcp
+ nodeB# NPtcp -h 10.0.0.1
+ Send and receive buffers are 16384 and 87380 bytes
+ (A bug in Linux doubles the requested buffer sizes)
+ Now starting the main loop
+ 0: 1 bytes 2778 times --> 0.22 Mbps in 34.04 usec
+ 1: 2 bytes 2937 times --> 0.45 Mbps in 33.65 usec
+ 2: 3 bytes 2971 times --> 0.69 Mbps in 33.41 usec
+ <snip>
+ 121: 8388605 bytes 3 times --> 2951.89 Mbps in 21680.99 usec
+ 122: 8388608 bytes 3 times --> 3008.08 Mbps in 21276.00 usec
+ 123: 8388611 bytes 3 times --> 2941.76 Mbps in 21755.66 usec
+
+
+
+ Now repeat the test, but force netpipe to use SDP rather than TCP.
+
+
+
+ nodeA# LD_PRELOAD=libsdp.so NPtcp
+ nodeB# LD_PRELOAD=libsdp.so NPtcp -h 10.0.0.1
+ Send and receive buffers are 16384 and 87380 bytes
+ (A bug in Linux doubles the requested buffer sizes)
+ Now starting the main loop
+ 0: 1 bytes 9765 times --> 1.45 Mbps in 5.28 usec
+ 1: 2 bytes 18946 times --> 2.80 Mbps in 5.46 usec
+ 2: 3 bytes 18323 times --> 4.06 Mbps in 5.63 usec
+ <snip>
+ 121: 8388605 bytes 5 times --> 7665.51 Mbps in 8349.08 usec
+ 122: 8388608 bytes 5 times --> 7668.62 Mbps in 8345.70 usec
+ 123: 8388611 bytes 5 times --> 7629.04 Mbps in 8389.00 usec
+
+
+
+ You should see a significant increase in performance when using SDP.
+
+ 88.. SSRRPP
+
+ SRP (SCSI Remote protocol or SCSI RDMA protocol) is a protocol that
+ allows the use of SCSI devices across infiniband. If you have
+ infiniband storage, use can use SRP to acess the devices.
+
+ 88..11.. CCoonnffiigguurraattiioonn
+
+ Ensure that your infiniband storage is presented to the host in
+ question. Check your storage controller documentation. Ensure that
+ the ib_srp kernel module is loaded and that the srptools package is
+ installed.
+
+
+ modprobe ib_srp
+
+
+ 88..22.. SSRRPP ddaaeemmoonn ccoonnffiigguurraattiioonn
+
+ srp_daemon is responsible for discovering and connecting to SRP
+ targets. The default configuration shipped with srp_daemon is to
+ ignore all presented devices; this is a failsafe to prevent devices
+ from being mounted by accident on the wrong hosts.
+
+ The srp_daemon config file /etc/srp_daemon.conf has a simply syntax,
+ and is described in the srp_daemon(1) manpage. Each line in this file
+ is a rule which can be either to allow connection or to disallow
+ connection according to the first character in the line (a or d
+ accordingly) and ID of the storage device.
+
+ 88..22..11.. DDeetteerrmmiinnee tthhee IIDDss ooff pprreesseenntteedd ddeevviicceess
+
+ You can determine the IDs of SRP devices presented to your hosts by
+ running the ibsrpdm -c command.
+
+
+ # ibsrpdm -c
+ id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050
+
+
+
+ 88..22..22.. CCoonnffiigguurree ssrrpp__ddeeaammoonn ttoo ccoonnnneecctt ttoo tthhee ddeevviicceess
+
+ Once we have the IDs of the devices, we can add them to
+ /etc/srp_daemon.conf. You can also specify other srp related options
+ for the target, such as max_cmd_per_lun and Max_sect. These are
+ storage specific; check your vendor documentation for reccomended
+ values.
+
+
+ # This rule allows connection to our target
+ a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,max_cmd_per_lun=32,max_sect=65535
+ # This rule disallows everything else
+ d
+
+
+
+ Restart the srp_daemon and the storage target should now become visi-
+ ble; check the kernel log to see if the disk has been detected.
+
+
+ /etc/init.d/srptools restart
+
+
+
+ In the example kernel log output the disk has been descovered as scsi
+ device sdb.
+
+
+ scsi 3:0:0:1: Direct-Access IBM DCS9900 5.03 PQ: 0 ANSI: 5
+ sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
+ sd 3:0:0:1: [sdb] Write Protect is off
+ sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
+ sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
+ sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
+ sd 3:0:0:1: [sdb] Write Protect is off
+ sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
+ sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
+ sdb:<6>scsi4 : SRP.T10:50001FF10005052A
+ unknown partition table
+ sd 3:0:0:1: [sdb] Attached SCSI disk
+ sd 3:0:0:1: Attached scsi generic sg5 type 0
+
+
+
+ 88..33.. MMuullttiippaatthhiinngg,, LLVVMM aanndd ffoorrmmaattttiinngg
+
+ The newly detected SRP device can be treated as an other scsi device.
+ If you have multiple infiniband adapters you can use multipath-tools
+ on top of the SRP devices to protects against a network failure. If
+ you are not using multipathed IO you can simply format the device as
+ normal.
+
+ 99.. BBuuiillddiinngg LLuussttrree aaggaaiinnsstt OOFFEEDD
+
+ Lustre is a scalable cluster filesystem popular on high performance
+ compute clusters. See http://www.lustre.org <http://www.lustre.org>
+ for more information. lustre can use infiniband as one of its network
+ transports in order to increase performance. The section describes how
+ to compile lustre against the OFED infiniband stack.
+
+ 99..11.. CChheecckk CCoommppaattiibbiilliittyy
+
+ Not all lustre versions are compatible with all OFED or kernel
+ versions. Read the lustre release notes for which versions are
+ supported.
+
+ 99..22.. BBuuiilldd aa lluussttrree ppaattcchheedd kkeerrnneell
+
+ Build a lustre patched kernel as per the instructions on the lustre
+ wiki. Once you have build the kernel keep the configured source tree.
+ It is required for the next step.
+
+ 99..33.. BBuuiilldd OOFFEEDD mmoodduulleess ffoorr tthhee lluussttrree ppaattcchheedd kkeerrnneell..
+
+ Build OFED modules against the newly build lustre patched kernel.
+
+
+
+ module-assistant prepare
+ module-assistant clean ofa-kernel
+ module-assistant -k/path/to/lustre/patched/kernel build ofa-kernel
+
+
+
+ Do not issue a "module-assistant clean" command after the build. The
+ ofa-kernel-module source tree is needed for the next step.
+
+ 99..44.. CCoonnffiigguurree lluussttrree
+
+ You can now configure lustre to build against the lustre patched
+ kernel and the ofa-kernel-module sources.
+
+
+
+ cd lustre-source
+ ./configure --with-o2ib=/usr/src/modules/ofa-kernel --with-linux=/path/to/patched/linux/source \
+ --other-options
+
+
+
+ 1100.. NNeettwwoorrkk TTrroouubblleesshhoooottiinngg
+
+ 1100..11.. iibbddiiaaggnneett
+
+ The ibdiagnet program can be used to troubleshoot potential issues
+ with your infiniband fabric.
+
+
+ ibdiagnet -r
+
+
+ 1111.. FFuurrtthheerr IInnffoorrmmaattiioonn
+
+ Extensive documentation on the OFED software is present in the ofed-
+ docs package.
+
+ The openfabrics alliance webpage can be found here:
+
+ http://www.openfabrics.org/ <http://www.openfabrics.org/>
+
+ The following mailing lists are also useful:
+
+ http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel
+ <http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel>: pkg-
+ ofed-devel: Discussion of debian specific problem or issues.
+
+ http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
+ <http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general>: ofa-
+ general: General discussion of the OFED software.
+
+ Books:
+
+
+
+ Infiniband Network Architecture
+ by MindShare, Inc.; Tom Shanley
+ Publisher: Addison-Wesley Professional
+ Pub Date: October 31, 2002
+ Print ISBN-10: 0-321-11765-4
+
+
+
Modified: branches/ofed-1.4.2/ofed-docs/trunk/debian/docs
URL: http://svn.debian.org/wsvn/pkg-ofed/branches/ofed-1.4.2/ofed-docs/trunk/debian/docs?rev=437&op=diff
==============================================================================
--- branches/ofed-1.4.2/ofed-docs/trunk/debian/docs (original)
+++ branches/ofed-1.4.2/ofed-docs/trunk/debian/docs Fri Aug 7 17:11:42 2009
@@ -3,3 +3,4 @@
ofed_net.conf-example
ofed.conf-example
*txt
+DEBIAN-HOWTO/
More information about the Pkg-ofed-commits
mailing list