[Pkg-ofed-commits] r359 - /trunk/howto/infiniband-howto.sgml

gmpc-guest at alioth.debian.org gmpc-guest at alioth.debian.org
Fri Jun 5 08:40:36 UTC 2009


Author: gmpc-guest
Date: Fri Jun  5 08:40:36 2009
New Revision: 359

URL: http://svn.debian.org/wsvn/pkg-ofed/?sc=1&rev=359
Log:
Add mpi test from the mpitest suite

Modified:
    trunk/howto/infiniband-howto.sgml

Modified: trunk/howto/infiniband-howto.sgml
URL: http://svn.debian.org/wsvn/pkg-ofed/trunk/howto/infiniband-howto.sgml?rev=359&op=diff
==============================================================================
--- trunk/howto/infiniband-howto.sgml (original)
+++ trunk/howto/infiniband-howto.sgml Fri Jun  5 08:40:36 2009
@@ -540,15 +540,19 @@
 <p>
 Uses who want to run MPI jobs will need to have write permissions for the following devices:
 <tscreen>
+<verb>
  /dev/infiniband/uverbs*
 /dev/infiniband/rdma_cm*
+</verb>
 </tscreen>
 The simplest way to do this is to add the users to the rdma group. If that is not suitiable for
 your site, you can change the permissions and ownership of these devices by editing the following
 udev rules:
 <tscreen>
+<verb>
 /etc/udev/rules.d/50-udev.rules
 /etc/udev/rules.d/91-permissions.rules
+</verb>
 </tscreen>
 
 <p>
@@ -557,44 +561,12 @@
 *               hard    memlock         unlimited
 </tscreen>
 
-<sect1>Compile the test program
-<p>
-Install the openmpi development libraries:
-<tscreen>
-aptitude install openmpi-dev
-</tscreen>
- 
-We will use a simple "hello world" program to test MPI functionality.
-<tscreen>
-<code>
-/* C Example */
-#include <stdio.h>
-#include <mpi.h>
-
-
-int main (argc, argv)
-     int argc;
-     char *argv[];
-{
-  int rank, size;
-  MPI_Init (&amp;argc, &amp;argv);/* starts MPI */
-  MPI_Comm_rank (MPI_COMM_WORLD, &amp;rank);/* get current process id */
-  MPI_Comm_size (MPI_COMM_WORLD, &amp;size);/* get number of processes */
-  printf( "Hello world from process %d of %d\n", rank, size );
-  MPI_Finalize();
-  return 0;
-}
-</code>
-</tscreen>
-
-Compile the hello-world.c program:
-
-<tscreen>
-mpicc hello-world.c -o hello-world
-</tscreen>
-
-Place the resulting binary on each node in you cluster. You can either copy  it to a local filesystem on each node, or
-more conveniently, place it on a network filesystem visible to all the hosts.
+<sect1>Install the mpi test programs
+<p>
+Check the mpitests package is installed. 
+<tscreen>
+aptitude install mpitests
+</tscreen>
 
 <sect1>Configure Hosts
 <p>
@@ -610,39 +582,69 @@
 </verb>
 </tscreen>
 
-<sect1>Launch the program
-<p>
-
-Lauch the test program using the mpirun command. The -n option specifies the number of mpi processes to launch and
-the -h option points to the hostfile created above.
-For our
-testing we will also force mpi to ignore any TCP interfaces and make it extra-chatty during startup. 
-(--mca btl_openib_verbose 1 --mca btl ^tcp options). These options are normally uncessesary, as openMPI will use 
-any infiniband networks it detects in  preference to any  TCP networks.
-
-<verb>
-mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile /path/to/hostfile /path/to/hello-world
-</verb>
-
-If all goes well you should see debugging messages from both hosts, together with the job output.
-
-<tscreen>
-<verb>
-[hostA][[64185,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 25418
-[hostA][[64185,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
-[hostA][[64185,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
-[hostA][[64185,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: default
-[hostB][[64185,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 25418
-[hostB][[64185,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
-[hostB][[64185,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
-[hostB][[64185,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: default
-Hello world from process 0 of 2
-Hello world from process 1 of 2
+<sect1>Run the MPI PingPong benchmark.
+<p>
+
+We will use the MPI PingPong benchmark for our testing.  By default, openmpi should use inifiniband networks in preference to any tcp networks it finds. However, we will force mpi to be extra-chatty during the test  to ensure  that we are really using the infiniband interfaces.
+
+(ADDME: Is there a better way to confirm which networks openmpi is using?)
+
+
+<verb>
+mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile /path/to/hostfile IMB-MPI1 PingPong
+</verb>
+
+If all goes well you should see openib debugging messages from both hosts, together with the job output.
+
+<tscreen>
+<verb>
+&lt;snip&gt;
+# PingPong
+[HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+[HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+[HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+[HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
+
+#---------------------------------------------------
+# Benchmarking PingPong 
+# #processes = 2 
+#---------------------------------------------------
+       #bytes #repetitions      t[usec]   Mbytes/sec
+            0         1000         1.53         0.00
+            1         1000         1.44         0.66
+            2         1000         1.42         1.34
+            4         1000         1.41         2.70
+            8         1000         1.48         5.15
+           16         1000         1.50        10.15
+           32         1000         1.54        19.85
+           64         1000         1.79        34.05
+          128         1000         3.01        40.56
+          256         1000         3.56        68.66
+          512         1000         4.46       109.41
+         1024         1000         5.37       181.92
+         2048         1000         8.13       240.25
+         4096         1000        10.87       359.48
+         8192         1000        15.97       489.17
+        16384         1000        30.54       511.68
+        32768         1000        55.01       568.12
+        65536          640       122.20       511.46
+       131072          320       207.20       603.27
+       262144          160       377.10       662.96
+       524288           80       706.21       708.00
+      1048576           40      1376.93       726.25
+      2097152           20      1946.00      1027.75
+      4194304           10      3119.29      1282.34
 </verb>
 </tscreen>
 
 If you encounter any errors read the excellent OpenMPI troubleshooting guide. <url url="http://www.openmpi.org"
 name="http://www.openmpi.org">
+
+If you want to compare infiniband performance with your ethernet/TCP networks, you can re-run the tests using flags to tell openmpi to use your ethernet network. (The example below assumes that your test nodes are connected via eth0).
+
+<verb>
+ mpirun --mca btl ^openib --mca btl_tcp_if_include eth0 --hostfile hostfile -n 2 IMB-MPI1 -benchmark PingPong
+</verb>
 
 
 <sect>SDP




More information about the Pkg-ofed-commits mailing list