Run MPI On Grid'5000

From Grid5000
Revision as of 10:53, 21 January 2019 by Dloup (talk | contribs) (Setting up and starting Open MPI to use high performance interconnect)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
Note.png Note

This tutorial is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Introduction

MPI is a programming interface that enables the communication between processes of a distributed memory system. This tutorial focuses on setting up MPI environments on Grid'5000 and only requires a basic understanding of MPI concepts. For instance, you should know that standard MPI processes live in their own memory space and communicate with other processes by calling library routines to send and receive messages. For a comprehensive tutorials on MPI, see the IDRIS course on MPI. There are several freely-available implementations of MPI, including Open MPI, MPICH2, MPICH, LAM, etc. In this practical session, we focus on the Open MPI implementation.

Before following this tutorial you should already have some basic knowledge of OAR (see the Getting Started tutorial) . For the second part of this tutorial, you should also know the basics about OARGRID (see the Advanced OAR tutorial) and Kadeploy (see the Getting Started tutorial).

Running MPI on Grid'5000

When attempting to run MPI on Grid'5000 you will face a number of challenges, ranging from classical setup problems for MPI software to problems specific to Grid'5000. This practical session aims at driving you through the most common use cases, which are:

  • Setting up and starting Open MPI on a default environment using oarsh.
  • Setting up and starting Open MPI on a default environment using the allow_classic_ssh option.
  • Setting up and starting Open MPI to use high performance interconnect.
  • Setting up and starting Open MPI to run on several sites using oargridsub.
  • Setting up and starting your own Open MPI library inside a Kadeploy image.

Using Open MPI on a default environment

The default Grid'5000 environment provides Open MPI 2.0.2 (see ompi_info).

Creating a sample MPI program

For the purposes of this tutorial, we create a simple MPI program where the MPI process of rank 0 broadcasts an integer (42) to all the other processes. Then, each process prints its rank, the total number of processes and the value he received from the process 0.

In your home directory, create a file ~/mpi/tp.c and copy the source code:

Terminal.png frontend:
mkdir ~/mpi
Terminal.png frontend:
vi ~/mpi/tp.c
#include <stdio.h>
#include <mpi.h>
#include <time.h> /* for the work function only */
#include <unistd.h>

int main (int argc, char *argv []) {
       char hostname[257];
       int size, rank;
       int i, pid;
       int bcast_value = 1;

       gethostname(hostname, sizeof hostname);
       MPI_Init(&argc, &argv);
       MPI_Comm_rank(MPI_COMM_WORLD, &rank);
       MPI_Comm_size(MPI_COMM_WORLD, &size);
       if (!rank) {
            bcast_value = 42;
       }
       MPI_Bcast(&bcast_value,1 ,MPI_INT, 0, MPI_COMM_WORLD );
       printf("%s\t- %d - %d - %d\n", hostname, rank, size, bcast_value);
       fflush(stdout);

       MPI_Barrier(MPI_COMM_WORLD);
       MPI_Finalize();
       return 0;
}

You can then compile your code:

Terminal.png frontend:
mpicc ~/mpi/tp.c -o ~/mpi/tp

Setting up and starting Open MPI on a default environment using oarsh

Submit a job:

Terminal.png frontend:
oarsub -I -l nodes=3

oarsh is the remote shell connector of the OAR batch scheduler. It is a wrapper around the ssh command that handles the configuration of the SSH environment. You can connect to the reserved nodes using oarsh from the submission frontal of the cluster or from any node. As Open MPI defaults to using ssh for remote startup of processes, you need to add the option --mca orte_rsh_agent "oarsh" to your mpirun command line. Open MPI will then use oarsh in place of ssh.

Terminal.png node:
mpirun --mca orte_rsh_agent "oarsh" -machinefile $OAR_NODEFILE ~/mpi/tp

You can also set an environment variable (usually in your .bashrc):

Terminal.png bashrc:
export OMPI_MCA_orte_rsh_agent=oarsh
Terminal.png node:
mpirun -machinefile $OAR_NODEFILE ~/mpi/tp

Open MPI also provides a configuration file for --mca parameters. In your home directory, create a file as ~/.openmpi/mca-params.conf

orte_rsh_agent=oarsh
filem_rsh_agent=oarcp

You should have something like:

helios-52       - 4 - 12 - 42
helios-51       - 0 - 12 - 42
helios-52       - 5 - 12 - 42
helios-51       - 2 - 12 - 42
helios-52       - 6 - 12 - 42
helios-51       - 1 - 12 - 42
helios-51       - 3 - 12 - 42
helios-52       - 7 - 12 - 42
helios-53       - 8 - 12 - 42
helios-53       - 9 - 12 - 42
helios-53       - 10 - 12 - 42
helios-53       - 11 - 12 - 42

You may have (lot's of) warning messages if Open MPI cannot take advantage of any high performance hardware. At this point of the tutorial, this is not important as we will learn how to select clusters with high performance interconnect in greater details below. Error messages might look like this:

[[2616,1],2]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: helios-8.sophia.grid5000.fr

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
warning:regcache incompatible with malloc
warning:regcache incompatible with malloc
warning:regcache incompatible with malloc

or like this:

[griffon-80.nancy.grid5000.fr:04866] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_mtl_mx: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[griffon-80.nancy.grid5000.fr:04866] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_btl_mx: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[griffon-80.nancy.grid5000.fr:04865] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_mtl_mx: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[griffon-80.nancy.grid5000.fr:04867] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_mtl_mx: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
...

You could use FAQ#How_to_use_MPI_in_Grid5000.3F to avoid this warnings.

Setting up and starting Open MPI on a default environment using allow_classic_ssh

If you prefer using ssh as a connector instead of oarsh, submit a job with the allow_classic_ssh type:

Terminal.png frontend:
oarsub -I -t allow_classic_ssh -l nodes=3

Launch your parallel job:

Terminal.png node:
mpirun -machinefile $OAR_NODEFILE ~/mpi/tp
Note.png Note

Using allow_classic_ssh option avoids OAR resources confinement mechanism with cpuset to restrict the jobs on assigned resources. Therefore, allow_classic_ssh cannot be used with jobs sharing nodes between users (i.e. for reservations at the core level).

Setting up and starting Open MPI to use high performance interconnect

By default, Open MPI tries to use any high performance interconnect (e.g. Infiniband, Omni-Path) it can find. Options are available to either select or disable an interconnect:

MCA parameters (--mca) can be used to select the drivers that are used at run-time by Open MPI. To learn more about the MCA parameters, see also:

To learn more about specific Omni-Path tools, refer to this page.

If you want to disable native support for high performance networks, use --mca btl self,sm,tcp --mca mtl ^psm2. The first part disables the openib backend, and the second part disables the psm2 backend (used by Omni-Path). This will switch to TCP backend of Open MPI.

Nodes with Infiniband or Omni-Path interfaces also provide an IP over Infiniband interface (these interfaces are named ibX), and can still be used by the TCP backend. To also disable their use, use --mca btl_tcp_if_exclude ib0,lo or select a specific interface with --mca btl_tcp_if_include eno2. You will ensure that 'regular' Ethernet interface is used.

We will be using NetPIPE to check the performances of high performance interconnects.

To download, extract and compile NetPIPE, do:

Terminal.png frontend:
cd ~/mpi
Terminal.png frontend:
tar -xf NetPIPE-3.7.1.tar.gz
Terminal.png frontend:
cd NetPIPE-3.7.1
Terminal.png frontend:
make mpi

As NetPipe only works between two MPI processes, reserve only one core in two distinct nodes. If your reservation includes more resources, you will have to create a MPI machinefile file with only two entries, such as follow:

Terminal.png frontend:
oarsub -I -l nodes=2
Terminal.png node:
uniq $OAR_NODEFILE | head -n 2 > /tmp/machinefile

Infiniband hardware is available on several sites. For example, you will find clusters with Infiniband interconnect in Rennes (20G), Nancy (20G) and Grenoble (20G & 40G).

To reserve one core of two distinct nodes with:

  • a 20G InfiniBand interconnect (DDR, Double Data Rate):
Terminal.png frontend:
oarsub -I -l /nodes=2/core=1 -p "ib='DDR'"
  • a 40G InfiniBand interconnect (QDR, Quad Data Rate):
Terminal.png frontend:
oarsub -I -l /nodes=2/core=1 -p "ib='QDR'"

To check if the support for InfiniBand is available in Open MPI, run:

Terminal.png node:
ompi_info | grep openib

you should see something like this:

                MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v2.0.2)

To start the network benchmark, use:

Terminal.png node:
cd ~/mpi/NetPIPE-3.7.1
Terminal.png node:
mpirun --mca orte_rsh_agent "oarsh" -machinefile /tmp/machinefile NPmpi

Without high performance interconnect, results looks like this:

    0:         1 bytes   4080 times -->      0.31 Mbps in      24.40 usec     
    1:         2 bytes   4097 times -->      0.63 Mbps in      24.36 usec     
    ...
    122: 8388608 bytes      3 times -->    896.14 Mbps in   71417.13 usec
    123: 8388611 bytes      3 times -->    896.17 Mbps in   71414.83 usec

The latency is given by the last column for a 1 byte message; the maximum throughput is given by the last line (896.17 Mbps in that case).

With Infiniband 40G (QDR), you should have much better performance that using Ethernet:

 0:       1 bytes  30716 times -->      4.53 Mbps in       1.68 usec
 1:       2 bytes  59389 times -->      9.10 Mbps in       1.68 usec
...
121: 8388605 bytes     17 times -->  25829.13 Mbps in    2477.82 usec
122: 8388608 bytes     20 times -->  25841.35 Mbps in    2476.65 usec
123: 8388611 bytes     20 times -->  25823.40 Mbps in    2478.37 usec

Less than 2 ms of latency and almost 26 Gbit/s of bandwitdh !

Specificity of uva and uvb cluster on Sophia

uva and uvb are connected to a 40G QDR Infiniband network. But this network uses a partition key : PKEY

As a result, if you want to use mpi with the infiniband network, you have to specify the partition key (which is 8100) with the following option --mca btl_openib_pkey "0x8100"

More advanced use cases

Running MPI on several sites at once

In this section, we are going to execute a MPI process over several Grid'5000 sites. In this example we will use the following sites: Rennes, Sophia and Grenoble, using oargrid for making the reservation (see the Advanced OAR tutorial for more information).

Warning.png Warning

Open MPI tries to figure out the best network interface to use at run time. However, selected networks are not always "production" Grid'5000 network which is routed between sites. In addition, only TCP implementation will work between sites, as high performance networks are only available from the inside of a site. To ensure correct network is selected, add the option --mca opal_net_private_ipv4 "192.168.0.0/16" --mca btl_tcp_if_exclude ib0,lo --mca btl self,sm,tcp to mpirun

Warning.png Warning

By default, Open MPI may use only the short name of the nodes specified into the nodesfile; but to join grid5000 nodes that are located on different sites, we must use the FQDN names. For Open Mpi to correctly use FQDN names of the nodes, you must add the following option to mpirun: --mca orte_keep_fqdn_hostnames t

The MPI program must be available on each site you want to use. From the frontend of one site, copy the mpi/ directory to the two other sites. You can do that with rsync. Suppose that you are connected in Sophia and that you want to copy Sophia's mpi/ directoy to Grenoble and Rennes.

Terminal.png fsophia:
rsync -avz ~/mpi/ nancy.grid5000.fr:mpi/
Terminal.png fsophia:
rsync -avz ~/mpi/ grenoble.grid5000.fr:mpi/

(you can also add the --delete option to remove extraneous files from the mpi directory of Nancy and Grenoble).

Reserve nodes in each site from any frontend with oargridsub (you can also add options to reserve nodes from specific clusters if you want to):

Terminal.png frontend:
oargridsub -w 02:00:00 nancy:rdef="nodes=2",grenoble:rdef="nodes=2",sophia:rdef="nodes=2" > oargrid.out

Get the oargrid Id and Job key from the output of oargridsub:

Terminal.png frontend:
export OAR_JOB_KEY_FILE=$(grep "SSH KEY" oargrid.out | cut -f2 -d: | tr -d " ")
Terminal.png frontend:
export OARGRID_JOB_ID=$(grep "Grid reservation id" oargrid.out | cut -f2 -d=)

Get the node list using oargridstat and copy the list to the first node:

Terminal.png frontend:
oargridstat -w -l $OARGRID_JOB_ID | grep -v ^$ > ~/gridnodes
Terminal.png frontend:
oarcp ~/gridnodes $(head -1 ~/gridnodes):

Connect to the first node:

Terminal.png frontend:
oarsh $(head -1 ~/gridnodes)

And run your MPI application:

Terminal.png node:
cd ~/mpi/
Terminal.png node:
mpirun -machinefile ~/gridnodes --mca orte_rsh_agent "oarsh" --mca opal_net_private_ipv4 "192.168.0.0/16" --mca btl_tcp_if_exclude ib0,lo --mca btl self,sm,tcp --mca orte_keep_fqdn_hostnames t tp


Make your own Kadeploy image with latest Open MPI version

Build a Kadeploy image

If you need latest Open MPI version, or use Open MPI with specific compilation options, you must recompile Open MPI from sources. In this section we are going to build an image that includes latest version of Open MPI built from sources. Note that you could also build and install this custom Open MPI in your home directory, without requiring deploying (i.e., using ./configure --prefix=$HOME/openmpi/).

This image will be based on jessie-x64-base. Let's deploy it:

Terminal.png frontend:
oarsub -I -t deploy -l nodes=1,walltime=2
Terminal.png frontend:
kadeploy3 -f $OAR_NODEFILE -e jessie-x64-base -k

Connect to reserved node as root:

Terminal.png frontend:
ssh root@$(head -1 $OAR_NODEFILE)

Download Open MPI sources from official website:

Terminal.png node:
cd /tmp/
Terminal.png node:
tar -xf openmpi-1.10.2.tar.bz2
Terminal.png node:
cd openmpi-1.10.2

Install build dependencies:

Terminal.png node:
apt-get update
Terminal.png node:
apt-get -y install g++ make gfortran f2c libblas-dev

Configure, compile and install:

Terminal.png node:
./configure --libdir=/usr/local/lib64 --with-memory-manager=none
Terminal.png node:
make -j8
Terminal.png node:
make install

To run our MPI applications, we create a dedicated user named mpi. We add it to the rdma group to allow access to Infiniband hardware. Also, we copy the ~root/authorized_keys files so that we can login as user mpi from the frontend. We also create an SSH key for identifying the mpi user (needed by Open MPI).

groupadd rdma
useradd -m -g rdma mpi -d /var/mpi -s /bin/bash
echo "* hard memlock unlimited" >> /etc/security/limits.conf
echo "* soft memlock unlimited" >> /etc/security/limits.conf

mkdir ~mpi/.ssh
cp ~root/.ssh/authorized_keys ~mpi/.ssh
chown -R mpi ~mpi/.ssh
su - mpi
ssh-keygen -N "" -P "" -f /var/mpi/.ssh/id_rsa
cat .ssh/id_rsa.pub >> ~/.ssh/authorized_keys
echo "        StrictHostKeyChecking no" >> ~/.ssh/config
exit # exit session as MPI user
exit # exit the root connection to the node

# You can then copy your file from the frontend to the mpi home directory:
rsync -avz ~/mpi/ mpi@$(head -1 $OAR_NODEFILE):mpi/ # copy the tutorial

Save the newly created image by using tgz-g5k:

Terminal.png frontend:
ssh root@$(head -1 $OAR_NODEFILE) tgz-g5k > $HOME/public/jessie-openmpi.tgz

Copy the description file of jessie-x64-base:

Terminal.png frontend:
kaenv3 -p jessie-x64-base | grep -v visibility > $HOME/public/jessie-openmpi.dsc

Change the image name in the description file; we will use an http URL for multi-site deployment:

Terminal.png frontend:
sed -i "s,server://.*images/jessie-x64-base.*,http://public.$(hostname -f| cut -d. -f2).grid5000.fr/~$USER/jessie-openmpi.tgz," $HOME/public/jessie-openmpi.dsc

Release your job:

Terminal.png frontend:
oardel $OAR_JOB_ID

Use your Kadeploy image

Single site

Reserve some nodes and deploy them:

Terminal.png frontend:
oarsub -I -t deploy -l /nodes=3
Terminal.png frontend:
kadeploy3 -a $HOME/public/jessie-openmpi.dsc -f $OAR_NODEFILE -k

Copy machines file and connect to first node:

Terminal.png frontend:
scp $OAR_NODEFILE mpi@$(head -1 $OAR_NODEFILE):nodes
Terminal.png frontend:
ssh mpi@$(head -1 $OAR_NODEFILE)

Copy your MPI application to other nodes and run it:

Terminal.png node:
cd ~/mpi/
Terminal.png node:
/usr/local/bin/mpicc tp.c -o tp
Terminal.png node:
for node in $(uniq ~/nodes | grep -v $(hostname)); do scp ~/mpi/tp $node:~/mpi/tp; done
Terminal.png node:
/usr/local/bin/mpirun -machinefile ~/nodes ./tp

Multiple sites

Choose three clusters from 3 different sites.

Terminal.png frontend:
oargridsub -t deploy -w 02:00:00 cluster1:rdef="nodes=2",cluster2:rdef="nodes=2",cluster3:rdef="nodes=2" > oargrid.out
Terminal.png frontend:
export OARGRID_JOB_ID=$(grep "Grid reservation id" oargrid.out | cut -f2 -d=)

Get the node list using oargridstat:

Terminal.png frontend:
oargridstat -w -l $OARGRID_JOB_ID | grep grid > ~/gridnodes

Deploy on all sites using the --multi-server option :

Terminal.png frontend:
kadeploy3 -f gridnodes -a $HOME/public/jessie-openmpi.dsc -k --multi-server

Copy machines file and connect to first node:

Terminal.png frontend:
scp ~/gridnodes mpi@$(head -1 ~/gridnodes):
Terminal.png frontend:
ssh mpi@$(head -1 ~/gridnodes)

Copy your MPI application to other nodes and run it:

Terminal.png node:
cd ~/mpi/
Terminal.png node:
/usr/local/bin/mpicc tp.c -o tp
Terminal.png node:
for node in $(uniq ~/gridnodes | grep -v $(hostname)); do scp ~/mpi/tp $node:~/mpi/tp; done
Terminal.png node:
/usr/local/bin/mpirun -machinefile ~/gridnodes --mca btl self,sm,tcp --mca opal_net_private_ipv4 "192.168.0.0/16" --mca btl_tcp_if_exclude ib0,lo --mca orte_keep_fqdn_hostnames 1 tp