Run MPI On Grid'5000

See also: HPC and HTC tutorial | Run MPI On Grid'5000 | GPUs on Grid5000 | GNU Parallel | Modules | Conda | Guix | Singularity | Deep Learning Frameworks

	Note
	This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Introduction

MPI is a programming interface that enables the communication between processes of a distributed memory system. This tutorial focuses on setting up MPI environments on Grid'5000 and only requires a basic understanding of MPI concepts. For instance, you should know that standard MPI processes live in their own memory space and communicate with other processes by calling library routines to send and receive messages. For a comprehensive tutorials on MPI, see the IDRIS course on MPI. There are several freely-available implementations of MPI, including Open MPI, MPICH2, MPICH, LAM, etc. In this practical session, we focus on the Open MPI implementation.

Before following this tutorial you should already have some basic knowledge of OAR (see the Getting Started tutorial) . For the second part of this tutorial, you should also know the basics about OARGRID (see the Advanced OAR tutorial).

Running MPI on Grid'5000

When attempting to run MPI on Grid'5000 you will face a number of challenges, ranging from classical setup problems for MPI software to problems specific to Grid'5000. This practical session aims at driving you through the most common use cases, which are:

• Setting up and starting Open MPI on a default environment using oarsh.
• Setting up and starting Open MPI on a default environment using the allow_classic_ssh option.
• Setting up and starting Open MPI to use high performance interconnect.
• Setting up and starting latest Open MPI library version.
• Setting up and starting Open MPI to run on several sites using oargridsub.

Using Open MPI on a default environment

The default Grid'5000 environment provides Open MPI 4.1.0 (see ompi_info).

Creating a sample MPI program

For the purposes of this tutorial, we create a simple MPI program where the MPI process of rank 0 broadcasts an integer (42) to all the other processes. Then, each process prints its rank, the total number of processes and the value he received from the process 0.

In your home directory, create a file ~/mpi/tp.c and copy the source code:

frontend:

mkdir ~/mpi

frontend:

vi ~/mpi/tp.c

#include <stdio.h>
#include <mpi.h>
#include <time.h> /* for the work function only */
#include <unistd.h>

int main (int argc, char *argv []) {
       char hostname[257];
       int size, rank;
       int bcast_value = 1;

       gethostname(hostname, sizeof hostname);
       MPI_Init(&argc, &argv);
       MPI_Comm_rank(MPI_COMM_WORLD, &rank);
       MPI_Comm_size(MPI_COMM_WORLD, &size);
       if (!rank) {
            bcast_value = 42;
       }
       MPI_Bcast(&bcast_value,1 ,MPI_INT, 0, MPI_COMM_WORLD );
       printf("%s\t- %d - %d - %d\n", hostname, rank, size, bcast_value);
       fflush(stdout);

       MPI_Barrier(MPI_COMM_WORLD);
       MPI_Finalize();
       return 0;
}

You can then compile your code:

frontend:

mpicc ~/mpi/tp.c -o ~/mpi/tp

Setting up and starting Open MPI on a default environment using oarsh

Submit a job:

frontend:

oarsub -I -l nodes=3

OAR batch scheduler provides the $OAR_NODEFILE file, containing the list of job's nodes (one line per core). It also uses oarsh as remote shell connector. It is a wrapper around the ssh command that handles the configuration of the SSH environment. You can connect to the reserved nodes using oarsh from the submission frontal of the cluster or from any node. As Open MPI defaults to using ssh for remote startup of processes, you need to add the option --mca orte_rsh_agent "oarsh" to your mpirun command line. Open MPI will then use oarsh in place of ssh.

node:

mpirun --mca orte_rsh_agent "oarsh" -machinefile $OAR_NODEFILE ~/mpi/tp

You can also set an environment variable (usually in your .bashrc):

bashrc:

export OMPI_MCA_orte_rsh_agent=oarsh

node:

mpirun -machinefile $OAR_NODEFILE ~/mpi/tp

Open MPI also provides a configuration file for --mca parameters. In your home directory, create a file as ~/.openmpi/mca-params.conf

orte_rsh_agent=oarsh
filem_rsh_agent=oarcp

You should have something like:

helios-52       - 4 - 12 - 42
helios-51       - 0 - 12 - 42
helios-52       - 5 - 12 - 42
helios-51       - 2 - 12 - 42
helios-52       - 6 - 12 - 42
helios-51       - 1 - 12 - 42
helios-51       - 3 - 12 - 42
helios-52       - 7 - 12 - 42
helios-53       - 8 - 12 - 42
helios-53       - 9 - 12 - 42
helios-53       - 10 - 12 - 42
helios-53       - 11 - 12 - 42

You may have (lot's of) warning messages if Open MPI cannot take advantage of any high performance hardware. At this point of the tutorial, this is not important as we will learn how to select clusters with high performance interconnect in greater details below. Error messages might look like this:

[1637577186.512697] [taurus-10:2765 :0]      rdmacm_cm.c:638  UCX  ERROR rdma_create_event_channel failed: No such device
[1637577186.512752] [taurus-10:2765 :0]     ucp_worker.c:1432 UCX  ERROR failed to open CM on component rdmacm with status Input/output error
[taurus-10.lyon.grid5000.fr:02765] ../../../../../../ompi/mca/pml/ucx/pml_ucx.c:273  Error: Failed to create UCP worker

To tell OpenMPI not try to use high performance hardware and avoid those warnings message, use the following options:

node:

mpirun --mca orte_rsh_agent "oarsh" --mca pml ^ucx -machinefile $OAR_NODEFILE $HOME/mpi_programm

Setting up and starting Open MPI on a default environment using allow_classic_ssh

If you prefer using ssh as a connector instead of oarsh, submit a job with the allow_classic_ssh type:

frontend:

oarsub -I -t allow_classic_ssh -l nodes=3

Launch your parallel job:

node:

mpirun -machinefile $OAR_NODEFILE ~/mpi/tp

	Note
	Using allow_classic_ssh option avoids OAR resources confinement mechanism with cpuset to restrict the jobs on assigned resources. Therefore, allow_classic_ssh cannot be used with jobs sharing nodes between users (i.e. for reservations at the core level).

Setting up and starting Open MPI to use high performance interconnect

Open MPI provides several alternative compononents to use High Performance Interconnect hardware (such as Infiniband or Omni-Path).

MCA parameters (--mca) can be used to select the component that are used at run-time by OpenMPI. To learn more about the MCA parameters, see also:

• The Open MPI FAQ about tuning parameters
• How do I tell Open MPI which IP interfaces / networks to use?
• The Open MPI documentation about OpenFabrics (ie: Infiniband)
• The Open MPI documention about Omni-path. Intel documentation about Omni-Path tools.

Open MPI packaged in Grid'5000 debian11 includes ucx, ofi and openib components to make use of Infiniband network, and psm2 and ofi to make use of Omnipath network.

If you want some of these components, you can for example use --mca pml ^ucx --mca mtl ^psm2,ofi --mca btl ^ofi,openib. This disables all high performance components mentionned above and will force Open MPI to use its TCP backend.

Infiniband network

By default, OpenMPI tries to use Infiniband high performance interconnect using the UCX component. When Infiniband network is available, it will provide best results in most of cases (UCX even uses multiple interfaces at a time when available).

Omni-Path network

For Open MPI to work with Omni-Path network hardware, PSM2 component must be used. It is necessary to explicitely disable other components:

node:

mpirun -machinefile $OAR_NODEFILE -mca mtl psm2 -mca pml ^ucx,ofi -mca btl ^ofi,openib ~/mpi/tp

IP over Infiniband or Omni-Path

Nodes with Infiniband or Omni-Path network interfaces also provide an IP over Infiniband interface (these interfaces are named ibX). The TCP backend of Open MPI will try to use them by default.

You can explicitely select interfaces used by the TCP backend using for instance --mca btl_tcp_if_exclude ib0,lo (to avoid using IP over Infiniband and local interfaces) or --mca btl_tcp_if_include eno2 (to force using the 'regular' Ethernet interface eno2).

Benckmarking

We will be using OSU micro benchmark to check the performances of high performance interconnects.

To download, extract and compile our benchmark, do:

frontend:

cd ~/mpi

frontend:

wget https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.8.tgz

frontend:

tar xf osu-micro-benchmarks-5.8.tgz

frontend:

cd osu-micro-benchmarks-5.8/

frontend:

./configure CC=$(which mpicc) CXX=$(which mpicxx)

frontend:

make

As we will benchmark two MPI processes, reserve only one core in two distinct nodes.

frontend:

oarsub -I -l nodes=2

To start the network benchmark, use:

node:

mpirunmpirun --mca orte_rsh_agent "oarsh" -machinefile $OAR_NODEFILE -npernode 1 ~/mpi/osu-micro-benchmarks-5.8/mpi/pt2pt/osu_latency

The option ``-npernode 1`` tells to only spawn one process on each node, as the benchmark requires only two processes to communicate.

You can then try to compare performance of various network hardware available on Grid'5000. See for instance Network section of Hardware#Network_interface_models.

OAR can select nodes according to properties related to network performance. For example:

• To reserve one core of two distinct nodes with a 56Gbps InfiniBand interconnect:

frontend:

oarsub -I -l /nodes=2/core=1 -p "ib_rate=56"

• To reserve one core of two distinct nodes with a 100Gbps Omni-Path interconnect:

frontend:

oarsub -I -l /nodes=2/core=1 -p "opa_rate=100"

Use a newer Open MPI version using modules

If you need latest Open MPI version you should use the module command.

frontend:

module av openmpi

[...]
openmpi/4.1.1_gcc-8.3.0 (D)
[...]

frontend:

module load openmpi

frontend:

mpirun --version

You must recompile simple MPI example on the frontend with this new version.

frontend:

mpicc ~/mpi/tp.c -o ~/mpi/tp

From your job, you must ensure the same Open MPI version is used on every nodes:

frontend:

oarsub -I -l nodes=3

node:

module load openmpi

node:

$(which mpirun) --mca orte_rsh_agent "oarsh" -machinefile $OAR_NODEFILE ~/mpi/tp

Note that $(which mpirun) command is used in this last step to ensure mpirun from the module environment is used.

More advanced use cases

Running MPI on several sites at once

In this section, we are going to execute a MPI process over several Grid'5000 sites. In this example we will use the following sites: Rennes, Sophia and Grenoble, using oargrid for making the reservation (see the Advanced OAR tutorial for more information).

Warning

Open MPI tries to figure out the best network interface to use at run time. However, selected networks are not always "production" Grid'5000 network which is routed between sites. In addition, only TCP implementation will work between sites, as high performance networks are only available from the inside of a site. To ensure correct network is selected, add the option --mca opal_net_private_ipv4 "192.168.0.0/16" --mca btl_tcp_if_exclude ib0,lo --mca btl self,sm,tcp to mpirun

	Warning
	By default, Open MPI may use only the short name of the nodes specified into the nodesfile; but to join grid5000 nodes that are located on different sites, we must use the FQDN names. For Open Mpi to correctly use FQDN names of the nodes, you must add the following option to mpirun: --mca orte_keep_fqdn_hostnames t

The MPI program must be available on each site you want to use. From the frontend of one site, copy the mpi/ directory to the two other sites. You can do that with rsync. Suppose that you are connected in Sophia and that you want to copy Sophia's mpi/ directoy to Grenoble and Rennes.

fsophia:

rsync -avz ~/mpi/ nancy.grid5000.fr:mpi/

fsophia:

rsync -avz ~/mpi/ grenoble.grid5000.fr:mpi/

(you can also add the --delete option to remove extraneous files from the mpi directory of Nancy and Grenoble).

Reserve nodes in each site from any frontend with oargridsub (you can also add options to reserve nodes from specific clusters if you want to):

frontend:

oargridsub -w 02:00:00 nancy:rdef="nodes=2",grenoble:rdef="nodes=2",sophia:rdef="nodes=2" > oargrid.out

Get the oargrid Id and Job key from the output of oargridsub:

frontend:

export OAR_JOB_KEY_FILE=$(grep "SSH KEY" oargrid.out | cut -f2 -d: | tr -d " ")

frontend:

export OARGRID_JOB_ID=$(grep "Grid reservation id" oargrid.out | cut -f2 -d=)

Get the node list using oargridstat and copy the list to the first node:

frontend:

oargridstat -w -l $OARGRID_JOB_ID | grep -v ^$ > ~/gridnodes

frontend:

oarcp ~/gridnodes $(head -1 ~/gridnodes):

Connect to the first node:

frontend:

oarsh $(head -1 ~/gridnodes)

And run your MPI application:

node:

cd ~/mpi/

node:

mpirun -machinefile ~/gridnodes --mca orte_rsh_agent "oarsh" --mca opal_net_private_ipv4 "192.168.0.0/16" --mca btl_tcp_if_exclude ib0,lo --mca btl self,sm,tcp --mca orte_keep_fqdn_hostnames t tp

FAQ

Passing environment variables to nodes

While some batch schedulers (e.g. Slurm) transparently pass environment variables from the head node shell to all execution nodes given to mpirun. OAR does not (OAR provides no more than what OpenSSH does, be it used directly when oarsub is called with -t all_classic_ssh or through the oarsh wrapper). Thus OAR leaves this responsibility of environment variables passing to mpirun.

Therefore, in order to have more than the default environment variables (OMPI_* variables) passed/set on execution nodes, one has different options:

use the -x VAR option of mpirun, possibly for each variable to pass (WARNING, -x is depreacted)

Example:

mpirun -machinefile $OAR_NODE_FILE --mca orte_rsh_agent "oarsh" -x MY_ENV1 -x MY_ENV2 -x MY_ENV3="value3" ~/bin/mpi_test

use the --mca mca_base_env_list "ENV[;...]" option of mpirun

Example:

mpirun -machinefile $OAR_NODE_FILE --mca orte_rsh_agent "oarsh" --mca mca_base_env_list "MY_ENV1;MY_ENV2;MY_ENV3=value3" ~/bin/mpi_test

set the mca_base_env_list "ENV[;...]" option in the ~/.openmpi/mca-params.conf file

This way passing variable become transparent to the mpirun command line, which becomes:

mpirun -machinefile $OAR_NODE_FILE --mca orte_rsh_agent "oarsh" ~/bin/mpi_test

Rq:

• orte_rsh_agent="oarsh" can be set in the ~/.openmpi/mca-params.conf configuration file as well (but only if using oarsh as the connector)
• -x and --mca mca_base_env_list cannot coexist.

This could especially be useful to pass OpenMP variables, such as OMP_NUM_THREADS.

More info in OpenMPI manual pages.

See also: HPC and HTC tutorial | Run MPI On Grid'5000 | GPUs on Grid5000 | GNU Parallel | Modules | Conda | Guix | Singularity | Deep Learning Frameworks