Run MPI On Grid'5000

	Warning
	Practical session under construction.

Running MPI on Grid'5000

When attempting to run MPI on Grid'5000 you'll be faced with a number of challenges, ranging from classical setup problems for MPI software to problems specific to Grid'5000. This practical session aims at driving you through the most common uses cases, which are

• setting up and starting openMPI on a default environment using allow_classic_ssh
• setting up and starting openMPI on a default environment using oarsh
• setting up and starting openMPI on a kadeploy image
• setting up and starting openMPI to use high performance interconnect

Pre-requisite

• Basic knowledge of MPI; if you don't know MPI, you can read: Grid_computation
• Get OpenMPI here : http://www.open-mpi.org/software/ompi/v1.4/

=Overwiew Currently, the default environment is not the same on every sites, therefore, you don't have the same version of openmpi on every site. If you want to use OpenMPI for a grid experiment, you will have to install your own MPI version; You have two possibility:

• install Openmpi on your home dir (but you should recompile and install on a all the sites you want to use! )
• use the same kadeploy image and deploy it on the the sites you want to use

If you are only interested on a single site experiement, you may use the version providede by the default environment.

Compilation

• Make a reservation (or connect to the compilation machine if available (compil.<node>.site.grid5000.fr) and compile it with the option you like.

mkdir -p $HOME/src/mpi
oarsub -I  
./configure --prefix=$HOME/openmpi/  --with-memory-manager=none
make -j4

• Install it on your home directory

make install

Setting up and starting OpenMPI on a default environment using allow_classic_ssh

• oarsub -I -t allow_classic_ssh -l nodes=3
• Code to test :

$ mkdir -p $HOME/src/mpi
$ vi $HOME/src/mpi/tp.c
#include <stdio.h>
#include <mpi.h>
#include <time.h> /* for the work function only */

int main (int argc, char *argv []) {
       char hostname[257];
       int size, rank;
       int i, pid;
       int bcast_value = 1;

       gethostname (hostname, sizeof hostname);
       MPI_Init (&argc, &argv);
       MPI_Comm_rank (MPI_COMM_WORLD, &rank);
       MPI_Comm_size (MPI_COMM_WORLD, &size);
       if (!rank) {
            bcast_value = 42;
       }
       MPI_Bcast (&bcast_value,1 ,MPI_INT, 0, MPI_COMM_WORLD );
       printf("%s\t- %d - %d - %d\n", hostname, rank, size, bcast_value);
       fflush(stdout);

       MPI_Barrier (MPI_COMM_WORLD);
       MPI_Finalize ();
       return 0;
}

• Compile your code

$ $HOME/openmpi/bin/mpicc src/mpi/tp.c -o src/mpi/tp

• Use this script to launch

NODES=`uniq < $OAR_NODEFILE | wc -l | tr -d ' '`
echo "nodes = $NODES"
NPROCS=`wc -l < $OAR_NODEFILE | tr -d ' '`
echo "nprocs= $NPROCS"
$HOME/openmpi/bin/mpdboot --rsh=ssh --totalnum=$NODES --file=$OAR_NODEFILE
sleep 1
$HOME/openmpi/bin/mpirun -n $NPROCS $HOME/src/mpi/tp

Setting up and starting mpich-2 on a default environment using oarsh

• oarsub -I -l nodes=3

oarsh is the default connector used when you reserve a node. To be able to use this connector, you need to specify it on the mpdboot command line. So you can use exactly the same script than for the previous allow_classic_ssh part and replacing --rsh=ssh by --rsh=oarsh.

Setting up and starting mpich-2 on a kadeploy image

• oarsub -I -t deploy -l nodes=3

Now, you can try to create a kadeploy environment with mpich2 installed on it an then deploy a set of three machines to launch the mpi code in the previous section.

Setting up and starting OpenMPI to use high performance interconnect

Myrinet hardware : MPICH-MX

• oarsub -I -l nodes=3 -p "myri2g='YES'"

or

• oarsub -I -l nodes=3 -p "myri10g='YES'"

Infiniband hardware : MVAPICH

• oarsub -I -l nodes=3 -p "ib10g='YES'"

or

• oarsub -I -l nodes=3 -p "ib20g='YES'"

Setting up openMPI to accept private networks as routable between hosts

OpenMpi site : OpenMPI

• Connect on one site and reserve a node

ssh nancy.grid5000.fr
oarsub -I

• Connect to another site and reserve a node

ssh rennes.grid5000.fr
oarsub -I

• Try to launch the code that you previously create between the two reserved nodes