Run Nemo On G5K

From Grid5000
Jump to: navigation, search


Contents

Running Nemo on G5K

Nemo is the state-of-the-art modeling framework for oceanographic research: NEMO (Nucleus for European Modelling of the Ocean) http://www.nemo-ocean.eu/.

In the ANR project SPADES (Servicing Petascale Architectures and DistributEd System, 08-ANR-SEGI-025) we investigate NEMO to demonstrate the feature on SBAM (Spades BAsed Middleware).

The main goal with this tutorial is to:

 - run NEMO
 - over G5K
 - from NEMO svn sources
 - into a squeeze-x64-min environment
 - and produce a nice gif animation from the simulation.

Getting NEMO from SVN

# Register on http://www.nemo-ocean.eu 
svn co --username nemo_user http://forge.ipsl.jussieu.fr/nemo/svn/trunk/NEMOGCM

# Get input data from IDRIS
cd NEMOGCM/CONFIG/ORCA2_LIM/EXP00/
wget http://dodsp.idris.fr/reee512/NEMO/ORCA2_LIM_nemo_v3.4.tar

Booking a node and deploy environment

This tutorial is write for a node on the Reims site.

# Book a node
OAR_JOB_ID=`oarsub -t deploy -l nodes=1,walltime=4 -p "cluster='stremi'" 'sleep 5d' | grep OAR_JOB_ID | cut -d "=" -f2`
echo OAR_JOB_ID=$OAR_JOB_ID

# Get machine name
MYNODE=`oarstat -fj $OAR_JOB_ID | grep assigned_hostnames | cut -d " " -f 7`
echo MYNODE=$MYNODE

## Deploy new squeeze-x64-min on this node
kadeploy3 -e squeeze-x64-min -m $MYNODE -k ~/.ssh/id_rsa.pub

## Copy Nemo sources and ferret for visualization (Get packaged ferret version from my home or visit http://ferret.wrc.noaa.gov/Ferret/)
scp -r  /home/fchuffart/NEMOGCM /home/fchuffart/ferret_install root@$MYNODE:

Reaching the node

## reach the node
ssh root@$MYNODE

Configuring the node

## On this just deployed node, install needed packages
export http_proxy=http://proxy:3128/
apt-get -y update
# apt-get -y upgrade
apt-get install -y bc perl-modules liburi-perl gfortran libopenmpi-dev mpi-default-bin libnetcdf-dev netcdf-bin libcurl4-openssl-dev imagemagick

## Install ferret to convert .nc files (visualization)
mkdir -p /tmp/data_disk/fer_dsets
cd /tmp/data_disk/fer_dsets
tar xfvz ~/ferret_install/fer_dsets.tar.gz 
mkdir -p /usr/local/ferret
cd /usr/local/ferret
tar xfvz /root/ferret_install/fer_environment.tar.gz
tar xzfz /root/ferret_install/fer_executables.tar.gz
cp /root/ferret_install/ferret_path* /usr/local
echo "source /usr/local/ferret_paths" > /root/.bashrc
source /root/.bashrc

Compiling Nemo

NEMO_DIR=/root/NEMOGCM

## Compile nemo with the correct ARCH conf
cd ${NEMO_DIR}/CONFIG/
sed 's/gfortran/mpif90/' ../ARCH/arch-gfortran_linux.fcm | sed 's/\/local\/netcdf//' | sed 's/-lnetcdf/-lnetcdf -lnetcdff/' | sed 's/gmake/make/' > ../ARCH/arch-g5k.fcm
./makenemo -n ORCA2_LIM -m g5k -j24 

Running a basic simulation

NEMO_DIR=/root/NEMOGCM

## Prepare a Simulation with the parameter file namelist (24 mpi proc and 465 timestep)
rm -Rf /tmp/run_nemo 
mkdir /tmp/run_nemo
cd /tmp/run_nemo
cp ${NEMO_DIR}/CONFIG/ORCA2_LIM/BLD/bin/nemo.exe /tmp/run_nemo/my_nemo.bin

JPNI=4
JPNJ=6
NN_ITEND=465

EXP_DIR=${NEMO_DIR}/CONFIG/ORCA2_LIM/EXP00/
# changing #core and #iteration
sed "s/nn_itend    =    5475/nn_itend    =      ${NN_ITEND}/" ${EXP_DIR}/namelist | sed "s/jpni        =   0/jpni        =   ${JPNI}/" | sed "s/nn_write    =    5475/nn_write    =    5/" | sed "s/jpnj        =   0/jpnj        =   ${JPNJ}/" > /tmp/run_nemo/namelist
sed  "s/ln_nc4zip      =   .TRUE./ln_nc4zip      =   .FALSE./" ${EXP_DIR}/xmlio_server.def  > /tmp/run_nemo/xmlio_server.def
cp ${EXP_DIR}/iodef.xml /tmp/run_nemo/.
cp ${EXP_DIR}/namelist_ice_lim2 /tmp/run_nemo/namelist_ice 

rm machinefile
JPNIJ=`echo "\${JPNI}*\${JPNJ}" | bc`
for i in $(seq 1 $JPNIJ)
do
   echo $HOSTNAME >> machinefile
done
tar xfv ${EXP_DIR}/ORCA2_LIM_nemo_v3.4.tar 
for GZIPS in `ls *.gz` 
do
  gzip -d $GZIPS
done

## Run a simulation (~60 min, time step overwrite in /tmp/run_nemo/time.step)
time mpiexec -machinefile machinefile my_nemo.bin

Results vizualisation

# Write ferret script 
OUTNCFILE=`ls ORCA2_1d_00010101_*_grid_T.nc`
cat >  plot_sst.jnl << EOF
use $OUTNCFILE
set var/bad=0 sst_1d_ave[d=1]
repeat/l=1:93 (fill/xlim=-180:180/ylim=-90:90/lev=(0,30,2) SST_1D_AVE,nav_lon,nav_lat; go land ;FRAME/FILE=whirl-\`l\`.gif)
quit
EOF
## Convert .nc to gif
ferret -memsize 10.  -gif -nojnl -script plot_sst.jnl
# rename gifs
for file in whirl*; do file2=`echo $file | sed -e s/'-.[0-9]\./-00&'/ -e s/'-[0-9]\./-000&'/ -e s/'-00-'/-00/ -e s/'-000-'/-000/` ; mv $file $file2; done
# gifs 2 anim
convert -delay 1 -loop 0 whirl* anim.gif
echo "scp root@$HOSTNAME:/tmp/run_nemo/anim.gif /tmp/anim.gif; open -a Safari /tmp/anim.gif"
Personal tools
Namespaces

Variants
Actions
Public Portal
Users Portal
Admin portal
Wiki special pages
Toolbox