Execo On G5K

From Grid5000
Revision as of 10:19, 20 August 2013 by Lpouilloux (talk | contribs)
Jump to navigation Jump to search


Warning.png Warning

This page is outdated, please refer to the execo tutorial http://execo.gforge.inria.fr/doc/tutorial.html

Installing and Configuring execo for g5k

Execo offers a Python API for local or remote, standalone or parallel, processes execution. It is especially well suited for scripting workflows of parallel/distributed operations on local or remote hosts: conducting experiments, performing automated tests, etc. The core python package is execo. The execo_g5k package provides a set of tools and extensions for Grid5000. The execo_engine package and execo-run script provide an extendable experiment engine.

For more information, visit the execo website http://execo.gforge.inria.fr .

This tutorial is a prerequisite to use execo over g5k. It is composed of two main steps : installation and configuration. Before using the following quick and dirty script to install execo , try to install it from git repository.

cd /tmp 

wget http://public.sophia.grid5000.fr/~fchuffart/execo_dep/httplib2-0.6.0.tar.gz
tar xfvz httplib2-0.6.0.tar.gz
cd httplib2-0.6.0
python setup.py install --prefix=$HOME/.local
cd ..

wget http://public.sophia.grid5000.fr/~fchuffart/execo_dep/py-restclient-1.3.3.tar.gz
tar xfvz py-restclient-1.3.3.tar.gz
cd py-restclient-1.3.3
python setup.py install --prefix=$HOME/.local
cd ..

wget http://public.sophia.grid5000.fr/~fchuffart/execo_dep/simplejson-2.6.2.tar.gz
tar xfvz simplejson-2.6.2.tar.gz 
cd simplejson-2.6.2
python setup.py install --prefix=$HOME/.local
cd ..

wget http://public.sophia.grid5000.fr/~fchuffart/execo.tgz
tar xfvz execo.tgz
cd execo
make install PREFIX=$HOME/.local

echo "NOW YOU HAVE TO CONFIGURE EXECO ENVIRONEMENT (PATH, SSH KEYS...)

Installing execo from git repository

Execo will be installed into the .local directory of your g5k home from which you will launch execo scripts.

Execo needs some other libraries:

httplib2

Be sure that you have also installed httplib2-0.6.0. Get source from [[1]] and install it from your g5k home (for example frontend.lyon):

cd httplib2-0.6.0
python setup.py install --prefix=$HOME/.local

py-restclient

Be sure that you have also installed py-restclient. Get source from [[2]] and install it from your g5k home (for example frontend.lyon):

cd py-restclient-1.3.3
python setup.py install --prefix=$HOME/.local

simplejson

Be sure that you have also installed simplejson , Get source from [[3]] and install it from your g5k home (for example frontend.lyon):

cd simplejson-2.1.6
python setup.py install --prefix=$HOME/.local

execo from git repository

Proceed to execo installation, from your laptop:

git clone git://scm.gforge.inria.fr/execo/execo.git 
scp -r execo frontend.lyon.grid5000.fr:
ssh frontend.lyon.grid5000.fr make -C execo install PREFIX=\$HOME/.local




Configuring execo environement (path, keys...)

Execo script will be launch from frontend.lyon and will drive some jobs running in Nancy. so We need to configure the PYTHONPATH of g5k home of Lyon and be sure that your ssh keys are well configure between Lyon and Nancy g5k enviroement.

PYTHONPATH

Your .bash_profile of your Lyon g5k home need to contain the following lines:

if [ -d "\$HOME/.local/lib/python2.6/site-packages" ] ; then
    export PYTHONPATH="\$HOME/.local/lib/python2.6/site-packages${PYTHONPATH+:}${PYTHONPATH}"
fi

ssh keys

In order to enable ssh interaction in a transparent way between Lyon and Nancy be sure that each site own the couple of keys .ssh/id_rsa and .ssh/id_rsa.pub. .ssh/id_rsa is a private key with no passphrase and .ssh/id_rsa.pub has previously been added into the .ssh/authorized_keys of each g5k site that you target ([here]).

OAR_JOB_KEY_FILE

Finally, the .bash_profile of the the site that you target (in our case Nancy) needs to contain the following line

export OAR_JOB_KEY_FILE=~/.ssh/id_rsa' >> .bash_profile

The OAR_JOB_KEY_FILE will be use by oar to allow you to access to your reservation. Execo will use it to drive your reservations.