|
|
| Line 3: |
Line 3: |
| {{Author|Florent Chuffart}} | | {{Author|Florent Chuffart}} |
| {{Author|Matthieu Imbert}} | | {{Author|Matthieu Imbert}} |
| | {{Author|Laurent Pouilloux}} |
| {{Maintainer|Florent Chuffart}} | | {{Maintainer|Florent Chuffart}} |
| {{Maintainer|Matthieu Imbert}} | | {{Maintainer|Matthieu Imbert}} |
| {{Status|Draft}} | | {{Maintainer|Laurent Pouilloux}} |
|
| |
|
| {{Warning|text=This page is outdated, please refer to the execo tutorial http://execo.gforge.inria.fr/doc/tutorial.html}}
| | Execo offers a Python API to manage processes execution. It is well |
| | suited for quick and easy creation of reproducible experiments on |
| | distributed hosts. It is a huge improvement over using shell script |
| | for experiment development. |
|
| |
|
| = Installing and Configuring execo for g5k =
| | It offers: |
| | * Asynchronous control of local and remote commands. |
| | * Efficient and scalable execution (transparent usage of Taktuk). |
| | * Interaction with Grid'5000 resources (planning, submission, deployment, API). |
| | * Tools for rapid development of reproducible experiments. |
|
| |
|
| Execo offers a Python API for local or remote, standalone or parallel, processes execution. It is especially well suited for scripting workflows of parallel/distributed operations on local or remote hosts: conducting experiments, performing automated tests, etc. The core python package is execo. The execo_g5k package provides a set of tools and extensions for Grid5000. The execo_engine package and execo-run script provide an extendable experiment engine.
| | ==Installation== |
| | Connect on a frontend and use easy_install: |
| | {{Term|location=frontend|cmd=<code class="command">easy_install</code> --user execo}} |
| | Configure your path in .bashrc: |
| | export PATH="$HOME/.local/bin${PATH:+:${PATH}}" |
| | Create a .execo.conf.py in your home directory if you need some specific |
| | [http://execo.gforge.inria.fr/doc/execo_g5k.html#execo-g5k-configuration configuration] |
|
| |
|
| For more information, visit the execo website http://execo.gforge.inria.fr .
| | ==Usage== |
|
| |
|
| This tutorial is a prerequisite to use execo over g5k. It is composed of two main steps : installation and configuration.
| | ===Process management=== |
| Before using the following ''quick and dirty'' script to install execo , try to install it from git repository.
| |
|
| |
|
| <pre>
| | ===Working with G5K=== |
| cd /tmp
| |
|
| |
|
| wget http://public.sophia.grid5000.fr/~fchuffart/execo_dep/httplib2-0.6.0.tar.gz
| | ===Automating experiments=== |
| tar xfvz httplib2-0.6.0.tar.gz
| |
| cd httplib2-0.6.0
| |
| python setup.py install --prefix=$HOME/.local
| |
| cd ..
| |
|
| |
|
| wget http://public.sophia.grid5000.fr/~fchuffart/execo_dep/py-restclient-1.3.3.tar.gz
| | ==Documentation== |
| tar xfvz py-restclient-1.3.3.tar.gz
| | Up-to-date documentation is available on [http://execo.gforge.inria.fr/doc/userguide.html#execo User Guide]. |
| cd py-restclient-1.3.3
| |
| python setup.py install --prefix=$HOME/.local
| |
| cd ..
| |
| | |
| wget http://public.sophia.grid5000.fr/~fchuffart/execo_dep/simplejson-2.6.2.tar.gz
| |
| tar xfvz simplejson-2.6.2.tar.gz
| |
| cd simplejson-2.6.2
| |
| python setup.py install --prefix=$HOME/.local
| |
| cd ..
| |
| | |
| wget http://public.sophia.grid5000.fr/~fchuffart/execo.tgz
| |
| tar xfvz execo.tgz
| |
| cd execo
| |
| make install PREFIX=$HOME/.local
| |
| | |
| echo "NOW YOU HAVE TO CONFIGURE EXECO ENVIRONEMENT (PATH, SSH KEYS...)
| |
| </pre>
| |
| | |
| == Installing execo from git repository ==
| |
| | |
| Execo will be installed into the ''.local'' directory of your g5k home from which you will launch execo scripts.
| |
| | |
| Execo needs some other libraries:
| |
| === httplib2 === | |
| Be sure that you have also installed ''httplib2-0.6.0''. Get source from [[http://code.google.com/p/httplib2/]] and install it from your g5k home (for example frontend.lyon):
| |
| <pre>
| |
| cd httplib2-0.6.0
| |
| python setup.py install --prefix=$HOME/.local
| |
| </pre>
| |
| | |
| === py-restclient ===
| |
| | |
| Be sure that you have also installed ''py-restclient''. Get source from [[http://pypi.python.org/pypi/py-restclient/1.3.3]] and install it from your g5k home (for example frontend.lyon):
| |
| <pre>
| |
| cd py-restclient-1.3.3
| |
| python setup.py install --prefix=$HOME/.local
| |
| </pre>
| |
| | |
| === simplejson ===
| |
| | |
| Be sure that you have also installed ''simplejson'' , Get source from [[http://pypi.python.org/pypi/simplejson/]] and install it from your g5k home (for example frontend.lyon):
| |
| | |
| <pre>
| |
| cd simplejson-2.1.6
| |
| python setup.py install --prefix=$HOME/.local
| |
| </pre>
| |
| | |
| === execo from git repository ===
| |
| | |
| Proceed to execo installation, from your laptop:
| |
| <pre>
| |
| git clone git://scm.gforge.inria.fr/execo/execo.git
| |
| scp -r execo frontend.lyon.grid5000.fr:
| |
| ssh frontend.lyon.grid5000.fr make -C execo install PREFIX=\$HOME/.local
| |
| </pre>
| |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| == Configuring execo environement (path, keys...) ==
| |
| | |
| | |
| | |
| | |
| | |
| Execo script will be launch from ''frontend.lyon'' and will drive some jobs running in Nancy.
| |
| so We need to configure the PYTHONPATH of g5k home of Lyon and be sure that your ssh keys are well configure between Lyon and Nancy g5k enviroement.
| |
| | |
| === PYTHONPATH ===
| |
| | |
| Your ''.bash_profile'' of your Lyon g5k home need to contain the following lines:
| |
| | |
| <pre>
| |
| if [ -d "\$HOME/.local/lib/python2.6/site-packages" ] ; then
| |
| export PYTHONPATH="\$HOME/.local/lib/python2.6/site-packages${PYTHONPATH+:}${PYTHONPATH}"
| |
| fi
| |
| </pre>
| |
| | |
| === ssh keys ===
| |
| | |
| In order to enable ssh interaction in a transparent way between Lyon and Nancy be sure that each site own the couple of keys ''.ssh/id_rsa'' and ''.ssh/id_rsa.pub''. ''.ssh/id_rsa'' is a private key with no passphrase and ''.ssh/id_rsa.pub'' has previously been added into the ''.ssh/authorized_keys'' of each g5k site that you target ([[https://www.grid5000.fr/mediawiki/index.php/SSH|more here]]).
| |
| | |
| === OAR_JOB_KEY_FILE ===
| |
| | |
| Finally, the ''.bash_profile'' of the the site that you target (in our case Nancy) needs to contain the following line
| |
| | |
| <pre>
| |
| export OAR_JOB_KEY_FILE=~/.ssh/id_rsa' >> .bash_profile
| |
| </pre>
| |
| | |
| The OAR_JOB_KEY_FILE will be use by oar to allow you to access to your reservation. Execo will use it to drive your reservations.
| |
Execo offers a Python API to manage processes execution. It is well
suited for quick and easy creation of reproducible experiments on
distributed hosts. It is a huge improvement over using shell script
for experiment development.
It offers:
- Asynchronous control of local and remote commands.
- Efficient and scalable execution (transparent usage of Taktuk).
- Interaction with Grid'5000 resources (planning, submission, deployment, API).
- Tools for rapid development of reproducible experiments.
Installation
Connect on a frontend and use easy_install:
|
frontend:
|
easy_install --user execo
|
Configure your path in .bashrc:
export PATH="$HOME/.local/bin${PATH:+:${PATH}}"
Create a .execo.conf.py in your home directory if you need some specific
configuration
Usage
Process management
Working with G5K
Automating experiments
Documentation
Up-to-date documentation is available on User Guide.
