Execo On G5K: Difference between revisions

From Grid5000
Jump to navigation Jump to search
(New page: = Installing and Configuring execo for g5k = == Installing execo from git repository == Execo will be installed into the ''.local'' directory of your g5k home from which you will launch...)
 
No edit summary
 
(48 intermediate revisions by 6 users not shown)
Line 1: Line 1:
= Installing and Configuring execo for g5k =
{{Portal|User}}
{{Portal|Tutorial}}
{{Author|Florent Chuffart}}
{{Author|Matthieu Imbert}}
{{Author|Laurent Pouilloux}}
{{Maintainer|Florent Chuffart}}
{{Maintainer|Matthieu Imbert}}
{{Maintainer|Laurent Pouilloux}}
{|
|
__TOC__
|valign=top padding-left:10px|
Execo offers a Python API to manage processes execution. It is well
suited for quick and easy creation of reproducible experiments on
distributed hosts. It is a huge improvement over using shell script
for experiment development.


You can have a look at the following presentation:


== Installing execo from git repository ==
[[File:Execo_G5K.pdf|presentation]].


Execo will be installed into the ''.local'' directory of your g5k home from which you will launch execo scripts.
It offers:
* Asynchronous control of local and remote commands.
* Efficient and scalable execution (transparent usage of [[Using TakTuk|TakTuk]]).
* Interaction with Grid'5000 resources (planning, submission, deployment, API).
* Tools for rapid development of reproducible experiments.
|
[[File:Execo.png|100px]]
|}


Execo needs some other libraries:
=== httplib2 ===
Be sure that you have also installed ''httplib2-0.6.0''. Get source from [[http://code.google.com/p/httplib2/]] and install it from your g5k home (for example frontend.lyon):
<code>
cd httplib2-0.6.0
python setup.py install --prefix=$HOME/.local
</code>


=== py-restclient ===
==Installation==
Connect on a frontend and use easy_install:
{{Term|location=frontend|cmd=<code class="command">easy_install</code> --user execo}}


Be sure that you have also installed ''py-restclient''. Get source from [[http://pypi.python.org/pypi/py-restclient/1.3.3]] and install it from your g5k home (for example frontend.lyon):
Optionaly create a file name ''.execo.conf.py'' in your home directory if you need some specific
<code>
[https://mimbert.gitlabpages.inria.fr/execo/execo_g5k.html#execo-g5k-configuration configuration]
cd py-restclient-1.3.3
python setup.py install --prefix=$HOME/.local
</code>


=== simplejson ===
==Usage==
===Process management===
To learn how to manage local and ssh process, have a look at [https://mimbert.gitlabpages.inria.fr/execo/userguide.html#processes Process examples]


Be sure that you have also installed ''simplejson'' , Get source from [[http://pypi.python.org/pypi/simplejson/]] and install it from your g5k home (for example frontend.lyon):
===Working with G5K===
cd simplejson-2.1.6
To know how to interact with Grid'5000 infrastrucure (jobs, deployment, planning, API), see the [https://mimbert.gitlabpages.inria.fr/execo/userguide.html#execo-g5k execo g5k section] of the user guide.
python setup.py install --prefix=$HOME/.local
</code>


=== execo from git repository ===
===Automating experiments===
Execo offers an engine that allows to automatize experiments. An example measuring TCP congestion on Grid'5000 can be found [https://mimbert.gitlabpages.inria.fr/execo/userguide.html#using-execo-engine-for-experiment-development here]


Proceed to execo installation, from your laptop:
==Documentation==
<code>
A complete user guide is available [https://mimbert.gitlabpages.inria.fr/execo/userguide.html here].
git clone git://scm.gforge.inria.fr/execo/execo.git
You may also want to have a look at [https://mimbert.gitlabpages.inria.fr/execo/apidoc.html API reference].
scp -r execo frontend.lyon.grid5000.fr:
A grid5000 tutorial is available: [[Execo Practical Session]]
ssh frontend.lyon.grid5000.fr make -C execo install PREFIX=\$HOME/.local
</code>
 
 
 
 
 
 
 
== Configuring execo environement (path, keys...) ==
 
 
 
 
 
Execo script will be launch from ''frontend.lyon'' and will drive some jobs running in Nancy.
so We need to configure the PYTHONPATH of g5k home of Lyon and be sure that your ssh keys are well configure between Lyon and Nancy g5k enviroement.
 
=== PYTHONPATH ===
 
Your ''.bash_profile'' of your Lyon g5k home need to contain the following lines:
 
<code>
if [ -d "\$HOME/.local/lib/python2.5/site-packages" ] ; then
    export PYTHONPATH="\$HOME/.local/lib/python2.5/site-packages${PYTHONPATH+:}${PYTHONPATH}"
fi
</code>
 
=== ssh keys ===
 
In order to enable ssh interaction in a transparent way between Lyon and Nancy be sure that each site own the couple of keys ''.ssh/id_rsa'' and ''.ssh/id_rsa.pub''. ''.ssh/id_rsa'' is a private key with no passphrase and ''.ssh/id_rsa.pub'' has previously been added into the ''.ssh/authorized_keys'' of each g5k site that you target  ([[https://www.grid5000.fr/mediawiki/index.php/SSH|more here]]).  
 
=== OAR_JOB_KEY_FILE ===
 
Finally, the ''.bash_profile'' of the the site that you target (in our case Nancy) needs to contain the following line
 
<code>
export OAR_JOB_KEY_FILE=~/.ssh/id_rsa' >> .bash_profile
</code>
 
The OAR_JOB_KEY_FILE will be use by oar to allow you to access to your reservation. Execo will use it to drive your reservations.

Latest revision as of 11:18, 28 September 2021

Execo offers a Python API to manage processes execution. It is well suited for quick and easy creation of reproducible experiments on distributed hosts. It is a huge improvement over using shell script for experiment development.

You can have a look at the following presentation:

presentation.

It offers:

  • Asynchronous control of local and remote commands.
  • Efficient and scalable execution (transparent usage of TakTuk).
  • Interaction with Grid'5000 resources (planning, submission, deployment, API).
  • Tools for rapid development of reproducible experiments.

Execo.png


Installation

Connect on a frontend and use easy_install:

Terminal.png frontend:
easy_install --user execo

Optionaly create a file name .execo.conf.py in your home directory if you need some specific configuration

Usage

Process management

To learn how to manage local and ssh process, have a look at Process examples

Working with G5K

To know how to interact with Grid'5000 infrastrucure (jobs, deployment, planning, API), see the execo g5k section of the user guide.

Automating experiments

Execo offers an engine that allows to automatize experiments. An example measuring TCP congestion on Grid'5000 can be found here

Documentation

A complete user guide is available here. You may also want to have a look at API reference. A grid5000 tutorial is available: Execo Practical Session