From Grid5000

Contents 1 Overview 2 Basis commands 2.1 User 2.1.1 Node states 2.1.2 Task states 2.1.3 Submit a task 2.1.3.1 Several options are available 2.1.3.2 To make a reservation 2.1.3.3 Using OAR for Kadeploy deployment 2.1.4 Stop a task 2.1.5 Quick Glossary 2.2 Administrator 2.2.1 Create a node 2.2.2 Delete a node 2.2.3 Set deployment feature avalability on a node 2.2.4 Add a property to a node 3 OAR submission frontals 4 OAR Monitoring tools

Overview

This submit interface is very similar to PBS or Torque.

You can also look at the OAR Grid1 page, which is an OAR spin-off to reserve nodes over the whole grid.

Basis commands

Follows a quick overview of the base commands, please have a look at the manual pages of each command for full explanations (man oarcommand).

Another documentation is available at Manuel user/administrator.

User

Node states

To list states and informations of all nodes:

oarnodes

Task states

To list all tasks which are running or waiting:

oarstat

Submit a task

The command for job submission is oarsub. This command returns a task identifier (a jobid number). The shell created for the execution of the task contains environment variables dedicated to the task (OAR_ prefixed): for instance, the following environment variable: OAR_FILE_NODES, stores the name of a file containing the list of the node that are reserved for the job. Keep in mind that this file may contains duplicated nodes names, corresponding to the weight of the job (SMP nodes).

There are two submission modes: interactive which gives you a interactive shell and often used for debugging purpose, or passive for batch mode execution.

 oarsub -I     <- interactive mode
 oarsub script <- passive mode

Using the passive mode, two files are created: OAR.«scriptname».«idjob».stdout and OAR.«script_name».«idjob».stderr that store the standard output and standart error output of the task.

Several options are available

• we want 10 nodes in interactive mode:

 oarsub -I -l nodes=10

• we want 10 nodes and run a script (will be launched on the first node)

 oarsub -l nodes=10 script

• we want the specified node

 oarsub -I -p "hostname='node38-002.imag.fr'"

• we want 8 nodes, ordered (from paravent33 to paravent40) and in interactive mode:

 oarsub -l nodes=8 -p"hostname >= 'paravent33.rennes.grid5000.fr' AND hostname <= 'paravent40.rennes.grid5000.fr'"  -I

• we want 2 particular nodes and a script is launched on the first one:

 oarsub -l nodes=2 -p "hostname='host1' or hostname='host2'" scriptname

• we want 3 noeuds with a Myrinet Lanai9 card (eg: cluster paraci from Rennes)

 oarsub -p"lanai9='YES'" -l nodes=3 -I

To make a reservation

A reservation is a time slot in the future that we want to reserve, to then connect to once it's time.

• reserve 12 nodes for 5 hours starting at 11am on april 27. 2004:

 oarsub -r "2004-04-27 11:00:00" -l  walltime=5,nodes=12 

• connect to reserved nodes (say at 11:05 on april 27, 2004)

 oarsub -c idjob -I 

Using OAR for Kadeploy deployment

Usually, deployments are done within an interactive job (at least during the experiment setup, passive mode being always a good idea to run experiment campaigns). Nodes are provided in the default environment, user environments are to be deployed at the user's premise. To reserve deployable nodes, you must specify the deploy queue when you run oarsub, this will give you the rights to deploy on your nodes.

• we want 10 nodes in interactive mode with the deploy rights

 oarsub -I -q deploy -l nodes=10

Stop a task

To stop the task whose identifier is jobid

oardel jobid

Quick Glossary

Please refer to the following definitions to better understand the different OAR mecanisms:

	submission	reservation
interactive	oarsub -I	oarsub -r ; oarsub -c jobId
passive	oarsub scriptName	oarsub -r scriptName

Administrator

Create a node

There is normally no need to add manually a node to OAR database, as this is done when the node boot (the hostname is used for the registration).

You can add a node using the following command as root or oar (OAR admin )

oarnodesetting -s Alive node-x.mysite.grid5000.fr

Delete a node

To remove the node node-xx.site.grid5000.fr from OAR database, you need to run the 2 following commands:

oarnodesetting -s Dead -h node-xx.site.grid5000.fr 
oaremovenode node-xx.site.grid5000.fr 

Be careful: All information (node state history, logs, aso) about the node will be lost.

Note: yes, oaremovenode lacks a 'R' ;)

Set deployment feature avalability on a node

To setup a node as being kadeployable, run:

oarnodesetting -s Alive -h node-xx.site.grid5000.fr -p "deploy=YES"

Add a property to a node

For instance, we want to specify if a node has a Myrinet interface or not. Thus a user can ask the network connection type he wants using the "-p" option of the oarsub command.

So we have to add a new field in the OAR database with the option name. Using mysql command :

ALTER TABLE nodeProperties ADD myrinet ENUM('YES','NO') DEFAULT 'NO';

Then, you have to specify which nodes are in myrinet with oarnodesetting command:

oarnodesetting -s Alive -h nomDuNoeud -p "myrinet=YES"

Now it is possible to submit a job like:

oarsub -I -p "myrinet = 'yes'"

OAR submission frontals

Frontals for job submissions are named like the following: oar.mysite.grid5000.fr

OAR Monitoring tools

Reservation status, and node states monitoring tools are available from the Status page.