Advanced OAR

OAR

This section will show you various details of OAR useful for an advanced usage, as well as some tips and tricks. It assumes you are familiar with OAR and Grid5000 basics. It assumes you are using the bash shell (but should be easy to adapt to another shell).

useful tips

• Take the time to carefuly configure ssh, as described in [www.loria.fr/~lnussbau/files/g5kss10-grid5000-efficiently.pdf Working efficiently on Grid’5000].
• Use screen so that your work is not lost if you loose the connection to Grid5000. Moreover, having a screen session opened with one or more shell sessions allows you to leave your work session when you want then get back to it later and recover it exactly as you leaved it.
• Most OAR commands (oarsub, oarstat, oarnodes) can provide output in various formats:
  • textual (this is the default mode)
  • PERL dumper (-D)
  • xml (-X)
  • yaml (-Y)
  • json (-J)
• Direct access to the OAR database: users can directly access the mysql OAR database oar2 on the server mysql.<site>.grid5000.fr with the read-only account oarreader. The password is read.

connexion to the job's nodes

You will normally use the oarsh wrapper to connect to the nodes instead of ssh, and oarcp instead of scp to copy files to/from the nodes. If you use taktuk (or a similar tools like pdsh), you have to configure it so that it uses oarsh instead of ssh.

oarsh and job keys

By default, OAR generates an ssh key pair for each job, and oarsh is used to connect the job's nodes. oarsh looks to environment variables OAR_JOB_ID or OAR_JOB_KEY_FILE to know the key to use. Thus oarsh works directly if you are connected (either you have submitted an interactive job or you have connected to it later with oarsub -C <JOB_ID>). You can also connect to the nodes without being connected to the job:

$ oarsub -I
[ADMISSION RULE] Set default walltime to 3600.
[ADMISSION RULE] Modify resource description with type constraints
Generate a job key...
OAR_JOB_ID=578440
Interactive mode : waiting...
Starting...
Connect to OAR job 578440 via the node sagittaire-74.lyon.grid5000.fr

then, in another terminal:

$ OAR_JOB_ID=578440 oarsh sagittaire-74
Linux sagittaire-74.lyon.grid5000.fr 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64
This is a Grid'5000 compute node.
You must have a reservation with OAR before using this host.
Last login: Thu Oct 25 23:03:03 2012 from flyon.lyon.grid5000.fr
$

If needed OAR allows to export the job key of a job.

sharing keys between jobs

Telling oar to always use the same key can be very convenient. If you have a passphrase-less ssh key dedicated for navigating inside grid5000, then in your ~/.profile or ~/.bash_profile you can set:

export OAR_JOB_KEY_FILE=<path_to_your_key>

Then, OAR will always use this key for all submitted jobs, which allows you to connect to your nodes with oarsh without beeing connected to the job.

Moreover, if this key is replicated between all Grid5000 sites, and if the environment variable OAR_JOB_KEY_FILE is exported in ~/.profile or ~/.bash_profile on all sites, you will be able to connect directly from any frontend to any reserved node of any site.

If using the same key for all jobs, be warned that this will raise issues if submitting two or more jobs that share a same subset of nodes on different cpusets, because in this case processes cannot be guarantied to run on the good cpuset.

allow_classic_ssh

Submitting with option -t allow_classic_ssh allows you to use ssh directly (instead of oarsh) to connect to the nodes, at the cost of not being able to select resources at a finer level than the node (cpu, core).

oarsh details

oarsh is a frontend to ssh. It opens an ssh connexion as user oar to the dedicated oar ssh server running on the node, listening on port 6667. It detects who you are based on your key, and if you have the right to use the node (if you have reserved it) it will su to your user on the node.

So, if you don't have oarsh installed, you can still connect to the nodes by simulating it. One use case is if you have reserved nodes and want to connect to them through an ssh proxy as described in SSH#Using_SSH_with_ssh_proxycommand_setup_to_access_hosts_inside_Grid.275000:

Copy your external public key in Grid5000:

user-laptop$ scp ~/.ssh/<keyname> ~/.ssh/<keyname>.pub g5k:.ssh/

In Grid5000, submit a job using this key (it will ask your passphrase to access the key):

$ oarsub -i ~/.ssh/<keyname> -I

Wait for the job to start. Then in another terminal, from outside Grid, try connecting to the node:

user-laptop$ ssh -p 6667 oar@<node name>.g5k

passive and interactive modes

In interactive mode: a shell is opened on the first node of the reservation (or on the frontend, with appropriate environment set, if the job is of type deploy). In interactive mode, the job will be killed as soon as this job's shell is closed and will be limited by the job's walltime. It can also be killed by an explicit oardel.

You can experiment with 3 shells. On first shell, to see the list of your running jobs, regularly run:

$ oarstat -u $USER

On the second shell, run an interactive job:

$ oarsub -I

Wait for the job to start, run oarstat, then leave the job, run oarstat again. Submit another interactive job, and on the third shell, kill it:

$ oardel <JOB_ID>

In passive mode: an executable is run by oar on the first node of the reservation (or on the frontend, with appropriate environment set, if the job is of type deploy). In passive mode, the limitation to the job's length is its walltime. It can also be killed by an explicit oardel.

JOBID = $(oarsub 'uname -a' | sed -n 's/OAR_JOB_ID=\(.*\)/\1/p')
cat OAR.$JOBID.stdout

You may not want a job to be interactive nor to run a script when the job starts, for example because you will use the reserved resources from a program whose lifecycle is longer than the job (and which will use the resources by connecting to the job). One trick to achieve this is to run the job in passive mode with a long sleep command. One drawback of this method is that the job may terminate with status error if the sleep is killed. This can be a problem in some situations, eg. when using job dependencies.

Submission and Reservation

• If you don't specify the job's start date (oar option -r), then your job is a submission and oar will choose the best schedule.
• If you specify the job's start date, this is a reservation, oar cannot decide the best schedule anymore, it is fixed

There are some consequences:

• Current Grid5000 user charter allows no more than 2 reservations per site
• in submission mode you're almost guaranteed to get your wanted resources, because oar can decide what resources to allocate at the last moment. You cannot get the list of resources until the job starts.
• in reservation mode, you're not guaranteed to get your wanted resources, because oar has to plan the allocation of resources at reservation time. If later resources become not available, you loose them for your job. You can get the list of resources as soon as the reservation is done.
• in submission mode, you cannot know the date at will your job will start until it starts. But OAR can give you an estimation of that date.
• to coordinate oar submissions on several sites, OARGRID must do OAR reservations.

example reservation:

$ oarsub -r "2012-1-10 18:00:00"

Complex resources selection

The complete selector format (oarsub option -l):

"{sql₁}/type₁=n₁/type₂=n₂+{sql₂}/type₃=n₃/type₄=n₄/type₅=n₅+...,walltime=hh:mm:ss"

where

• sql₁..sql_n are optional SQL predicates on property columns
• type₁=n₁..type_n=n_n are the wanted number of given resource types
• slashes between resource types express resource subtree selection
• + allows agregating different resource specifications
• walltime is the job walltime (defaults to 1 hour)

You can get the list of column names for SQL predicates by running oarnodes to get the available properties of a particular node, for example in Lyon:

$ oarnodes -Y --sql="network_address = 'sagittaire-1.lyon.grid5000.fr'"

You can also get this list by looking directly at the columns of table 'resources' in the OAR database. For example:

$ mysql -u oarreader -D oar2 -h mysql -p -e "describe resources"

These OAR properties are described in [OAR2 properties]

Selecting nodes from a specific cluster

For example in Nancy:

$ oarsub -I -l {'cluster="graphene"'}/nodes=2

Selecting specific nodes

For example in Lyon:

$ oarsub -I -l {'network_address in ("sagittaire-10.lyon.grid5000.fr", "sagittaire-11.lyon.grid5000.fr", "sagittaire-12.lyon.grid5000.fr")'}/nodes=1

By negating the SQL clause, you can also exclude some nodes.

Two nodes with virtualization capability, on different clusters + IP subnets

We want 2 nodes and 4 /22 subnets with the following constraints:

• Nodes are on 2 different clusters of the same site (Hint: use a site with several clusters :-D)
• Nodes have virtualization capability enabled
• /22 subnets are on two different /19 subnets
• 2 subnets belonging to the same /19 subnet are consecutive

$ oarsub -I -l /slash_19=2/slash_21=1+'{"virtual"!="none"}'/cluster=2/nodes=1

Lets verify the reservation:

 $ uniq $OAR_NODE_FILE
 paradent-6.rennes.grid5000.fr
 parapluie-7.rennes.grid5000.fr

 $ g5k-subnets -p
 10.158.8.0/22
 10.158.32.0/22
 10.158.36.0/22
 10.158.12.0/22

 $ g5k-subnets -ps
 10.158.8.0/21
 10.158.32.0/21