Advanced OAR: Difference between revisions
(→Funk) |
|||
(39 intermediate revisions by 10 users not shown) | |||
Line 20: | Line 20: | ||
** Yaml (-Y) | ** Yaml (-Y) | ||
** json (-J) | ** json (-J) | ||
* Regarding the <code class="command">oarsub</code> command line, you should mostly only see the "host" word, but the <code class="command">oarsub</code> command can use both the word "host" or "nodes" indifferently in Grid'5000, as nodes is just an alias for host. Prefer using "host". Besides, the word "host" is also to be preferred to the longer "network_address" word in the resources filters (both properties have sometime the same value, but not always). | * Regarding the <code class="command">oarsub</code> command line, you should mostly only see the "host" word, but the <code class="command">oarsub</code> command can use both the word "host" or "nodes" indifferently in Grid'5000, as nodes is just an alias for host. Prefer using "host". Besides, the word "host" is also to be preferred to the longer "network_address" word in the resources filters (both properties have sometime the same value, but not always). | ||
* At job submission time, only important information are printed out by <code class="command">oarsub</code>. To have more indication about what is done by OAR on Grid'5000 (like computed resource filter, exceptional granted privileges, …) the <code class="command">oarsub</code> verbose (<code>-v</code>) option can be used. | |||
* A syntax simplification mechanism was deployed on Grid'5000 to ease job submission, described at [[OAR Syntax simplification]]. | |||
== Connection to the job's nodes == | == Connection to the job's nodes == | ||
Two commands can be used to connect to nodes on Grid'5000, <code class="command">oarsh</code> and <code class="command">ssh</code>. | |||
=== | === Using ssh === | ||
<code class="command">ssh</code> can only be used when a node is entirely reserved in your job (all CPU cores). Other cases may not allow assigning processes to the correct job, thus connecting with <code class="command">ssh</code> is not allowed. | |||
; | For instance, when a node is entirely reserved as follows: | ||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -I}} | |||
# Set walltime to default (3600 s). | |||
OAR_JOB_ID=<JOB_ID> | |||
# Interactive mode: waiting... | |||
# Starting... | |||
user@node-32:~$ | |||
If you open a new shell and try to connect to the node with <code class="command">ssh</code>, it should work: | |||
{{Term|location=fontend.site|cmd=<code class=command>ssh</code> node-32 }} | |||
Linux node-32.site.grid5000.fr 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 | |||
Debian11-x64-std-2022013022 | |||
(Image based on Debian Bullseye for AMD64/EM64T) | |||
Maintained by support-staff <support-staff@lists.grid5000.fr> | |||
Last login: Wed Feb 23 15:20:32 2022 from 172.16.31.101 | |||
user@node-32:~$ | |||
However, when reserving for instance only one CPU core of a node: | |||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -I -l core=1}} | |||
# Set walltime to default (3600 s). | |||
OAR_JOB_ID=<JOB_ID> | |||
# Interactive mode: waiting... | |||
# Starting... | |||
user@node-32:~$ | |||
When trying to connect to the node with <code class="command">ssh</code> in another shell, you get: | |||
{{Term|location=fontend.site|cmd=<code class=command>ssh</code> node-32 }} | |||
To connect using 'ssh' directly, you must have a single job using all available cores on the node. | |||
Use 'oarsh' instead. | |||
Connection closed by node-32 port 22 | |||
=== Using oarsh === | |||
<code class=command>oarsh</code> is a frontend to <code class=command>ssh</code> (the <code class=command>oarsh</code> command wraps the OpenSSH <code class=command>ssh</code> command to add some required functions to connect to a job, but provides mostly the same interface/options). | |||
{{Note|text=Technical note about <code class=command>oarsh</code> internals: | |||
* It opens an ssh connection transiently as the <code class="command">oar</code> user to the OAR dedicated SSH server running on a node (TCP port 6667) | |||
* It detects who you are based on the job id ou a job key: if you indeed have the right to connect to the node (you reserved it in an OAR job), it switches back to your user for the execution of the shell or command on the node in the job's context (cgroup/cpuset). | |||
}} | |||
In case of nodes are not entirely reserved (all CPU cores), you have to use the <code class="command">oarsh</code> command to connect to nodes instead of <code class="command">ssh</code>, and <code class="command">oarcp</code> instead of <code class="command">scp</code> to copy files to/from the nodes. If you use <code class="command">taktuk</code> for parallel executions (or a similar tools like <code class="command">pdsh</code>) or <code class="command">rsync</code> to synchronize files to/from a node, you have to configure the ''connector'' so the command uses <code class="command">oarsh</code> instead of <code class="command">ssh</code> underneath (see the man pages of the command to find out how to change the connector, e.g. using ''-c'' or ''-e''). | |||
Please note that <code class=command>oarsh</code> also works for node entirely reserved in a job. | |||
==== Splitting job resources ==== | |||
<code class=command>oarsh</code> also allows splitting resources of a job, for instance to execute commands on different subsets of resources in a job (e.g. 1 GPU each instead of all the reserved GPUs). | |||
See an example of using this functionality with [[GNU_Parallel#Confining_GNU_Parallel_tasks_to_GPU.2C_CPU.2C_or_cores|GNU Parallel]]. | |||
==== About OAR job keys ==== | |||
By default, OAR generates a job key pair for each job. <code class="command">oarsh</code> can use either the <code class="command">OAR_JOB_ID</code> or <code class="command">OAR_JOB_KEY_FILE</code> environment variables to know what job to connect. If outside a job shell (e.g. on the frontend), you have to set one of those environment variable. This is not required if <code class="command">oarsh</code> is called from the shell of a job (e.g. on a node), since variables are already set. | |||
; Example using <code class="command">OAR_JOB_ID</code> | |||
For instance, create a job requesting 3 hosts (3 nodes): | For instance, create a job requesting 3 hosts (3 nodes): | ||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -I -l host=3}} | {{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -I -l host=3}} | ||
Line 45: | Line 102: | ||
{{Term|location=fontend.site|cmd=OAR_JOB_ID=<code class=replace>JOBID</code> <code class=command>oarsh</code> <code class=replace>node-2</code>}} | {{Term|location=fontend.site|cmd=OAR_JOB_ID=<code class=replace>JOBID</code> <code class=command>oarsh</code> <code class=replace>node-2</code>}} | ||
; | ; Example using <code class="command">OAR_JOB_KEY_FILE</code> | ||
OAR can expose the job key, using the <code class= | OAR can expose the job key, using the <code class=command>-e</code> option of <code class=command>oarsub</code> | ||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -I -l host=3 -e <code class=replace>my_job_key</code>}} | {{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -I -l host=3 -e <code class=replace>my_job_key</code>}} | ||
Line 54: | Line 111: | ||
{{Term|location=fontend.site|cmd=OAR_JOB_KEY_FILE=<code class=replace>my_job_key</code> <code class=command>oarsh</code> <code class=replace>node-2</code>}} | {{Term|location=fontend.site|cmd=OAR_JOB_KEY_FILE=<code class=replace>my_job_key</code> <code class=command>oarsh</code> <code class=replace>node-2</code>}} | ||
{{Note|text=Note that the following command also allows getting a shell in a job, but only on the first default resource (i.e. node). | |||
Note that the following command also allows getting a shell in a job, but only on the first default resource (i.e. node). | |||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -C <code class=replace>JOBID</code>}} | {{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -C <code class=replace>JOBID</code>}} | ||
}} | |||
=== | ==== Connecting to a job of a different site ==== | ||
Job keys are especially useful when having to connect from nodes of different sites, since each site is managed by a different OAR instance. | |||
Thus, a convenient way is to tell OAR to always use the same job key for all jobs. You can for instance use your Grid'5000 internal SSH key (This key is generated when your account is created) as the job key: In your <code class="command">~/.profile</code> or <code class="command">~/.bash_profile</code>, set: | |||
export OAR_JOB_KEY_FILE=<code class=replace>path_to_your_private_key</code> | export OAR_JOB_KEY_FILE=<code class=replace>path_to_your_private_key</code> | ||
Then, OAR will always use that key for all jobs, allowing you to connect to your nodes with <code class=command>oarsh</code> seamlessly from outside | Then, OAR will always use that key for all jobs, allowing you to connect to your nodes with <code class=command>oarsh</code> seamlessly from sites to sites, jobs to jobs, or even outside jobs. | ||
{{Warning|text=When using the same job key for 2 jobs that share some nodes (each job reserving part of the nodes), <code class=command>oarsh</code> may not execute in the expected job context (i.e. cgroup/cpuset) as the job key does not differentiate jobs. You may look at the <code class="command">OAR_JOB_ID</code> to notice that.}} | |||
==== oarsh vs ssh: tips and tricks ==== | |||
{{Note|text=The following is only interesting if you jobs do not reserve nodes entirely, as using <code class=command>oarsh</code> is useless otherwise}} | |||
=== oarsh vs ssh: tips and tricks === | |||
; 1st tip - hide <code class=command>oarsh</code>, ''rename'' it <code class=command>ssh</code> | ; 1st tip - hide <code class=command>oarsh</code>, ''rename'' it <code class=command>ssh</code> | ||
Line 108: | Line 159: | ||
You can just ssh to a reserved node directly from your workstation as follows: | You can just ssh to a reserved node directly from your workstation as follows: | ||
{{Term|location=workstation|cmd=ssh <code class=replace>reserved-node</code>.<code class=replace>site</code>.g5koar}} | {{Term|location=workstation|cmd=ssh <code class=replace>reserved-node</code>.<code class=replace>site</code>.g5koar}} | ||
== Passive and interactive job modes == | == Passive and interactive job modes == | ||
Line 137: | Line 182: | ||
In passive mode, the <code class=replace>command</code> that is given to <code class=command>oarsub</code> is executed on the first default resource (i.e. node) of the job (or on the site's frontend if the job is of type <code class=command>deploy</code> or <code class=command>cosystem</code>). The job's duration will be the shorter of the execution time of the <code class=replace>command</code> and the job's given <code class=replace>walltime</code>. That unless the job is terminated beforehand by an explicit <code class=command>oardel</code> call from the user or administrator. | In passive mode, the <code class=replace>command</code> that is given to <code class=command>oarsub</code> is executed on the first default resource (i.e. node) of the job (or on the site's frontend if the job is of type <code class=command>deploy</code> or <code class=command>cosystem</code>). The job's duration will be the shorter of the execution time of the <code class=replace>command</code> and the job's given <code class=replace>walltime</code>. That unless the job is terminated beforehand by an explicit <code class=command>oardel</code> call from the user or administrator. | ||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -l host=3,walltime=<code class=replace>walltime</code> "<code class=replace> | {{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -l host=3,walltime=<code class=replace>walltime</code> "<code class=replace>command</code>"}} | ||
'''command''' can be a simple script name or a more complex command line with arguments. | |||
To pass arguments, you have to quote the whole command line, like in the following example: | |||
<code class="command">oarsub</code> -l nodes=4,walltime=2 <code class="replace">"/path/to/myscript arg1 arg2 arg3"</code> | |||
'''Note:''' to avoid random code injection, <code class="command">oarsub</code> allows only alphanumeric characters (<code>[a-zA-Z0-9_]</code>), whitespace characters (<code>[ \t\n\r\f\v]</code>) and few others (<code>[/.-]</code>) inside its command line argument. | |||
Special case for jobs of type <code class=command>noop</code> which are always passive jobs: no command is executed for them. The duration of the job is the given <code class=replace>walltime</code>. | Special case for jobs of type <code class=command>noop</code> which are always passive jobs: no command is executed for them. The duration of the job is the given <code class=replace>walltime</code>. | ||
{{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -t <code class=command>noop</code> -l host=3,walltime=<code class=replace>walltime</code>}} | {{Term|location=fontend.site|cmd=<code class=command>oarsub</code> -t <code class=command>noop</code> -l host=3,walltime=<code class=replace>walltime</code>}} | ||
<code class=replace>oardel</code> can also be used to terminate a passive mode reservation. Note that it is only possible to remove the complete reservation, and not individual nodes. | |||
{{Term|location=fontend.site|cmd=<code class=command>oardel</code> <code class=replace>JOBID</code>}} | |||
=== Interactive mode without shell === | === Interactive mode without shell === | ||
Line 166: | Line 222: | ||
* If after 400 seconds, if not all resources are available, the job will start with fewer resources than initially allocated. This is however quite unusual. | * If after 400 seconds, if not all resources are available, the job will start with fewer resources than initially allocated. This is however quite unusual. | ||
* The list of allocated resources to an advance reservation job is fixed and known as soon as the advance reservation is validated. But you will get the actual list of resources (that is with unavailable resources removed for it) when the advance reservation starts. | * The list of allocated resources to an advance reservation job is fixed and known as soon as the advance reservation is validated. But you will get the actual list of resources (that is with unavailable resources removed for it) when the advance reservation starts. | ||
* To coordinate the start date of OAR jobs on several sites | * To coordinate the start date of OAR jobs on several sites you can use funk which has advanced reservation options. | ||
Example: a reservation for a job in one week from now | Example: a reservation for a job in one week from now | ||
Line 173: | Line 229: | ||
For advance reservations, there is no interactive mode. You can give OAR a command to execute or nothing. If you do not give a command, you'll have to connect to the jobs once the reservation starts (using oarsub -C <jobid> or oarsh). | For advance reservations, there is no interactive mode. You can give OAR a command to execute or nothing. If you do not give a command, you'll have to connect to the jobs once the reservation starts (using oarsub -C <jobid> or oarsh). | ||
=== Why did my advance reservation start with less than all the resources I requested ? === | |||
Since resources states are transitional, the advance reservation process considers indifferently the current state of resources, be it ''alive'', ''suspended'' or ''absent''. Indeed, at the requested start time of an advance reservation, all resources in any of those states should presumably be back in the ''alive'' state. | |||
This is different for resources in the ''dead'' state, which mark failed resources. Although ''dead'' resources may be repaired at some point, that state is less transitional, so ''dead'' resources are excluded from eligible resources for advance reservations. | |||
Also, the allocation of resources to an advance reservation is fixed at the time of validation of the submission (contrarily to batch jobs for which both the start time and allocated resources can change up until the job is effectively started, in order to fit with all requested resources available). As a consequence, resources allocated to an advance reservation which would end up unavailable at the job start time are not replaced by other ''alive'' resources. | |||
In fact, at the start time of an advance reservation, OAR looks after any unavailable resources (''absent'' or ''suspected''), and whenever some exists, wait for them to return to the ''alive'' state for 5 minutes. Then, if they are not back in time, the job starts with less resources than requested and initially allocated (assuming at least one resource is available). | |||
;NB | |||
Information about reduced number of resources or reduced walltime for a reservation due to this mechanism are available in the event part of the output of | |||
<code class='command'>oarstat -fj </code><code class='replace'>jobid</code> | |||
== Getting information about a job == | == Getting information about a job == | ||
Line 289: | Line 358: | ||
Ask for 10 cores of the cluster graphene | Ask for 10 cores of the cluster graphene | ||
$ oarsub -I -l core=10 -p "cluster='graphene'" | $ oarsub -I -l core=10 -p "cluster='graphene'" | ||
Ask for 2 nodes with 16384 | Ask for 2 nodes with 16384 MB of memory and Infiniband 20G | ||
$ oarsub -I -p "memnode='16384' and ib_rate='20'" -l host=2 | $ oarsub -I -p "memnode='16384' and ib_rate='20'" -l host=2 | ||
Ask for any 4 nodes except graphene-12 | Ask for any 4 nodes except graphene-12 | ||
$ oarsub -I -p "not host like 'graphene-12.%'" -l host=4 | $ oarsub -I -p "not host like 'graphene-12.%'" -l host=4 | ||
Ask for a node with 32 threads | |||
$ oarsub -I -p 'thread_count=32' | |||
; Examples of joint resources requests | ; Examples of joint resources requests | ||
Line 330: | Line 401: | ||
Walltime must always be the last argument of <code class="command">-l <...></code> | Walltime must always be the last argument of <code class="command">-l <...></code> | ||
{{Note|text=If no resource matches your request, oarsub will exit with the message | {{Note|text=If no resource matches your request, oarsub will exit with the message | ||
# Set default walltime to 3600. | |||
There are not enough resources for your request | |||
OAR_JOB_ID=-5 | OAR_JOB_ID=-5 | ||
# Error: oarsub failed, please verify your request syntax. | |||
}} | }} | ||
Line 436: | Line 504: | ||
; Using X11 forwarding in the <code class=command>oarsub</code> job shell | ; Using X11 forwarding in the <code class=command>oarsub</code> job shell | ||
If the <code class=command>DISPLAY</code> environment variable is set in the calling shell, <code class=command>oarsub</code> will automatically enable the X11 forwarding. | If the <code class=command>DISPLAY</code> environment variable is set in the calling shell, <code class=command>oarsub</code> will automatically enable the X11 forwarding. Verbose oarsub option (<code>-v</code>) is required to have the "Initialize X11 forwarding..." sentence. | ||
{{Term|location=frontend.site|cmd=<code class=command>oarsub</code> -I -l core=1}} | {{Term|location=frontend.site|cmd=<code class=command>oarsub</code> -v -I -l core=1}} | ||
# Set default walltime to 3600. | |||
# Computed global resource filter: -p "maintenance = 'NO'" | |||
OAR_JOB_ID=4926 | # Computed resource request: -l {"type = 'default'"}/core=1 | ||
Interactive mode : waiting... | # Generate a job key... | ||
OAR_JOB_ID=4926 | |||
# Interactive mode: waiting... | |||
# Starting... | |||
Connect to OAR job 4926 via | # Initialize X11 forwarding... | ||
# Connect to OAR job 4926 via node idpot-8.grenoble.grid5000.fr | |||
Then from the shell of the job, check again the display: | Then from the shell of the job, check again the display: | ||
Line 463: | Line 531: | ||
You can also just run the <code class=command>xterm</code> command directly in the <code class=command>oarsh</code> call: | You can also just run the <code class=command>xterm</code> command directly in the <code class=command>oarsh</code> call: | ||
{{Term|location=frontend.site|cmd=OAR_JOB_ID=4928 <code class=command>oarsh</code> -X idpot-8 <code class=command>xterm</code>}} | {{Term|location=frontend.site|cmd=OAR_JOB_ID=4928 <code class=command>oarsh</code> -X idpot-8 <code class=command>xterm</code>}} | ||
; Using X11 forwarding in a job with a deployed environment | |||
When an interactive job is used to deploy an environment, the spawned shell will not contain the <code class=comand>DISPLAY</code> environment variable, even if it was forwarded in the user connection shell. | |||
To use X11 forwarding in this situation, you can open a new (X11 forwarded) shell on the frontend, and then connect to the node using again X11 forwarding. | |||
you can also connect directly to the node from your laptop either by: | |||
* using the Grid'5000 [[VPN]] | |||
* following the recommendations about a better usage of ssh listed in [[Getting_Started#Recommended_tips_and_tricks_for_an_efficient_use_of_Grid.275000|Getting Started]] document. | |||
{{Note|text=X11 forwarding will suffer from the latency between your local network and the Grid'5000 network. | {{Note|text=X11 forwarding will suffer from the latency between your local network and the Grid'5000 network. | ||
Line 469: | Line 546: | ||
== Using best effort mode jobs == | == Using best effort mode jobs == | ||
The best-effort jobs of OAR are implemented to back-fill the cluster with jobs considered as less important without blocking "regular" jobs. | |||
To submit jobs under that policy, you simply have to select the besteffort type of job in your oarsub command. | |||
<code class="command">oarsub</code> <code class="replace">-t besteffort</code> script_to_launch | |||
Jobs submitted that way will only get scheduled on resources when no other job use them (any regular job | |||
overtake besteffort jobs in the waiting queue, regardless of submission times). | |||
Moreover, these jobs are killed (as if oardel were called) when a regular job recently submitted needs the nodes used by a besteffort job. | |||
By default, no checkpointing or automatic restart of besteffort jobs is provided. They are just killed. That is why this mode | |||
is best used with a tool which can detect the killed jobs and resubmit them. However OAR2 provides options for that. | |||
=== Best effort job campaign === | === Best effort job campaign === | ||
One can submit such jobs using the '''besteffort''' of job type (or indifferently in the '''besteffort''' queue). | |||
For instance you can run a job campaign as follows: | For instance you can run a job campaign as follows: | ||
Line 484: | Line 574: | ||
; Running a besteffort job in a first shell: | ; Running a besteffort job in a first shell: | ||
frennes:~$ '''oarsub -I -l host=10 -t besteffort''' | frennes:~$ '''oarsub -I -l host=10 -t besteffort''' | ||
# Set default walltime to 3600. | |||
OAR_JOB_ID=988535 | OAR_JOB_ID=988535 | ||
Interactive mode : waiting... | # Interactive mode: waiting... | ||
Starting... | # Starting... | ||
parasilo-26:~$ '''uniq $OAR_FILE_NODES''' | parasilo-26:~$ '''uniq $OAR_FILE_NODES''' | ||
parasilo-26.rennes.grid5000.fr | parasilo-26.rennes.grid5000.fr | ||
Line 510: | Line 594: | ||
frennes:~$ '''oarsub -I -l {"host in ('parasilo-9.rennes.grid5000.fr')"}/host=1''' | frennes:~$ '''oarsub -I -l {"host in ('parasilo-9.rennes.grid5000.fr')"}/host=1''' | ||
# Set default walltime to 3600. | |||
OAR_JOB_ID=988546 | OAR_JOB_ID=988546 | ||
Interactive mode : waiting... | # Interactive mode: waiting... | ||
[ | # [2022-01-10 16:00:07] Start prediction: 2022-01-10 16:00:07 (FIFO scheduling OK) | ||
Starting... | # Starting... | ||
Connect to OAR job 988546 via the node parasilo-9.rennes.grid5000.fr | Connect to OAR job 988546 via the node parasilo-9.rennes.grid5000.fr | ||
Line 524: | Line 606: | ||
parasilo-26:~$ Connection to parasilo-26.rennes.grid5000.fr closed by remote host. | parasilo-26:~$ Connection to parasilo-26.rennes.grid5000.fr closed by remote host. | ||
Connection to parasilo-26.rennes.grid5000.fr closed. | Connection to parasilo-26.rennes.grid5000.fr closed. | ||
# Error: job was terminated. | |||
Disconnected from OAR job 988545 | Disconnected from OAR job 988545 | ||
Line 545: | Line 627: | ||
; Running the job | ; Running the job | ||
We run the job on 1 core, and a walltime of 5 minutes, and ask the job to be checkpointed if it lasts (and it will indeed) more than walltime - 150 sec = 2 min 30. | We run the job on 1 core, and a walltime of 5 minutes, and ask the job to be checkpointed if it lasts (and it will indeed) more than walltime - 150 sec = 2 min 30. | ||
$ '''oarsub -l "core=1,walltime=0:05:00" --checkpoint 150 ./checkpoint.sh ''' | $ '''oarsub -v -l "core=1,walltime=0:05:00" --checkpoint 150 ./checkpoint.sh ''' | ||
# Modify resource description with type constraints | |||
OAR_JOB_ID=988555 | OAR_JOB_ID=988555 | ||
$ | $ | ||
Line 566: | Line 648: | ||
We submit the job again | We submit the job again | ||
$ '''oarsub -l "core=1,walltime=0:05:0" --checkpoint 150 ./checkpoint.sh ''' | $ '''oarsub -v -l "core=1,walltime=0:05:0" --checkpoint 150 ./checkpoint.sh ''' | ||
# Modify resource description with type constraints | |||
OAR_JOB_ID=988560 | OAR_JOB_ID=988560 | ||
Line 591: | Line 673: | ||
; First Job | ; First Job | ||
We run a first interactive job in a first Shell | We run a first interactive job in a first Shell | ||
frennes:~$ '''oarsub -I ''' | |||
# Set default walltime to 3600. | |||
OAR_JOB_ID=988571 | OAR_JOB_ID=988571 | ||
Interactive mode : waiting... | # Interactive mode: waiting... | ||
Starting... | # Starting... | ||
parasilo-28:~$ | parasilo-28:~$ | ||
And leave that job pending. | And leave that job pending. | ||
Line 605: | Line 684: | ||
Then we run a second job in another Shell, with a dependence on the first one | Then we run a second job in another Shell, with a dependence on the first one | ||
jdoe@idpot:~$ '''oarsub -I -a 988571''' | jdoe@idpot:~$ '''oarsub -I -a 988571''' | ||
# Set default walltime to 3600. | |||
OAR_JOB_ID=2071596 | |||
# Interactive mode: waiting... | |||
OAR_JOB_ID= | # [2018-01-15 14:27:08] Start prediction: 2018-01-15 15:30:23 (FIFO scheduling OK) | ||
Interactive mode : waiting... | |||
[2018-01-15 14:27:08] Start prediction: 2018-01-15 15:30:23 (FIFO scheduling OK) | |||
; Job dependency in action | ; Job dependency in action | ||
Line 619: | Line 696: | ||
... then watch the second Shell and see the second job starting | ... then watch the second Shell and see the second job starting | ||
[2018-01-15 14:27:08] Start prediction: 2018-01-15 15:30:23 (FIFO scheduling OK) | # [2018-01-15 14:27:08] Start prediction: 2018-01-15 15:30:23 (FIFO scheduling OK) | ||
Starting... | # Starting... | ||
parasilo-3:~$ | |||
== Container jobs == | == Container jobs == | ||
Line 730: | Line 807: | ||
=== Using the REST API === | === Using the REST API === | ||
Requesting the walltime change: | Requesting the walltime change: | ||
<code class="command>curl</code> -i -X POST <nowiki>https://api.grid5000.fr/ | <code class="command>curl</code> -i -X POST <nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">grenoble</code>/internal/oarapi/jobs/<code class="replace">1743185</code>.json -H'Content-Type: application/json' -d '{"method":"walltime-change", "walltime":"<code class="replace">+0:30:0</code>"}' | ||
Querying the status of the walltime change: | Querying the status of the walltime change: | ||
<code class="command>curl</code> -i -X GET <nowiki>https://api.grid5000.fr/ | <code class="command>curl</code> -i -X GET <nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">grenoble</code>/internal/oarapi/jobs/<code class="replace">1743185</code>/details.json -H'Content-Type: application/json' | ||
See the ''walltime-change'' and ''events'' keys of the output. | See the ''walltime-change'' and ''events'' keys of the output. | ||
Line 766: | Line 843: | ||
* a job will be kept 7 days before deletion (if it cannot be run because of lack of resources within a week), unless using <code>night=noretry</code> | * a job will be kept 7 days before deletion (if it cannot be run because of lack of resources within a week), unless using <code>night=noretry</code> | ||
= | == Use --project for users in multiple GGAs == | ||
A OAR job is linked to the user's Granting Group Access (GGA). Indeed, GGAs are used by OAR for applying the privilege levels defined in the [[Grid5000:UsagePolicy#Privilege_levels_table|usage policy]] and for statistics: | |||
* for users that belong to only one GGA, OAR automatically retrieves the GGA. | |||
* for users that belong to more than one GGA at once (eg., teaching, multiple affiliations, economic activities), they must use the <code>--project</code> parameter to indicate OAR which GGA is used for the job. | |||
Let us take an example for a user who is a member of group <code class="replace">projectA</code> for their research and of group <code class="replace">lab-session-B</code> for their teaching. | |||
To submit jobs related to their research (GGA <code class="replace">projectA</code>), they have to use the following command: | |||
{{Term|location=frontend|cmd=<code class="command">oarsub</code> <code class="command">--project=</code><code class="replace">projectA</code> ./myscript.sh}} | |||
while for reserving nodes for their lab sessions with students (GGA <code class="replace">lab-session-B</code>), they will use: | |||
{{Term|location=frontend|cmd=<code class="command">oarsub</code> <code class="command">-I -t cosystem -t container -l host=10,walltime=2:00:00 --project=</code><code class="replace">lab-session-B</code>}} | |||
{{Term|location=frontend|cmd=<code class="command"> | |||
Note that it is possible to define a default GGA (used by OAR without specifying a GGA with <code>--project</code> parameter) : | |||
* | * Login to https://api.grid5000.fr/ui/account | ||
* Go to the "Groups" tab (on the left menu), then select the default GGA by clicking on the button located on the 'Default' column. | |||
== About resources states == | |||
OAR resources can be in several states: | |||
* Alive: Free for use or running a job. | |||
* Absent: Temporarily unavailable for use, typically because rebooting after a deploy job. | |||
* Absent/standby: Free for use but not immediately available because shut down. Will be powered on and become Alive whenever needed for a job | |||
* Suspected: A fault has been detected, the resource is unavailable but it may be repaired soon. | |||
* Dead: The resource is definitively not available. | |||
Nodes in maintenance are nodes with a ''Dead'' state with the ''maintenance'' property set to ''YES''. | |||
''' | |||
= Funk = | = Funk = | ||
<code class="command">funk</code> is grid resources discovery tool that works at nodes level and generate complex <code class="command">oarsub | <code class="command">funk</code> is grid resources discovery tool that works at nodes level and generate complex <code class="command">oarsub</code> commands. It can help you in three cases: | ||
* to know the number of nodes availables for 2 hours at run time, on sites lille, rennes and on clusters taurus and suno | * to know the number of nodes availables for 2 hours at run time, on sites lille, rennes and on clusters taurus and suno | ||
{{Term|location=frontend|cmd=<code class="command">funk</code> -m <code class="replace">date</code> -r <code class="replace">lille,rennes,taurus,suno</code> -w <code class="replace">2:00:00</code>}} | {{Term|location=frontend|cmd=<code class="command">funk</code> -m <code class="replace">date</code> -r <code class="replace">lille,rennes,taurus,suno</code> -w <code class="replace">2:00:00</code>}} | ||
Line 836: | Line 884: | ||
= OAR in the Grid'5000 API = | = OAR in the Grid'5000 API = | ||
An other way to visualize nodes/jobs status is to use the [https://api.grid5000.fr/stable/ui/jobs.html Grid'5000 API] | An other way to visualize nodes/jobs status is to use the [https://api.grid5000.fr/stable/ui/jobs.html Grid'5000 API] | ||
= OAR database logs = | |||
Grid'5000 gives the possibility to all users to use a read only access to OAR's database. You should be able to connect using PostgresSQL client as user <code>oarreader</code> with password <code>read</code> to database <code>oar2</code> on all <code class=host>oardb.</code><code class=replace>site</code><code class=host>.grid5000.fr</code>. This gives you access to the complete history of jobs on all Grid'5000 sites. This gives you read-only access to the production database of OAR: please be careful with your queries to avoid overloading the testbed! | |||
{{Note|text=Careful: Grid'5000 is not a computation grid, nor HPC center, nor a Cloud (Grid'5000 is a research instrument). That means that the usage of Grid'5000 by itself (OAR logs of Grid'5000 users' reservations) does not reflect a typical usage of any such infrastructure. It is therefore not relevant to analyze Grid'5000 OAR logs to that purpose. As a user, one can however use Grid'5000 to emulate a HPC cluster or cloud on reserved resources (in a job), possibly injecting a real load from a real infrastructure.}} | |||
; Example of access to logs | |||
In this example, we use the PostgresSQL client to generate a CSV file, named '~/oardata.csv', containing all the jobs of the user 'toto'. Each row of the file will be one job of the user. The columns of the CSV file will be the list of nodes assigned to the job, the number of nodes, the number of cores, the cluster name, the submission time, start time and stop time, the job ID, the job name (if any), the job type, the command executed by the job (if any) and the request made by the user. | |||
First, on one of the frontend nodes, launch the client | |||
<syntaxhighlight lang="bash"> | |||
psql -h oardb.grenoble.grid5000.fr -U oarreader oar2 | |||
</syntaxhighlight> | |||
Then, after entering the password, run the following command (change the user name and the file name if needed): | |||
<syntaxhighlight lang="sql"> | |||
\copy (Select string_agg(Distinct host, '/') as hosts, Count(Distinct host) as nb_hosts, Count(Distinct resources.resource_id) as nb_cores,cluster,submission_time,start_time,stop_time,job_id,job_name,job_type,command,initial_request From jobs Inner Join assigned_resources on jobs.assigned_moldable_job = assigned_resources.moldable_job_id Inner Join resources on assigned_resources.resource_id = resources.resource_id Where job_user = 'toto' Group By jobs.submission_time,jobs.start_time,jobs.stop_time,jobs.job_id,jobs.job_name,jobs.job_type,jobs.command,resources.cluster) To '~/oardata.csv' With CSV; | |||
</syntaxhighlight> |
Latest revision as of 12:14, 12 June 2024
Note | |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
This tutorial consists of various independent sections describing various details of OAR useful for advanced usage, as well as some tips and tricks. It assumes you are familiar with OAR and Grid5000 basics. If not, please first look at the Getting Started page.
This OAR tutorial focuses on command line usages. It assumes you are using the bash shell (but should be easy to adapt to another shell). It can be read linearly, but you also may pick some random sections. Begin at least by #useful tips.
OAR
Useful tips
- Take the time to carefully configure ssh, as described in the SSH page.
- Use screen or tmux so that your work is not lost if you lose the connection to Grid5000. Moreover, having a screen session opened with one or more shell sessions allows you to leave your work session when you want then get back to it later and recover it exactly as you leaved it.
- Most OAR commands (
oarsub
,oarstat
,oarnodes
) can provide output in various formats:- text (this is the default mode)
- PERL dumper (-D)
- XML (-X)
- Yaml (-Y)
- json (-J)
- Regarding the
oarsub
command line, you should mostly only see the "host" word, but theoarsub
command can use both the word "host" or "nodes" indifferently in Grid'5000, as nodes is just an alias for host. Prefer using "host". Besides, the word "host" is also to be preferred to the longer "network_address" word in the resources filters (both properties have sometime the same value, but not always). - At job submission time, only important information are printed out by
oarsub
. To have more indication about what is done by OAR on Grid'5000 (like computed resource filter, exceptional granted privileges, …) theoarsub
verbose (-v
) option can be used. - A syntax simplification mechanism was deployed on Grid'5000 to ease job submission, described at OAR Syntax simplification.
Connection to the job's nodes
Two commands can be used to connect to nodes on Grid'5000, oarsh
and ssh
.
Using ssh
ssh
can only be used when a node is entirely reserved in your job (all CPU cores). Other cases may not allow assigning processes to the correct job, thus connecting with ssh
is not allowed.
For instance, when a node is entirely reserved as follows:
# Set walltime to default (3600 s). OAR_JOB_ID=<JOB_ID> # Interactive mode: waiting... # Starting... user@node-32:~$
If you open a new shell and try to connect to the node with ssh
, it should work:
Linux node-32.site.grid5000.fr 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 Debian11-x64-std-2022013022 (Image based on Debian Bullseye for AMD64/EM64T) Maintained by support-staff <support-staff@lists.grid5000.fr> Last login: Wed Feb 23 15:20:32 2022 from 172.16.31.101 user@node-32:~$
However, when reserving for instance only one CPU core of a node:
# Set walltime to default (3600 s). OAR_JOB_ID=<JOB_ID> # Interactive mode: waiting... # Starting... user@node-32:~$
When trying to connect to the node with ssh
in another shell, you get:
To connect using 'ssh' directly, you must have a single job using all available cores on the node. Use 'oarsh' instead. Connection closed by node-32 port 22
Using oarsh
oarsh
is a frontend to ssh
(the oarsh
command wraps the OpenSSH ssh
command to add some required functions to connect to a job, but provides mostly the same interface/options).
In case of nodes are not entirely reserved (all CPU cores), you have to use the oarsh
command to connect to nodes instead of ssh
, and oarcp
instead of scp
to copy files to/from the nodes. If you use taktuk
for parallel executions (or a similar tools like pdsh
) or rsync
to synchronize files to/from a node, you have to configure the connector so the command uses oarsh
instead of ssh
underneath (see the man pages of the command to find out how to change the connector, e.g. using -c or -e).
Please note that oarsh
also works for node entirely reserved in a job.
Splitting job resources
oarsh
also allows splitting resources of a job, for instance to execute commands on different subsets of resources in a job (e.g. 1 GPU each instead of all the reserved GPUs).
See an example of using this functionality with GNU Parallel.
About OAR job keys
By default, OAR generates a job key pair for each job. oarsh
can use either the OAR_JOB_ID
or OAR_JOB_KEY_FILE
environment variables to know what job to connect. If outside a job shell (e.g. on the frontend), you have to set one of those environment variable. This is not required if oarsh
is called from the shell of a job (e.g. on a node), since variables are already set.
- Example using
OAR_JOB_ID
For instance, create a job requesting 3 hosts (3 nodes):
# Set default walltime to 3600. OAR_JOB_ID=<JOBID> # Interactive mode: waiting... # Starting... ...
Then, in another terminal, assuming the 2nd host in the job is named node-2
:
- Example using
OAR_JOB_KEY_FILE
OAR can expose the job key, using the -e
option of oarsub
Then, in another terminal, assuming the 2nd host in the job is named node-2
:
Note | |
---|---|
Note that the following command also allows getting a shell in a job, but only on the first default resource (i.e. node). |
Connecting to a job of a different site
Job keys are especially useful when having to connect from nodes of different sites, since each site is managed by a different OAR instance.
Thus, a convenient way is to tell OAR to always use the same job key for all jobs. You can for instance use your Grid'5000 internal SSH key (This key is generated when your account is created) as the job key: In your ~/.profile
or ~/.bash_profile
, set:
export OAR_JOB_KEY_FILE=path_to_your_private_key
Then, OAR will always use that key for all jobs, allowing you to connect to your nodes with oarsh
seamlessly from sites to sites, jobs to jobs, or even outside jobs.
oarsh vs ssh: tips and tricks
Note | |
---|---|
The following is only interesting if you jobs do not reserve nodes entirely, as using |
- 1st tip - hide
oarsh
, rename itssh
Creating a symlink from ~/bin/ssh
(assuming it is in the execution PATH
) to /usr/bin/oarsh
allows hidding the wrapper use (as long as the OAR_JOB_ID
or OAR_JOB_KEY_FILE
environment variables are set when connecting from a frontend to a node).
- 2nd tip - using
ssh
directly, withoutoarsh
If using oarsh
does not suit your need, because you would like to use some of the options of ssh
that oarsh
does not support, you can also connect to reserved nodes by using the real ssh
by adding the right set of options to the command. It can also allow a connection to reserved nodes directly from some place where oarsh
is not available (e.g. from outside Grid'5000):
Assuming you have a passphrase-less SSH key (preferably just for internal uses in Grid5000), you can tell oarsub
to use that key as a job key instead of letting OAR generate a new one (see #sharing keys between jobs). Then you can use that key to connect to nodes, even from outside Grid'5000.
- Copy the key to your workstation, for instance outside of Grid5000:
- In Grid5000, submit a job using this key:
- Wait for the job to start. Then in another terminal, from outside Grid5000, try connecting to the node:
workstation :
|
ssh -i ~/
your_internal_private_key_file -p 6667 [any other ssh options] oar@reserved-node .site .g5k |
Finally, this can be hidden in a SSH ProxyCommand (See also SSH#Using_SSH_ProxyCommand_feature_to_ease_the_access_to_hosts_inside_Grid.275000):
After adding the following configuration in your OpenSSH configuration file on your workstation (~/.ssh/config
):
Host *.g5koar ProxyCommand sshg5k-username
@access.grid5000.fr -W "$(basename %h .g5koar):%p" User oar Port 6667 IdentityFile ~/your_internal_private_key_file
ForwardAgent no
Warning: the ProxyCommand
line works if your login shell is bash
. If not you may have to adapt it.
You can just ssh to a reserved node directly from your workstation as follows:
Passive and interactive job modes
Interactive mode
In interactive mode, a shell is opened on the first default resource (i.e. node) of the job (or on the frontend, if the job is of type deploy
or cosystem
). In interactive mode, the job will be terminated as soon as this job's shell is closed or will be killed earlier if the job's walltime
is reached. It can also be killed by an explicit oardel
.
You can experiment with 3 shells. On first shell, to see the list of your running jobs, regularly run:
To see your own jobs. On the second shell, run an interactive job:
Wait for the job to start, run oarstat
, then leave the job, run oarstat
again. Submit another interactive job, and on the third shell, kill it:
Passive mode
In passive mode, the command
that is given to oarsub
is executed on the first default resource (i.e. node) of the job (or on the site's frontend if the job is of type deploy
or cosystem
). The job's duration will be the shorter of the execution time of the command
and the job's given walltime
. That unless the job is terminated beforehand by an explicit oardel
call from the user or administrator.
command can be a simple script name or a more complex command line with arguments.
To pass arguments, you have to quote the whole command line, like in the following example:
oarsub
-l nodes=4,walltime=2"/path/to/myscript arg1 arg2 arg3"
Note: to avoid random code injection, oarsub
allows only alphanumeric characters ([a-zA-Z0-9_]
), whitespace characters ([ \t\n\r\f\v]
) and few others ([/.-]
) inside its command line argument.
Special case for jobs of type noop
which are always passive jobs: no command is executed for them. The duration of the job is the given walltime
.
oardel
can also be used to terminate a passive mode reservation. Note that it is only possible to remove the complete reservation, and not individual nodes.
Interactive mode without shell
You may not want a job to open a shell or to run a script when the job starts, for example because you will use the reserved resources from a program whose lifecycle is longer than the job (and which will use the resources by connecting to the job).
One trick to achieve this is to run the job in passive mode with a long sleep
command. One drawback of this method is that the job may terminate with status error if the sleep
is killed. This can be a problem in some situations, eg. when using job dependencies.
Another solution is to use an advance reservation (see below) with a starting date very close in the future, or even with the current date and time.
Batch jobs vs. advance reservation jobs
- Batch jobs
If you do not specify the job's start date (oarsub -r
option), then your job is a batch job. It lets OAR choose the best schedule (start date).
- With batch jobs, you're guaranteed to get the count of allocated resources you requested, because OAR chooses what resources to allocate to the job just before its start. If some resources suddenly become unavailable, OAR changes the assigned resources and/or the start date.
- Therefore, you cannot get the actual list of resources until the job starts (but a forecast is provided, such as what is shown in the Drawgantt diagrams).
- With batch jobs, you cannot know the start date of your job until it actually starts (any event can change the forecast). But OAR gives an estimation of the start date (such as shown in the Drawgantt diagram, which also changes after any event).
- Advance reservations
If you specify the job's start date, it is an advance reservation. OAR will just try to find resources for the given schedule, fixed by you.
- The Grid5000 usage policy allows no more than 2 advance reservations per site (excluding reservations that start in less than one hour)
- With advance reservation jobs, you're not guaranteed to get the count of resources you requested, because OAR planned the allocation of resources at the reservation time.
- If some resources became unavailable when the job has to start, the job is delayed a bit in case resources may come back (e.g. return from standby).
- If after 400 seconds, if not all resources are available, the job will start with fewer resources than initially allocated. This is however quite unusual.
- The list of allocated resources to an advance reservation job is fixed and known as soon as the advance reservation is validated. But you will get the actual list of resources (that is with unavailable resources removed for it) when the advance reservation starts.
- To coordinate the start date of OAR jobs on several sites you can use funk which has advanced reservation options.
Example: a reservation for a job in one week from now
$ oarsub -r "$(date +'%F %T' --date='+1 week')"
For advance reservations, there is no interactive mode. You can give OAR a command to execute or nothing. If you do not give a command, you'll have to connect to the jobs once the reservation starts (using oarsub -C <jobid> or oarsh).
Why did my advance reservation start with less than all the resources I requested ?
Since resources states are transitional, the advance reservation process considers indifferently the current state of resources, be it alive, suspended or absent. Indeed, at the requested start time of an advance reservation, all resources in any of those states should presumably be back in the alive state.
This is different for resources in the dead state, which mark failed resources. Although dead resources may be repaired at some point, that state is less transitional, so dead resources are excluded from eligible resources for advance reservations.
Also, the allocation of resources to an advance reservation is fixed at the time of validation of the submission (contrarily to batch jobs for which both the start time and allocated resources can change up until the job is effectively started, in order to fit with all requested resources available). As a consequence, resources allocated to an advance reservation which would end up unavailable at the job start time are not replaced by other alive resources.
In fact, at the start time of an advance reservation, OAR looks after any unavailable resources (absent or suspected), and whenever some exists, wait for them to return to the alive state for 5 minutes. Then, if they are not back in time, the job starts with less resources than requested and initially allocated (assuming at least one resource is available).
- NB
Information about reduced number of resources or reduced walltime for a reservation due to this mechanism are available in the event part of the output of
oarstat -fj
jobid
Getting information about a job
The oarstat
command gets jobs informations. By default it lists the current jobs of all users. You can restrict it to your own jobs or someone else's jobs with option -u
:
$ oarstat -u
You can get full details of a job:
$ oarstat -fj <JOBID>
If scripting OAR and regularly polling job states with oarstat, you can cause a high load on the OAR server (because default oarstat invocation causes costly SQL request in the OAR database). In this case, you should use option -s
which is optimized and only queries the current state of a given job:
$ oarstat -s -j <JOBID>
Complex resources selection
The complete selector format syntax (oarsub -l
option) is:
"-l {sql1}/name1=n1/name2=n2+{sql2}/name3=n3/name4=n4/name5=n5+...,walltime=hh:mm:ss"
where
- sqlN are optional SQL predicates on the resource properties (e.g. mem, ib_rate, gpu_count, ...)
- nameN=n are the wanted number of given resources of name nameN (e.g. host, cpu, core, disk...).
- slashes (/) between resources express resource subtree selection
- + allows aggregating different resource specifications
- walltime=hh:mm::ss (separated by a comma) sets the job walltime (expected duration), which defaults to 1 hour
- List resource properties
You can get the list of resource properties for SQL predicates by running the oarprint -l
command on a node:
sagittaire-1 $ oarprint -l List of properties: disktype, gpu_count, ...
You can get the property values set to resources using the oarnodes
:
flyon $ oarnodes -Y --sql="host = 'sagittaire-1.lyon.grid5000.fr'"
These OAR properties are described in the OAR Properties page.
Using the resources hierarchies
The OAR resources define implicit hierarchies to be used on the resource requests (oarsub -l). These hierarchies are specific to Grid'5000.
- For instance
- request 1 core on 15 hosts (nodes) on a same cluster (total = 15 cores)
$ oarsub -I -l /cluster=1/host=15/core=1
- request 1 core on 15 hosts (nodes) on 2 clusters (total = 30 cores)
$ oarsub -I -l /cluster=2/host=15/core=1
- request 1 core on 2 cpus on 15 hosts (nodes) on a same cluster (total = 30 cores)
$ oarsub -I -l /cluster=1/host=15/cpu=2/core=1
- request 10 cpus on 2 clusters (total = 20 cpus, the number of hosts and cores depends on the topology of the machines)
$ oarsub -I -l /cluster=2/cpu=10
- request 1 core on 3 different network switches (total = 3 cores)
$ oarsub -I -l /switch=3/core=1
- Examples for GPUs
- request 3 GPUs on 1 single host (node). Obviously eligible nodes for the job need to have at least 3 GPU.
$ oarsub -I -l host=1/gpu=3
- request 3 GPUs, possibly on different nodes depending on availability (other jobs, possible resources):
$ oarsub -I -l gpu=3
- request a full node (possibly featuring more than 3 GPUs) with at lease 3 GPUs:
$ oarsub -p "gpu_count >= 3" -l host=1 [...]
- In the job, running oarprint as follows shows what GPUs are available in the job:
$ oarprint gpu -P host,gpudevice
(you may also look at nvidia-smi's output)
- Valid resource hierarchies are
- Compute and disk resources
- both switch > cluster, or cluster > switch can be valid (some clusters spread their hosts (nodes) on many switches, some clusters share a same switch), we note below cluster|switch to reflect that ambiguity.
- cluster|switch > chassis > host > cpu > gpu > core
- cluster|switch > chassis > host > disk
- vlan resources
- vlan only
- subnet resources
- slash16 > slash17 > slash18 > slash19 > slash20 > slash21 > slash22
Of course not all hierarchy levels have to be given in a resource request.
Selecting resources using properties
The properties of the resources are described in the OAR Properties page.
- Selecting nodes from a specific cluster
For example in Nancy:
$ oarsub -I -l {"cluster='graphene'"}/host=2
Or, alternative syntax:
$ oarsub -I -p "cluster='graphene'" -l /host=2
- Selecting nodes with a specific CPU architecture
For classical x86_64:
$ oarsub -I -p "cpuarch='x86_64'"
Other architectures are "exotic" so a specific type of job is needed:
$ oarsub -I -t exotic -p "cpuarch='ppc64le'"
- Selecting specific nodes
For example in Lyon:
$ oarsub -I -l {"host in ('sagittaire-10.lyon.grid5000.fr', 'sagittaire-11.lyon.grid5000.fr', 'sagittaire-12.lyon.grid5000.fr')"}/host=1
or, alternative syntax:
$ oarsub -I -p "host in ('sagittaire-10.lyon.grid5000.fr', 'sagittaire-11.lyon.grid5000.fr', 'sagittaire-12.lyon.grid5000.fr')" -l /nodes=1
By negating the SQL clause, you can also exclude some nodes.
- Other examples using properties
Ask for 10 cores of the cluster graphene
$ oarsub -I -l core=10 -p "cluster='graphene'"
Ask for 2 nodes with 16384 MB of memory and Infiniband 20G
$ oarsub -I -p "memnode='16384' and ib_rate='20'" -l host=2
Ask for any 4 nodes except graphene-12
$ oarsub -I -p "not host like 'graphene-12.%'" -l host=4
Ask for a node with 32 threads
$ oarsub -I -p 'thread_count=32'
- Examples of joint resources requests
Ask for 2 nodes with virtualization capability, on different clusters + IP subnets:
- We want 2 nodes (hosts) and 4 /22 subnets with the following constraints:
- Nodes are on 2 different clusters of the same site (Hint: use a site with several clusters :-D)
- Nodes have virtualization capability enabled
- /22 subnets are on two different /19 subnets
- 2 subnets belonging to the same /19 subnet are consecutive
$ oarsub -I -l /slash_19=2/slash_22=2+{"virtual!='none'"}/cluster=2/host=1
Lets verify the reservation:
$ uniq $OAR_NODE_FILE graphene-43.nancy.grid5000.fr graphite-3.nancy.grid5000.fr
$ g5k-subnets -p 10.144.32.0/22 10.144.36.0/22 10.144.0.0/22 10.144.4.0/22
$ g5k-subnets -ps 10.144.0.0/21 10.144.32.0/21
Another example, ask for both
- 1 core on 2 hosts (nodes) on the same cluster with 16384 MB of memory and Infiniband 20G
- 1 cpu on 2 hosts (nodes) on the same switch with 8 cores processors for a walltime of 4 hours
$ oarsub -I -l "{memnode=16384 and ib_rate='20'}/cluster=1/host=2/core=1+{cpucore=8}/switch=1/host=2/cpu=1,walltime=4:0:0"
Walltime must always be the last argument of -l <...>
Handling the resources allocated to my job with oarprint
The oarprint
allows to print nicely the resources of a job.
We first submit a job
$ oarsub -I -l host=4 ... OAR_JOB_ID=178361
- Retrieve the nodes list
We want the list of the nodes (hosts) we got, identified by unique hostnames
$ oarprint host sagittaire-32.lyon.grid5000.fr capricorne-34.lyon.grid5000.fr sagittaire-63.lyon.grid5000.fr sagittaire-28.lyon.grid5000.fr
(We get 1 line per host, not per core !)
- Retrieve the core list
$ oarprint core 63 241 64 163 243 244 164 242
Obviously, retrieving OAR internal core Id might not help much. Hence the use of a customized output format below.
- Retrieve core list with host and cpuset Id as identifier
We want to identify our cores by their associated host names and cpuset Ids:
$ oarprint core -P host,cpuset capricorne-34.lyon.grid5000.fr 0 sagittaire-32.lyon.grid5000.fr 0 capricorne-34.lyon.grid5000.fr 1 sagittaire-28.lyon.grid5000.fr 0 sagittaire-63.lyon.grid5000.fr 0 sagittaire-63.lyon.grid5000.fr 1 sagittaire-28.lyon.grid5000.fr 1 sagittaire-32.lyon.grid5000.fr 1
- A more complex example with a customized output format
We want to identify our cores by their associated host name and cpuset Id, and get the memory information as well, with a customized output format
$ oarprint core -P host,cpuset,memnode -F "NODE=%[%] MEM=%" NODE=capricorne-34.lyon.grid5000.fr[0] MEM=2048 NODE=sagittaire-32.lyon.grid5000.fr[0] MEM=2048 NODE=capricorne-34.lyon.grid5000.fr[1] MEM=2048 NODE=sagittaire-28.lyon.grid5000.fr[0] MEM=2048 NODE=sagittaire-63.lyon.grid5000.fr[0] MEM=2048 NODE=sagittaire-63.lyon.grid5000.fr[1] MEM=2048 NODE=sagittaire-28.lyon.grid5000.fr[1] MEM=2048 NODE=sagittaire-32.lyon.grid5000.fr[1] MEM=2048
- From the submission frontend
If you are not in a job shell ($OAR_RESOURCE_PROPERTIES_FILE
is not defined), running oarprint
will give:
$ oarprint /usr/bin/oarprint: no input data available
In that case, you can however pipe the output of the oarstat
command in oarprint
, e.g.:
$ oarstat -j <JOB_ID> -p | oarprint core -P host,cpuset,memnode -F "%[%] (%)" -f - capricorne-34.lyon.grid5000.fr[0] (2048) sagittaire-32.lyon.grid5000.fr[0] (2048) capricorne-34.lyon.grid5000.fr[1] (2048) sagittaire-28.lyon.grid5000.fr[0] (2048) sagittaire-63.lyon.grid5000.fr[0] (2048) sagittaire-63.lyon.grid5000.fr[1] (2048) sagittaire-28.lyon.grid5000.fr[1] (2048) sagittaire-32.lyon.grid5000.fr[1] (2048)
- List the OAR properties to use with oarprint
Properties are descibed in the OAR Properties page, but they can also be listed using the oarprint -l
command:
$ oarprint -l List of properties: disktype, gpu_count, ...
X11 forwarding
X11 forwarding is enabled in the shell opened in interactive job (oarsub -I
).
X11 forwarding can also be enabled in a shell opened on a node of a job with oarsh
, just like with a classic ssh
command: The -X
or -Y
option must be passed to oarsh
.
Note | |
---|---|
Please mind that for X11 forwarding to work in the job, X11 forwarding must already work in the shell from which the OAR commands are run. Check the |
We will use xterm
to test X11.
- Enabling X11 forwarding up to the frontend
Connect to a frontend with ssh
(reminder: read the getting started tutorial about the use of the ssh proxycommand), and make sure the X11 forwarding is operational so far:
Look at the DISPLAY
environment variable, which ssh should have set to localhost
:10.0
or the like (the 10.0
part may vary from hop to hop in the X11 forwarding chain, with numbers greater than 10).
It requires to use the -X
or -Y
option in the ssh
command line, or to have ForwardX11=yes
set in your SSH configuration.
In any case, check:
localhost:11.0
- Using X11 forwarding in the
oarsub
job shell
If the DISPLAY
environment variable is set in the calling shell, oarsub
will automatically enable the X11 forwarding. Verbose oarsub option (-v
) is required to have the "Initialize X11 forwarding..." sentence.
# Set default walltime to 3600. # Computed global resource filter: -p "maintenance = 'NO'" # Computed resource request: -l {"type = 'default'"}/core=1 # Generate a job key... OAR_JOB_ID=4926 # Interactive mode: waiting... # Starting... # Initialize X11 forwarding... # Connect to OAR job 4926 via node idpot-8.grenoble.grid5000.fr
Then from the shell of the job, check again the display:
jdoe@idpot-8:~$ echo $DISPLAY localhost:10.0
And run xterm
jdoe@idpot-8:~$ xterm
Wait for the window to open: it may be pretty long!
- Using X11 forwarding in a job via
oarsh
With oarsh
, the -X
or -Y
option must be used to enable the X11 forwarding:
Then in the opened shell, you can again check that the DISPLAY
is set, and run xterm
.
You can also just run the xterm
command directly in the oarsh
call:
- Using X11 forwarding in a job with a deployed environment
When an interactive job is used to deploy an environment, the spawned shell will not contain the DISPLAY
environment variable, even if it was forwarded in the user connection shell.
To use X11 forwarding in this situation, you can open a new (X11 forwarded) shell on the frontend, and then connect to the node using again X11 forwarding.
you can also connect directly to the node from your laptop either by:
- using the Grid'5000 VPN
- following the recommendations about a better usage of ssh listed in Getting Started document.
Note | |
---|---|
X11 forwarding will suffer from the latency between your local network and the Grid'5000 network.
|
Using best effort mode jobs
The best-effort jobs of OAR are implemented to back-fill the cluster with jobs considered as less important without blocking "regular" jobs. To submit jobs under that policy, you simply have to select the besteffort type of job in your oarsub command.
oarsub
-t besteffort
script_to_launch
Jobs submitted that way will only get scheduled on resources when no other job use them (any regular job overtake besteffort jobs in the waiting queue, regardless of submission times). Moreover, these jobs are killed (as if oardel were called) when a regular job recently submitted needs the nodes used by a besteffort job.
By default, no checkpointing or automatic restart of besteffort jobs is provided. They are just killed. That is why this mode is best used with a tool which can detect the killed jobs and resubmit them. However OAR2 provides options for that.
Best effort job campaign
One can submit such jobs using the besteffort of job type (or indifferently in the besteffort queue).
For instance you can run a job campaign as follows:
for param in $(< ./paramlist); do oarsub -t besteffort -l core=1 "./my_script.sh $param" done
In this example, the file ./paramlist
contains a list of parameters for a parametric application.
The following demonstrates the mechanism.
Note | |
---|---|
Please have a look at the UsagePolicy to avoid abuses. |
Best effort job mechanism
- Running a besteffort job in a first shell
frennes:~$ oarsub -I -l host=10 -t besteffort # Set default walltime to 3600. OAR_JOB_ID=988535 # Interactive mode: waiting... # Starting... parasilo-26:~$ uniq $OAR_FILE_NODES parasilo-26.rennes.grid5000.fr parasilo-27.rennes.grid5000.fr parasilo-28.rennes.grid5000.fr parasilo-3.rennes.grid5000.fr parasilo-4.rennes.grid5000.fr parasilo-5.rennes.grid5000.fr parasilo-6.rennes.grid5000.fr parasilo-7.rennes.grid5000.fr parasilo-8.rennes.grid5000.fr parasilo-9.rennes.grid5000.fr
- Running a non best effort job on the same set of resources in a second shell
frennes:~$ oarsub -I -l {"host in ('parasilo-9.rennes.grid5000.fr')"}/host=1 # Set default walltime to 3600. OAR_JOB_ID=988546 # Interactive mode: waiting... # [2022-01-10 16:00:07] Start prediction: 2022-01-10 16:00:07 (FIFO scheduling OK) # Starting... Connect to OAR job 988546 via the node parasilo-9.rennes.grid5000.fr
As expected, meanwhile the best effort job was stopped (watch the first shell):
parasilo-26:~$ Connection to parasilo-26.rennes.grid5000.fr closed by remote host. Connection to parasilo-26.rennes.grid5000.fr closed. # Error: job was terminated. Disconnected from OAR job 988545
Using the checkpointing trigger mechanism
- Writing the test script
Here is a script which features an infinite loop and a signal handler trigged by SIGUSR2 (default signal for OAR's checkpointing mechanism).
#!/bin/bash handler() { echo "Caught checkpoint signal at: `date`"; echo "Terminating."; exit 0; } trap handler SIGUSR2 cat <<EOF Hostname: `hostname` Pid: $$ Starting job at: `date` EOF while : ; do sleep 10; done
- Running the job
We run the job on 1 core, and a walltime of 5 minutes, and ask the job to be checkpointed if it lasts (and it will indeed) more than walltime - 150 sec = 2 min 30.
$ oarsub -v -l "core=1,walltime=0:05:00" --checkpoint 150 ./checkpoint.sh # Modify resource description with type constraints OAR_JOB_ID=988555 $
- Result
Taking a look at the job output:
$ cat OAR.988555.stdout Hostname: parasilo-9.rennes.grid5000.fr Pid: 12013 Starting job at: Mon Jan 15 14:05:50 CET 2018 Caught checkpoint signal at: Mon Jan 15 14:08:30 CET 2018 Terminating.
The checkpointing signal was sent to the job 2 minutes 30 before the walltime as expected so that the job can finish nicely.
- Interactive checkpointing
The oardel
command provides the capability to raise a checkpoint event interactively to a job.
We submit the job again
$ oarsub -v -l "core=1,walltime=0:05:0" --checkpoint 150 ./checkpoint.sh # Modify resource description with type constraints OAR_JOB_ID=988560
Then run the oardel -c #jobid
command...
$ oardel -c 988560 Checkpointing the job 988560 ...DONE. The job 988560 was notified to checkpoint itself (send SIGUSR2).
And then watch the job's output:
$ cat OAR.988560.stdout Hostname: parasilo-4.rennes.grid5000.fr Pid: 11612 Starting job at: Mon Jan 15 14:17:25 CET 2018 Caught checkpoint signal at: Mon Jan 15 14:17:35 CET 2018 Terminating.
The job terminated as expected.
Using jobs dependency
A job can wait for the termination of a previous job.
- First Job
We run a first interactive job in a first Shell
frennes:~$ oarsub -I # Set default walltime to 3600. OAR_JOB_ID=988571 # Interactive mode: waiting... # Starting... parasilo-28:~$
And leave that job pending.
- Second Job
Then we run a second job in another Shell, with a dependence on the first one
jdoe@idpot:~$ oarsub -I -a 988571 # Set default walltime to 3600. OAR_JOB_ID=2071596 # Interactive mode: waiting... # [2018-01-15 14:27:08] Start prediction: 2018-01-15 15:30:23 (FIFO scheduling OK)
- Job dependency in action
We do a logout on the first interactive job...
parasilo-28:~$ logout Connection to parasilo-28.rennes.grid5000.fr closed. Disconnected from OAR job 988571
... then watch the second Shell and see the second job starting
# [2018-01-15 14:27:08] Start prediction: 2018-01-15 15:30:23 (FIFO scheduling OK) # Starting... parasilo-3:~$
Container jobs
With the container job functionality, OAR allows for someone to execute inner jobs within the boundaries of the container job. Inner jobs are scheduled using the same algorithm as other jobs, but restricted to the container job's resources and timespan.
A typical use case is to submit first a container job, then have inner jobs submitted, with referring to the container job_id.
Mind that the inner jobs that will not fit in the container's boundaries will stay in the waiting state in the queue, not scheduled and not executed. They will be deleted when the container job is terminated.
Container jobs are especially useful when organizing tutorial of teaching labs, with the container job created by the organizer, and inner jobs created by the attendees.
Mind that if in your use case, all inner job are to be created by the same user as the container job, it is preferable to use a tool such as GNU Parallel.
Inner job are killed when the container job is terminated.
- First a job of the type container must be submitted
... OAR_JOB_ID=42 ...
- Then it is possible to use the inner type to schedule the new jobs within the previously created container job
Note | |
---|---|
A job created with: will never be scheduled because the container job "42" only reserved 10 nodes. |
cosystem and noop jobs
- cosystem
- Jobs of type cosystem, just like jobs of type deploy, do not execute on the first node assigned to the job but on the frontend.
- But unlike deploy jobs, cosystem jobs do not grant any special privileges (e.g. no kareboot right).
- noop
- Jobs of type noop do not execute anything at all. They just allocate resources for a time frame.
- noop jobs cannot be interactive (oarsub -I).
- noop jobs have the advantage over the cosystem job that they are not affected by a reboot (e.g. due to a maintenance or a failure) of the frontend.
If running a script on the frontend is not required, noop job should probably be preferred over the cosystem jobs.
Changing the walltime of a running job (oarwalltime)
Users can request a extension of the walltime (duration of the resource reservation) of a running job. This can be achieved using the oarwalltime
command or Grid'5000's API.
This change can be specified by giving either a new walltime value or an increase (begin with +).
Please note that a request may stay partially or completely unsatisfied if a job is already scheduled to occupy the resources right after the running job.
Job must be running for a walltime change. For Waiting job, delete and resubmit.
Warning | |
---|---|
While changes of walltime are not limited a priori (by the |
Command line interface
Querying the walltime change status:
Walltime change status for job 1743185 (job is running): Current walltime: 1:0:0 Possible increase: UNLIMITED Already granted: 0:0:0 Pending/unsatisfied: 0:0:0
Requesting the walltime change:
Accepted: walltime change request updated for job 1743185, it will be handled shortly.
Querying right afterward:
Walltime change status for job 1743185 (job is running): Current walltime: 1:0:0 Possible increase: UNLIMITED Already granted: 0:0:0 Pending/unsatisfied: +1:30:0
The request is still to be handled by OAR's scheduler.
Querying again a bit later:
Walltime change status for job 1743185 (job is running): Current walltime: 2:30:0 Possible increase: UNLIMITED Already granted: +1:30:0 Pending/unsatisfied: 0:0:0
May a job exist on the resources and partially prevent the walltime increase, the query output would be:
Walltime change status for job 1743185 (job is running): Current walltime: 2:30:0 Possible increase: UNLIMITED Already granted: +1:10:0 Pending/unsatisfied: +0:20:0
Changes events are also reported in oarstat
.
See man oarwalltime
for more information.
Using the REST API
Requesting the walltime change:
curl
-i -X POST https://api.grid5000.fr/stable/sites/grenoble
/internal/oarapi/jobs/1743185
.json -H'Content-Type: application/json' -d '{"method":"walltime-change", "walltime":"+0:30:0
"}'
Querying the status of the walltime change:
curl
-i -X GET https://api.grid5000.fr/stable/sites/grenoble
/internal/oarapi/jobs/1743185
/details.json -H'Content-Type: application/json'
See the walltime-change and events keys of the output.
Restricting jobs to daytime or night/week-end time
To help submitting batch jobs fitting inside the time frames defined in the usage policy (day vs. night and week-end), the types day
and night
can be used (oarsub -t <type>…
).
- Submit a job to run during the current day time
As such:
- It will be forced to run between 9:00 and 19:00, or the next day if the job is submitted during the night.
- If the job did not succeed to run before 19:00, it will be deleted.
- Submit a job to run during the coming (or current) night (or week-end on Friday)
As such:
- It will be forced to run after 19:00, and before 9:00 for week nights (Monday to Thursday nights), or before 9:00 on the next Monday for a job which runs during a week-end.
- If a job could not be scheduled during the current night (not enough resources available), it will be kept in the queue and then postponed in the morning for a retry the next night (hour constraints will be changed to the next night slot), that for 7 days.
- If the walltime of the job is more than 13h59, the job will obviously not run before a weekend.
- Submit a job to run exclusively during the coming (or current) night (or week-end on Friday)
If job is not scheduled and run during the coming (or current) night (or week-end on Friday), it will not be postponed to the next night for a new try, but just set to error.
Note that:
- the maximum walltime for a night is 14h, but due to some overhead in the system (resources state changes, reboots...), it is strongly advised to limit walltime to at most 13h30. Furthermore, a shorter walltime (max a few hours)? will result in more chances to get a job scheduled in case many jobs are already in queue.
- jobs with a walltime greater than 14h will be required to run during the week-ends. But even if submitted at the beginning of the week, they will not be scheduled before the Friday morning. Thus, any advance reservation done before Friday will take precedence. Also, given that the rescheduling happens on a daily basis for the next night, advance reservations take precedence if they are submitted before the daily rescheduling. In practice, this mechanism thus provides a low priority way to submit batch jobs during nights and week-ends.
- a job will be kept 7 days before deletion (if it cannot be run because of lack of resources within a week), unless using
night=noretry
Use --project for users in multiple GGAs
A OAR job is linked to the user's Granting Group Access (GGA). Indeed, GGAs are used by OAR for applying the privilege levels defined in the usage policy and for statistics:
- for users that belong to only one GGA, OAR automatically retrieves the GGA.
- for users that belong to more than one GGA at once (eg., teaching, multiple affiliations, economic activities), they must use the
--project
parameter to indicate OAR which GGA is used for the job.
Let us take an example for a user who is a member of group projectA
for their research and of group lab-session-B
for their teaching.
To submit jobs related to their research (GGA projectA
), they have to use the following command:
while for reserving nodes for their lab sessions with students (GGA lab-session-B
), they will use:
Note that it is possible to define a default GGA (used by OAR without specifying a GGA with --project
parameter) :
- Login to https://api.grid5000.fr/ui/account
- Go to the "Groups" tab (on the left menu), then select the default GGA by clicking on the button located on the 'Default' column.
About resources states
OAR resources can be in several states:
- Alive: Free for use or running a job.
- Absent: Temporarily unavailable for use, typically because rebooting after a deploy job.
- Absent/standby: Free for use but not immediately available because shut down. Will be powered on and become Alive whenever needed for a job
- Suspected: A fault has been detected, the resource is unavailable but it may be repaired soon.
- Dead: The resource is definitively not available.
Nodes in maintenance are nodes with a Dead state with the maintenance property set to YES.
Funk
funk
is grid resources discovery tool that works at nodes level and generate complex oarsub
commands. It can help you in three cases:
- to know the number of nodes availables for 2 hours at run time, on sites lille, rennes and on clusters taurus and suno
- to know when 40 nodes on sagittaire and 4 nodes on taurus will be available, with deploy job type and a subnet
- to find the time when the maximum number of nodes are available during 10 hours, before next week deadline, avoiding usage policy periods, and not using genepi
More information on its dedicated page.
OAR in the Grid'5000 API
An other way to visualize nodes/jobs status is to use the Grid'5000 API
OAR database logs
Grid'5000 gives the possibility to all users to use a read only access to OAR's database. You should be able to connect using PostgresSQL client as user oarreader
with password read
to database oar2
on all oardb.
site
.grid5000.fr
. This gives you access to the complete history of jobs on all Grid'5000 sites. This gives you read-only access to the production database of OAR: please be careful with your queries to avoid overloading the testbed!
- Example of access to logs
In this example, we use the PostgresSQL client to generate a CSV file, named '~/oardata.csv', containing all the jobs of the user 'toto'. Each row of the file will be one job of the user. The columns of the CSV file will be the list of nodes assigned to the job, the number of nodes, the number of cores, the cluster name, the submission time, start time and stop time, the job ID, the job name (if any), the job type, the command executed by the job (if any) and the request made by the user.
First, on one of the frontend nodes, launch the client
psql -h oardb.grenoble.grid5000.fr -U oarreader oar2
Then, after entering the password, run the following command (change the user name and the file name if needed):
\copy (Select string_agg(Distinct host, '/') as hosts, Count(Distinct host) as nb_hosts, Count(Distinct resources.resource_id) as nb_cores,cluster,submission_time,start_time,stop_time,job_id,job_name,job_type,command,initial_request From jobs Inner Join assigned_resources on jobs.assigned_moldable_job = assigned_resources.moldable_job_id Inner Join resources on assigned_resources.resource_id = resources.resource_id Where job_user = 'toto' Group By jobs.submission_time,jobs.start_time,jobs.stop_time,jobs.job_id,jobs.job_name,jobs.job_type,jobs.command,resources.cluster) To '~/oardata.csv' With CSV;