FAQ: Difference between revisions

From Grid5000
Jump to navigation Jump to search
(47 intermediate revisions by 8 users not shown)
Line 4: Line 4:
== About this document ==
== About this document ==
=== How to add/correct an entry to the FAQ? ===
=== How to add/correct an entry to the FAQ? ===
Just like any other page of the wiki, you can edit the FAQ yourself to improve it. If you click on one of the little "edit" placed after each question, you'll get the possibility to edit that particular question. To edit the whole page, simply choose the edit tab at the top of the page.
Just like any other page of this wiki, you can edit the FAQ yourself to improve it. If you click on one of the little "edit" placed after each question, you'll get the possibility to edit that particular question. To edit the whole page, simply choose the edit tab at the top of the page.




== Publications and Grid'5000 ==
== Publications and Grid'5000 ==
=== Is there an official acknowledgement ? ===
=== Is there an official acknowledgement ? ===
Yes there is : you agreed to it when accepting the user Charter. As the charter might have been updated since, please refer to the [[Grid5000:UserCharter|latest version]]. You should use it on all publication presenting result obtained (partially) using Grid'5000.
Yes there is: you agreed to it when accepting the usage policy. As the policy might have been updated since, please refer to the [[Grid5000:UsagePolicy|latest version]]. You should use it on all publications presenting results obtained (even partially) using Grid'5000.


=== How to mention Grid'5000 in HAL  ? ===
=== How to mention Grid'5000 in HAL  ? ===
[http://hal.inria.fr HAL] is an open archive you're invited to use. If you do so, the recommended way of mentioning Grid'5000 is to use the collaboration field of submission form, with the '''Grid'5000''' keyword, capitalized as such.
[http://hal.inria.fr HAL] is an open archive you're invited to use. If you do so, the recommended way of mentioning Grid'5000 is to use the collaboration field of submission form, with the '''Grid'5000''' keyword, capitalized as such.
== Accessing Grid'5000 ==
== Accessing Grid'5000 ==
=== What is the theory ? ===
=== How can I connect to Grid'5000 ? ===
In theory, you should be able to access Grid'5000 only from the production network of the lab that is hosting the Grid'5000 site hosting your account. For off-site users, a whitelist of IPs that are allowed to access the site through the <code class="host">access.<em>site</em>.grid5000.fr</code> machine is generally maintained. You'll need ssh keys properly configured (please refer to [[SSH#SSH_Key_usage| the page dedicated to ssh]] if you don't understand these last words) as this machine should not allow you to log using a password.
This is documented at length in the [[Getting Started]] tutorial.


In practise, some sites have an <code class="host">access.<em>site</em>.grid5000.fr</code> machine reachable from any IP. You can use the [[External_access|machines with external access]] listed as accessible from anywhere as a backup.
You should be able to access Grid'5000 from anywhere on the Internet, by connecting to <code class="host">access.grid5000.fr</code> using SSH. You'll need SSH keys properly configured (please refer to [[SSH#SSH_Key_usage| the page dedicated to SSH]] if you don't understand these last words) as this machine will not allow you to log using a password.
 
Some sites have an <code class="host">access.<em>site</em>.grid5000.fr</code> machine, which is only reachable from an IP address coming from local laboratory.


=== How to connect from different workstations with the same account? ===
=== How to connect from different workstations with the same account? ===
Line 30: Line 32:


=== How to directly connect by SSH to any machine within Grid'5000 from my workstation? ===
=== How to directly connect by SSH to any machine within Grid'5000 from my workstation? ===
This tip consists of customizing SSH configuration file <code class="file">~/.ssh/config</code>. We use for this the <code class="command">nc</code> (<code class="command">tcpconnect</code> could also be used) in order to bind <code>stdin</code> and <code>stdout</code> to a network connection:
This tip consists of customizing SSH configuration file <code class="file">~/.ssh/config</code>.


  Host *.g5k
  Host *.g5k
     User <code class="replace">login</code>
     User <code class="replace">login</code>
     ProxyCommand <code class="command">ssh</code> <code class="replace">login@access</code>.grid5000.fr "<code class="command">nc  -w 60 `basename %h .g5k` %p</code>"
     ProxyCommand <code class="command">ssh</code> <code class="replace">login</code>@access.grid5000.fr -W "`basename %h .g5k`:%p"
 
Your are now able to connect to any machine using <code class="command">ssh</code> <code class="replace">machine.site</code><code>.g5k</code>


Please have a look at the [[SSH#Using_ssh_proxy_to_access_ssh_servers_.28ie_hosts.29_behind_a_firewall|SSH page]] to a deeper understanding of this proxy feature.
Please have a look at the [[SSH#Using_ssh_proxy_to_access_ssh_servers_.28ie_hosts.29_behind_a_firewall|SSH page]] to a deeper understanding of this proxy feature.


'''Note''': Grid'5000 internal network hostnames like <code class="host">*.grid5000.fr</code> are unknown outside of Grid'5000.
'''Note''': Grid'5000 internal network uses private IP addresses and are not directly reachable from outside of Grid'5000.
 
'''Warning''': <code class="command">nc</code> and <code class="command">tcpconnect</code> commands are not always available on frontends.
 
=== How to access Internet from nodes? ===
[[Security_policy|For security reason]], '''it is not possible to connect to Internet''' from inside Grid'5000, except using the  [[Web proxy client]]. However, [[SSH#Using_port_forwarding|SSH port forwarding]] is not disabled and you can use the following command, that forward connection to ''<code class="host">host</code><code>:port</code>'' when we connect to ''<code class="host">host_g5k</code><code>:port_g5k</code>'':
 
<code class="command">ssh</code> -R ''port_g5k'':''<code class="host">host</code>'':''port'' <code class="host">''host_g5k''.''site''.grid5000.fr</code>
 
{{Note|text=If your need is only to access web sites that are not allowed through the Web proxy server, you may [[Support|ask the staff]] to add them to the white listed sites.}}
 
Note that HTTP proxies are also configured to allow some TCP connections, such as outgoing connections to the port 22 (SSH). You can use <code class="command">connect-proxy</code>:
 
* Add this config to your ssh config:
 
cat <<EOF >> ~/.ssh/config
Host *.http
  ProxyCommand connect-proxy -H proxy:3128 $(basename %h .http) %p
EOF
 
* To ssh to inria's gforge for instance, just do:
ssh user@scm.gforge.inria.fr.http
 
=== How to access InriaGforge from frontends ===
 
Accessing [https://gforge.inria.fr/ INRIA's Gforge] repository directly from within Grid'5000 for checkout, commit or even synchronize your project data between Grid'5000's sites is very useful. To make it easy, every proxies and frontends are configured to allow transparent access to InriaGforge using webdav protocol with <code>https</code>. You can checkout and commit your InriaGforge <code class="replace">project</code> repository:
 
<code class="command">svn</code> checkout --username <code class="replace">gforgelogin</code> https://scm.gforge.inria.fr/svn/<code class="replace">project</code>
 
You will be asked to accept InriaGforge SSL server certificate. If you accept it permanently, you will never be bothered again.
 
If you have previously checkouted a repository using <code>svn+ssh</code>, you can easily relocate your working copy with the following commands:
 
<code class="command">cd</code> <code class="replace">working_copy</code>/
<code class="command">svn</code> switch --relocate svn+ssh://scm.gforge.inria.fr/svn/<code class="replace">project</code> https://scm.gforge.inria.fr/svn/<code class="replace">project</code> .
 
Then, all your svn commands will use the webdav protocol without the need of maintaining SSH tunnels.
 
=== How to access InriaGforge using SSH ===
 
{{Warning|text=The easiest way to access to InriaGforge is by [[FAQ#How_to_access_InriaGforge_from_frontends|using webdav protocol on frontend nodes]]. Using SSH is generally more difficult since you need to understand properly the basics of [[SSH]] and the global architecture of grid'5000 to get it working.}}
 
==== From frontends ====
 
Due to [[Security policy|security policy]], direct access to InriaGforge from inside Grid'5000 is by default prohibited. Decision to allow access to InriaGforge depends on each site. Following frontends can access InriaGforge:
frontend.nancy.grid5000.fr
frontend.rennes.grid5000.fr
frontend.toulouse.grid5000.fr
 
For other frontends, that do not allow access to InriaGforge, there is a little SSH trick. Edit your SSH configuration file <code class="file">~/.ssh/config</code> on the host used to access Grid'5000 sites (do not forget to replace <code class="replace">site</code> by targeting site, <code class="replace">g5klogin</code> by your Grid'5000 username and <code class="replace">fwdport</code> by an arbitrary port number > 1024):
Host acces.<code class="replace">site</code>.grid5000.fr
    User <code class="replace">g5klogin</code>
    RemoteForward <code class="replace">fwdport</code> scm.gforge.inria.fr:22
 
Now connect to configured host (<code class="host">localhost</code>:<code class="replace">fwdport</code> will be forwarded to <code class="host">scm.gforge.inria.fr</code>:22):
<code class="command">ssh</code> access.<code class="replace">site</code>.grid5000.fr
 
Then modify your SSH configuration file <code class="file">~/.ssh/config</code> on this Grid'5000 frontend (do not forget to replace <code class="replace">gforgelogin</code> by your InriaGforge username and <code class="replace">fwdport</code> by the previously defined forwarded port number):
Host scm.gforge.inria.fr
    Hostname localhost
    Port <code class="replace">fwdport</code>
    User <code class="replace">gforgelogin</code>
 
You can checkout and commit your InriaGforge <code class="replace">project</code> repository:
<code class="command">svn</code> co svn+ssh://scm.gforge.inria.fr/svn/<code class="replace">project</code>
 
==== From nodes ====


{{Note|text=First of all, remember that your home directory is mounted (or can be mounted, in case of a deployed node). As a consequence, you should be able to perform the <code class="command">svn</code> commands from the frontend, while actually using the files on the nodes also.}}
=== Is access to the Internet possible from nodes? ===
Since end of 2015, full Internet access is allowed from Grid'5000 network. For security reasons, connections are logged.


If you want to connect to a server (e.g. scm.gforge.inria.fr SSH server) that lives outside of Grid'5000, '''from a Grid'5000 node''', you must use both of the previous tips.
=== How can I connect to an HTTP or HTTPS service running on a node? ===


First, [[FAQ#How_to_directly_connect_by_SSH_to_any_machine_within_Grid.275000_from_my_workstation.3F|connect to the node from your workstation directly]], and create a reverse tunnel:
Network connections (tcp, udp, ...) to any Grid'5000 node from outside Grid'5000 are possible using Grid'5000 [[VPN]].
See also the [[SSH]] page (forwarding and SOCKS proxy).


ssh <code class="replace">gdx-42.orsay</code>.g5k -R <code class="replace">fwdport</code>:scm.gforge.inria.fr:22
However, if you only need to connect to a web interface or web API served by a node using HTTP or HTTPS, you can use Grid'5000 reverse proxy. The base URL to use is ''https://mynode.mysite.http.proxy.grid5000.fr/'' for HTTP and ''https://mynode.mysite.https.proxy.grid5000.fr/'' for HTTPS.


Then, [[FAQ#How_to_access_InriaGforge_from_frontends|reverse connect from the node to the server that is outside of Grid'5000 through the tunnel]], with:
If you can't run your service on port 80 or 443 (because you are not root for example), you can use a port 8080 or 8443 (you don't need to be root to bind them). You can then use ''https://mynode.mysite.http8080.proxy.grid5000.fr/'' for HTTP and ''https://mynode.mysite.https8443.proxy.grid5000.fr/'' for HTTPS.
Host scm.gforge.inria.fr
    Hostname localhost
    Port <code class="replace">fwdport</code>
    User <code class="replace">gforgelogin</code>


You can checkout and commit your InriaGforge <code class="replace">project</code> repository:
Please note that the reverse proxy needs authentication using your Grid'5000 credentials.


<code class="command">svn</code> co svn+ssh://scm.gforge.inria.fr/svn/<code class="replace">project</code>
{{Note|text=You may have to add an exception as required by you web browser to access the target web service, because of a SSL certificate mismatch.}}


== Account management ==
== Account management ==
Line 128: Line 64:
To retrieve your password, you can use [[Special:G5KChangePassword|this form]], or ask your account manager to reset it.
To retrieve your password, you can use [[Special:G5KChangePassword|this form]], or ask your account manager to reset it.


=== Why does my home directory not contain the same files on every site? ===
=== My account expired, how can I extend it? ===
Use the account management interface (''Manage account'' link in the sidebar).
 
=== Why doesn't my home directory contain the same files on every site? ===
Every site has its own file server, this is the user's responsibility to synchronize the personal data between his home directory on the different sites. You may use the <code class="command">rsync</code> command to synchronize a remote site home directory (be careful this will erase any file that are not the same as on the local home directory):
Every site has its own file server, this is the user's responsibility to synchronize the personal data between his home directory on the different sites. You may use the <code class="command">rsync</code> command to synchronize a remote site home directory (be careful this will erase any file that are not the same as on the local home directory):
  <code class="command">rsync</code> -n --delete -avz <code class="file">~</code> <code class="host">frontend.</code><code class="replace">site</code><code class="host">.grid5000.fr</code>:<code class="file">~</code>
  <code class="command">rsync</code> -n --delete -avz <code class="file">~</code> <code class="host">frontend.</code><code class="replace">site</code><code class="host">.grid5000.fr</code>:<code class="file">~</code>
Line 145: Line 84:
* the soft limit is set to what the admins find a reasonable limit for an account on a more or less permanent basis. You can use more disk space temporarily, but you should not try and trick the system to keep that data on the shared file system.  
* the soft limit is set to what the admins find a reasonable limit for an account on a more or less permanent basis. You can use more disk space temporarily, but you should not try and trick the system to keep that data on the shared file system.  
* the hard limit is set so as to preserve usability for other users if one of your scripts produces unexpected amounts of data. You'll not be able to override that limit.
* the hard limit is set so as to preserve usability for other users if one of your scripts produces unexpected amounts of data. You'll not be able to override that limit.
More information is available in the [[quotas]] page.


=== How to increase my quota disk limitation? ===
 
=== How to increase my disk quota limitation? ===
Should you need higher quotas, please visit your user account settings page at https://api.grid5000.fr/ui/account (my storage tab), or send an email to [mailto:support-staff@lists.grid5000.fr support-staff@lists.grid5000.fr], explaining how much storage space you want and what for.
Should you need higher quotas, please visit your user account settings page at https://api.grid5000.fr/ui/account (my storage tab), or send an email to [mailto:support-staff@lists.grid5000.fr support-staff@lists.grid5000.fr], explaining how much storage space you want and what for.


== SSH related questions ==
=== How do I unsubscribe from the mailing-list ? ===
 
Users' mailing-list subscription is tied to your Grid'5000 account.  You can configure your subscriptions in your account settings:


=== How to fetch all the SSH host keys of one site? ===
[[File:Grid5000-unsubscribe-mailing-list.png|thumb|How to unsubscribe from the mailing list]]
To avoid answering 'yes' when connecting with SSH for the first time to hosts, the <code class="file">~/.ssh/known_hosts</code> file can be automatically generated for one ''site'':
* Login to https://api.grid5000.fr/ui/account
<code class="command">nodelist</code> <code class="replace">site</code> | <code class="command">ssh-keyscan</code> -tdsa,rsa  -f -
* Go to the "My account" tab, then click on the "Actions" button, then choose "Manage mailing lists"


Please have a look at "[[#How_to_get_a_site_list_of_nodes_.3F|How to get a site list of nodes?]]", for information on the <code>nodelist</code> command.
Alternate method, by configuring Sympa to stop receiving any email from the list (while still being subscribed):


{{Warning|text=This hint is deprecated with OAR2 and oarsh (but still worths the read if using ssh thru the ''allow_classic_ssh'' mode). [[Support|Report to the staff]] if needed.}}
* If you haven't done it before, ask for a password on ''sympa.inria.fr'' from this form: https://sympa.inria.fr/sympa/firstpasswd/. '''Use the email address you used to register to Grid'5000'''.
* Connect to https://sympa.inria.fr using your email address you used to register to Grid'5000 and your ''sympa.inria.fr'' password.
* From the left panel, select ''users_grid5000''. Then go to your subscriber options (''Options d'abonné'') and in the ''reception'' field (''Mode de réception''), select ''suspend'' (''interrompre la réception des messages'').
 
== SSH related questions ==


=== How to avoid SSH host key checking? ===
=== How to avoid SSH host key checking? ===
Line 207: Line 152:


To convert ssh public keys from SSH2 to OpenSSH, see [http://burnz.wordpress.com/2007/12/14/ssh-convert-openssh-to-ssh2-and-vise-versa/ this tutorial].
To convert ssh public keys from SSH2 to OpenSSH, see [http://burnz.wordpress.com/2007/12/14/ssh-convert-openssh-to-ssh2-and-vise-versa/ this tutorial].
== Experiments issues ==
=== How-to add an http site in proxy white-list ? ===
Internet is not accessible directly from Grid'5000, the queries go through an http proxy.
However, you can request the addition of a site in the white list. For that see [[Web_proxy_access#How to ask for a new common access.3F|WebProxy Request]]
=== Why and How to fill a experiment report ===
You should document the experiments that you are doing on [[Special:G5KReports|your user report]]. This way, you can help the team in charge of the instrument to document the interest of the platform.


== Software installation issues ==
== Software installation issues ==
=== What is the general philosophy ? ===
=== What is the general philosophy ? ===
This is how things should work : a minimal set of software is installed on the frontends and nodes of each site. You should find the same version on the frontend and on the nodes of one site, but no effort is made for all sites to use coherent versions. If you need some other software package, you should should create a Kadeploy image including it, as you have root access on deployed images. The crux of the subject is that it seems impossible to install all the software users could need in different versions that are sometimes conflicting, therefore we don't even try.
This is how things should work: a basic set of software is installed on the frontends and nodes' standard environment of each site. If you need some other software packages on nodes, you can create a Kadeploy image including then, and deploy it. You can also use at sudo-g5k. If you think those software should be installed by default, you can contact the [[Support|support-staff]].
 
=== What are the exceptions ? ===
* For editing, file management and scripting software that could be useful for all users, admins often accept requests to install additional packages available for the distribution used on the frontend machine
* For specific libraries or compilers some sites have a secondary frontend where you can compile and where we will be happy to install additional packages available for the distribution. Other sites have a specific file hierarchy where users can install software for the benefit of all.
=== Are any evolutions to this general philosophy planned ?===
Yes, but don't expect to see big changes before 2009, were we hope to try to see how we could make the same version of some of the usual software available on all sites. A standard compilation frontend could also be put into production on all sites by the end of 2008.


== Deployment related issues ==  
== Deployment related issues ==  
=== Deployments seem to fail for unknown reasons ===
From time to time, kadeploy gives me the following message for some of my nodes:
paravent-25.rennes.grid5000.fr error not there on last check
These errors are annoying. They happen when at one point in the deployment, the specified node did not come up as expected by Kadeploy, meaning that no ssh deamon could be reached before the cluster-wide timeout. This happens for different reasons
* the node did not reboot. This happens because remote management software is not dependable at the scale of a cluster. Very few vendors plan for it to be used many times each day, therefore the failure rate isn't to good on some hardware.
* the node did not boot correctly or fast enough. Maybe the DHCP request took more time than usual or was lost, or that the tftp server sending the kernel to boot is slow because of local network traffic or of users using the frontend as a compilation machine or as a head node for their experiment.
* the node did copy the environment to disk fast enough. If the disk is slow for some reason, temporary or permanently, this will happen. Kadeploy v2.x series does not report this as a specific error, and therefore it can take time before the failure is identified and corrected. 
* the environment did not boot correctly or did not start its ssh server fast enough because of configuration scripts
You can do 3 things to handle these failures:
* use a wrapper script such as katapult3 (available on every sites) to retry a deployment as long as necessary
* manually check the real status of the node using <code class="command">kaconsole -m node-XX.site.grid5000.fr</code>
* report failing nodes to the admins or suggest an increase in the timeout if a manual check reports hardware failures or correct deployment despite report by Kadeploy


=== My environment does not work on all clusters ===
=== My environment does not work on all clusters ===
There are a few reasons why a working environment is not always portable:
It some rare occasion, an environment may not work on a given cluster:
# The kernel used does not support all hardware. You are advised to base your environment on one of the reference environments to avoid dealing with this, or to carefully read the hardware section of each site to see the list of kernel drivers that need to be compiled in your environment for it to be able to boot on all clusters. Of course, when a new cluster is integrated, you might need to update your kernel for portability.
# The kernel used does not support all hardware. You are advised to base your environment on one of the reference environments to avoid dealing with this, or to carefully read the hardware section of each site to see the list of kernel drivers that need to be compiled in your environment for it to be able to boot on all clusters. Of course, when a new cluster is integrated, you might need to update your kernel for portability.
# The boot method used on all sites is not the same. To make a long story short, it is not possible to use grub on all sites, and sites without grub can't boot some environments where the initrd is a symbolic link. Before kadeploy 2.1.7, sites without grub could also not boot Xen environments. Here is a list of the clusters where Grub is enabled (and working) in kadeploy boot process (as of 2008-01-17) :
# The post-installation scripts do not recognize your environment, and therefore network access, console access or site specific configurations are not taken into account. You can check the contents of the default post-installation scripts to see the variables set by kadeploy by looking at environment's description using kaenv.
#* azur (sophia)
 
#* bordemer (bordeaux)
#* gdx (orsay)
# The pre and post-installation scripts do not recognize your environment, and therefore network access, console access or site specific configurations are not taken into account. Here, you can check the contents of the default pre and post-installation scripts to see the variables set by kadeploy to enable customization at post-install stage
=== Kadeploy fails with ''Image file not found!'' ===
=== Kadeploy fails with ''Image file not found!'' ===
This means that <code>kadeploy</code> is not able to read your environment's main archive. This can be caused by many reasons:
This means that <code>kadeploy</code> is not able to read your environment's main archive. This can be caused by many reasons, i.e:
* registered filename is wrong, this can be verified retrieving the registered information with <code>kaenvironments -e environment_name</code>
* registered filename is wrong
* extension is not right (for example <code>.tar.gz</code> does not work, whereas <code>.tgz</code> is OK)
* extension is not right (for example <code>.tar.gz</code> does not work, whereas <code>.tgz</code> is OK)
* the directory rights are not good: <code>kadeploy</code> reads this file as the <code>deploy</code> user, so it has not the same access permission as yours: everyone should be able to read the files implied in an environment (ie: main archive and postinstall).
=== Some nodes fail with the message ''Not there on last check!'' ===
This means that the node was deployed, but a problem occured during the last phase of the deployment process: boot on the deployed partition.
A lot of different causes can lead to this state.
The first thing is to check the failure rate: if all the target nodes have the same deployment problem, you can be sure that there is a problem with your environment, assuming that the number of target nodes is significant.
In this case, you should check if your environment follows all the minimum [[Environments requirements]] for Grid5000.
<code>kadeploy</code>'s final check for deployment is to check if a ssh server is running on the target node. You have to check with <code>kaconsole</code>, in this order:
* network configuration: if the network is not configured, it won't work (a <code>ping</code> is enough to check this problem)
* ssh server startup: ssh performs a lot of various verifications before launching the service, and only one problem is enough to prevent it from working. In any case, you should try to launch it on the partially deployed node with <code>kaconsole</code> to get a good hint to solve this issue:
gdx-174:~# /etc/init.d/ssh start
Starting OpenBSD Secure Shell server: sshd/var/run/sshd must be owned by root and not group or
world-writable.
and effectively:
gdx-174:~# ls -ld /var/run/sshd/
drwxrwxrwx  2 root root 4096 Oct  7  2004 /var/run/sshd/


=== Kadeploy is complaining about a node already involved in an other deployment===
=== Kadeploy is complaining about a node already involved in an other deployment===
Line 277: Line 175:
* when 2 concurrent deployments are attempted on the same node. If you have 2 simultaneous deployments, make sure you have 2 distinct sets of nodes.
* when 2 concurrent deployments are attempted on the same node. If you have 2 simultaneous deployments, make sure you have 2 distinct sets of nodes.
* when there is a problem in the kadeploy database: typically when a deployment ended in a strange way, this can happen. The best is to wait for about 15 minutes and retry the deployment: kadeploy can correct its database automatically.
* when there is a problem in the kadeploy database: typically when a deployment ended in a strange way, this can happen. The best is to wait for about 15 minutes and retry the deployment: kadeploy can correct its database automatically.
=== How to quickly check for nodes health ===
You can check for nodes health, based on ICMP request, with <code class="command">nmap</code> command if it is available on a site:
<code class="command">nodelist</code> <code class="replace">site</code> | <code class="command">nmap</code> -iL - -sP
Or with <code class="command">fping</code> command, if it is available:
<code class="command">nodelist</code> <code class="replace">site</code> | <code class="command">fping</code> -a 2> /dev/null
(See [[TakTuk]]).


=== How to kill all my processes on a host ? ===
=== How to kill all my processes on a host ? ===
Line 291: Line 181:


=== How do I exit from kaconsole on cluster X from site Y ===
=== How do I exit from kaconsole on cluster X from site Y ===
Depending on the hardware (cluster) your are doing the kaconsole, the exit sequence may differ.
You can try '&.' sequence (french keyboard), but this may not work on all clusters.The [[Kaconsole#Escape_sequence_for_every_site|Kaconsole]] page may give you more information.
The [[Kaconsole#Escape_sequence_for_every_site|Kaconsole]] page gives sequences for every Grid'5000 clusters.


=== Why are the network interfaces named eth2,eth3...eth''n'' in my deployed environment? ===
=== Why are the network interfaces named eth2,eth3...eth''n'' in my deployed environment? ===
Line 313: Line 202:


By default, no checkpointing or automatic restart of besteffort jobs is provided. They are just killed. That is why this mode  
By default, no checkpointing or automatic restart of besteffort jobs is provided. They are just killed. That is why this mode  
is best used with a tool which can detect the killed jobs and resubmit them. However OAR2 provides options for that. You may also have a look at tools like [http://cigri.ujf-grenoble.fr/ CIGRI] or [http://grail.sdsc.edu/projects/apst/ APST].
is best used with a tool which can detect the killed jobs and resubmit them. However OAR2 provides options for that. You may also have a look at tools like CiGri.
 
=== Why are nodes suspected after attempted kill of best-effort jobs ? ===
This is a recurring issue on Grid'5000, Bugs [https://www.grid5000.fr/cgi-bin/bugzilla3/show_bug.cgi?id=1355 1355] [https://www.grid5000.fr/cgi-bin/bugzilla3/show_bug.cgi?id=2362 2362] is the first known trace of this, and a generic bug ([https://www.grid5000.fr/cgi-bin/bugzilla3/show_bug.cgi?id=3072 3072]) has been opened to track occurrences of this behaviour.
 
The scenario is the following:
# a user requests resource, and his request can only be fulfilled by killing best-effort jobs
# after best-effort jobs are killed, some nodes become suspected instead of coming online to fulfil the request. What happens here is
## oar sends a kill request to the best-effort jobs
## oar timeouts on the kill request for some nodes
## oar suspects the node
# the initial requesting user has his request either trimmed (case of a reservation) or rejected
## In the case of a reservation, oar waits for absent or suspected node to come up for 5 minutes before starting the job with a reduced number of resources
# phoenix, a script running every 15mn to attempt to automatically repair suspected nodes puts the node back online, maybe thanks to a hard reboot
# the script running the best-effort jobs finds this new resource, and schedules a new best-effort job on it, to the intense frustration of the initial user
Our best guess on the reason for step 2.2 is that some best-effort jobs trash the resources so hard that time for clean-up is longer than the timeout. Waiting for a reproducible but simple use-case to be able to investigate further. The problem is that inspecting a node in such a state to understand how we could handle this better is near to impossible


=== How to pass arguments to my script ===
=== How to pass arguments to my script ===
Line 341: Line 215:
Jobs with type <code>deploy</code> or type <code>allow_classic_ssh</code> imply the exclusive usage of a node. Therefore, specifying core information for your submission can only lead to some inconsistencies. It is therefore prohibited by an admission rule.
Jobs with type <code>deploy</code> or type <code>allow_classic_ssh</code> imply the exclusive usage of a node. Therefore, specifying core information for your submission can only lead to some inconsistencies. It is therefore prohibited by an admission rule.


=== Why did my advance reservation started with not all the resources I requested ? ===
=== Why did my advance reservation start with less than all the resources I requested ? ===
Since OAR 2.2.12, an advance reservation is validated regardless of the state of resources being either:
Since OAR 2.2.12, an advance reservation is validated regardless of the state of resources being either:
# ''alive''  
# ''alive''  
Line 348: Line 222:
(but not ''dead'') at the time the reservation is required to start and during the panned walltime (because those states are transitional).  
(but not ''dead'') at the time the reservation is required to start and during the panned walltime (because those states are transitional).  


Moreover, resources allocated to an advance reservation are definitely fixed upon this validation, which means that if any of those resources becomes ''absent'' or ''suspected'' after the validation, that resource won't be replaced.  
Moreover, resources allocated to an advance reservation are definitely fixed upon this validation, which means that if any of those resources becomes ''dead'', ''absent'' or ''suspected'' after the validation, that resource won't be replaced.  


At the start time of the advance reservation then, OAR looks after any unavailable resources (''absent'' or ''suspected''), and whenever some exists, wait for them to return for 5 minutes, shall it append:
At the start time of the advance reservation then, OAR looks after any unavailable resources (''absent'' or ''suspected''), and whenever some exists, wait for them to return for 5 minutes, shall it append:
Line 360: Line 234:
Information about reduced number of resources or reduced walltime for a reservation due to this mechanism are available in the event part of the output of
Information about reduced number of resources or reduced walltime for a reservation due to this mechanism are available in the event part of the output of
  <code class='command'>oarstat -fj </code><code class='replace'>jobid</code>
  <code class='command'>oarstat -fj </code><code class='replace'>jobid</code>
=== How can I check whether my reservations are respecting the Grid'5000 Usage Policy ? ===
You can use the script <code>usagepolicycheck</code>, present on all frontends. See if your current reservations are respecting the Policy with <code>usagepolicycheck -t</code>, use <code>usagepolicycheck -h</code> to see the other options.
== Access to logs ==
== Access to logs ==
=== OAR logs ===
=== OAR database logs ===
A little known feature in Grid'5000 is the possibility for all users to use a read only access to oar's database. You should be able to connect as user <code>oarreader</code> with password <code>read</code> to database <code>oar2</code> on all <code class=host>mysql.</code><code class=replace>site</code><code class=host>.grid5000.fr</code>. This gives you access to the complete history of jobs on all Grid'5000 sites. This gives you access to the production database of OAR: please be careful with your queries !!!
Grid'5000 gives the possibility to all users to use a read only access to OAR's database. You should be able to connect using PostgresSQL client as user <code>oarreader</code> with password <code>read</code> to database <code>oar2</code> on all <code class=host>oardb.</code><code class=replace>site</code><code class=host>.grid5000.fr</code>. This gives you access to the complete history of jobs on all Grid'5000 sites. This gives you read-only access to the production database of OAR: please be careful with your queries to avoid overloading the testbed!


= About OAR 2.4 =
{{Note|text=Careful: Grid'5000 is not a computation grid, nor HPC center, nor a Cloud (Grid'5000 is a research instrument). That means that the usage of Grid'5000 by itself (OAR logs of Grid'5000 users' reservations) does not reflect a typical usage of any such infrastructure. It is therefore not relevant to analyze Grid'5000 OAR logs to that purpose. As a user, one can however use Grid'5000 to emulate a HPC cluster or cloud on reserved resources (in a job), possibly injecting a real load from a real infrastructure.}}
 
= About OAR =
== How to know if a node is in energy saving mode or really absent ? ==
== How to know if a node is in energy saving mode or really absent ? ==


Line 371: Line 251:
Nodes in this state will be automatically started by OAR when it will be needed.
Nodes in this state will be automatically started by OAR when it will be needed.


Advanced users who check directly the OAR database can determine if a node is in energy saving mode or  absent with the field "available_upto" in the resources table (In versions older than OAR 2.4, the field "available_upto" is named "cm_availability").<br>
Advanced users who check directly the OAR database can determine if a node is in energy saving mode or  absent with the field "available_upto" in the resources table.<br>
If energy saving is enabled on the cluster, the field "available_upto" provides a date (unix timestamp) until when the resource will be available.
If energy saving is enabled on the cluster, the field "available_upto" provides a date (unix timestamp) until when the resource will be available.


Line 384: Line 264:


An example of SQL query listing really absent nodes:
An example of SQL query listing really absent nodes:
  SELECT distinct(network_address) FROM resources WHERE state="Absent" AND (available_upto < UNIX_TIMESTAMP() OR available_upto = 0)
  SELECT distinct(network_address) FROM resources WHERE state='Absent' AND (available_upto < UNIX_TIMESTAMP() OR available_upto = 0)


== How to detect nodes in maintenance ? ==
== How to detect nodes in maintenance ? ==
Nodes in maintenance are nodes with a really absent state.<br>
Nodes in maintenance are nodes with a ''Dead'' state, a OAR ''maintenance'' property set to ''YES'', or temporary in a really absent state (see above).
See above: ''How to know if a node is in energy saving mode or really absent ?''


== How to execute jobs within another one ? ==
== How to execute jobs within another one ? ==
This functionality is named ''container jobs'' (available only since OAR 2.3).<br>
This functionality is named ''container jobs''.
With this functionality it is possible to execute jobs within another one. So it is like a sub-scheduling mechanism.
With this functionality it is possible to execute jobs within another one. So it is like a sub-scheduling mechanism.


* First a job of the type container must be submitted, for example:
* First a job of the type container must be submitted, for example:
  oarsub -I -t container -l nodes=10,walltime=2:10:00
  oarsub -I -t container -t cosystem -l nodes=10,walltime=2:10:00
  ...
  ...
  OAR_JOB_ID=13542
  OAR_JOB_ID=13542
Line 403: Line 282:
  oarsub -I -t inner=13542 -l nodes=7,walltime=0:30:00
  oarsub -I -t inner=13542 -l nodes=7,walltime=0:30:00
  oarsub -I -t inner=13542 -l nodes=3,walltime=0:45:00
  oarsub -I -t inner=13542 -l nodes=3,walltime=0:45:00
More information in [[Advanced_OAR#Container_jobs]]


= How to use MPI in Grid5000? =
= How to use MPI in Grid5000? =
Line 408: Line 289:
See also : [[Run_MPI_On_Grid'5000|The MPI Tutorial]]
See also : [[Run_MPI_On_Grid'5000|The MPI Tutorial]]


== MPI Warnings  ==
On many clusters, it is likely to encounter the following warnings. These warnings may come from collisions between INFINIBAND and MYRINET technologies <br>
<pre>
*  1 -  mca: base: component_find: unable to open /usr/lib/openmpi/mca_mtl_mx: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
*  2 -  Error in mx_open_endpoint (error Busy)
*  3 -  librdmacm: couldn't read ABI version.
*  4 -  Signal: Segmentation fault (11)
*  5 -  Error in mx_init (error No MX device entry in /dev.)
</pre>


== MPI options to use ==
== MPI options to use ==
It is preferable to use the following options to avoid warnings or errors:<br>
It is preferable to use the following options to avoid warnings or errors:<br>


* INFINIBAND clusters: Parapluie, Parapide, Griffon, Graphene, Edel, Adonis, Genepi
* INFINIBAND clusters: Parapluie, Parapide, Griffon, Graphene, Edel, Genepi
 
{{Term|location=node|cmd=<code class="command"> mpirun --mca plm_rsh_agent "oarsh" --mca btl openib,sm,self --mca pml ^cm -machinefile $OAR_NODEFILE  $HOME/mpi_programm</code>}}
 
<br>
 
* MYRINET clusters:
#*suno, chinqchint
{{Term|location=node|cmd=<code class="command"> mpirun --mca plm_rsh_agent "oarsh" --mca pml ob1 --mca btl tcp,self -machinefile $OAR_NODEFILE  $HOME/mpi_programm</code>}}
<br>
#*Sol
{{Term|location=node|cmd=<code class="command"> mpirun --mca plm_rsh_agent "oarsh" --mca pml cm  -machinefile $OAR_NODEFILE  $HOME/mpi_programm</code>}}
<br>


<pre>
{{Term|location=node|cmd=<code class="command"> mpirun --mca orte_rsh_agent "oarsh" --mca btl openib,sm,self --mca pml ^cm -machinefile $OAR_NODEFILE  $HOME/mpi_programm</code>}}
For other clusters, you may use the following options:


* --mca btl ^mx
* For other clusters, you may use the following options:
* --mca pml ob1 --mca btl tcp,self
** --mca pml ob1 --mca btl tcp,self
* --mca btl ^openib
** --mca btl ^openib
</pre>
** --mca btl ^mx

Revision as of 11:51, 17 April 2019

About this document

How to add/correct an entry to the FAQ?

Just like any other page of this wiki, you can edit the FAQ yourself to improve it. If you click on one of the little "edit" placed after each question, you'll get the possibility to edit that particular question. To edit the whole page, simply choose the edit tab at the top of the page.


Publications and Grid'5000

Is there an official acknowledgement ?

Yes there is: you agreed to it when accepting the usage policy. As the policy might have been updated since, please refer to the latest version. You should use it on all publications presenting results obtained (even partially) using Grid'5000.

How to mention Grid'5000 in HAL  ?

HAL is an open archive you're invited to use. If you do so, the recommended way of mentioning Grid'5000 is to use the collaboration field of submission form, with the Grid'5000 keyword, capitalized as such.

Accessing Grid'5000

How can I connect to Grid'5000 ?

This is documented at length in the Getting Started tutorial.

You should be able to access Grid'5000 from anywhere on the Internet, by connecting to access.grid5000.fr using SSH. You'll need SSH keys properly configured (please refer to the page dedicated to SSH if you don't understand these last words) as this machine will not allow you to log using a password.

Some sites have an access.site.grid5000.fr machine, which is only reachable from an IP address coming from local laboratory.

How to connect from different workstations with the same account?

You can associate multiple public SSH keys to your account. In order to do so, you have to:

  • login
  • go to User Portal > Manage Account
  • select the 'My account' tab
  • in actions list select «Edit Profil»
  • then, paste the public SSH key in a new line just after the other(s).

More information about SSH and Public key authentication.

How to directly connect by SSH to any machine within Grid'5000 from my workstation?

This tip consists of customizing SSH configuration file ~/.ssh/config.

Host *.g5k
   User login
   ProxyCommand ssh login@access.grid5000.fr -W "`basename %h .g5k`:%p"

Your are now able to connect to any machine using ssh machine.site.g5k

Please have a look at the SSH page to a deeper understanding of this proxy feature.

Note: Grid'5000 internal network uses private IP addresses and are not directly reachable from outside of Grid'5000.

Is access to the Internet possible from nodes?

Since end of 2015, full Internet access is allowed from Grid'5000 network. For security reasons, connections are logged.

How can I connect to an HTTP or HTTPS service running on a node?

Network connections (tcp, udp, ...) to any Grid'5000 node from outside Grid'5000 are possible using Grid'5000 VPN. See also the SSH page (forwarding and SOCKS proxy).

However, if you only need to connect to a web interface or web API served by a node using HTTP or HTTPS, you can use Grid'5000 reverse proxy. The base URL to use is https://mynode.mysite.http.proxy.grid5000.fr/ for HTTP and https://mynode.mysite.https.proxy.grid5000.fr/ for HTTPS.

If you can't run your service on port 80 or 443 (because you are not root for example), you can use a port 8080 or 8443 (you don't need to be root to bind them). You can then use https://mynode.mysite.http8080.proxy.grid5000.fr/ for HTTP and https://mynode.mysite.https8443.proxy.grid5000.fr/ for HTTPS.

Please note that the reverse proxy needs authentication using your Grid'5000 credentials.

Note.png Note

You may have to add an exception as required by you web browser to access the target web service, because of a SSL certificate mismatch.

Account management

I forgot my password, how can I retrieve it ?

To retrieve your password, you can use this form, or ask your account manager to reset it.

My account expired, how can I extend it?

Use the account management interface (Manage account link in the sidebar).

Why doesn't my home directory contain the same files on every site?

Every site has its own file server, this is the user's responsibility to synchronize the personal data between his home directory on the different sites. You may use the rsync command to synchronize a remote site home directory (be careful this will erase any file that are not the same as on the local home directory):

rsync -n --delete -avz ~ frontend.site.grid5000.fr:~

NB : please remove the -n argument once you are sure you actually don't want to do a dry-run only...;)

How to get my home mounted on deployed nodes?

This is completely automatic if you deploy a *-nfs or *-big image. You can then connect using your own login, and once connected into the node, just enter your home:

 cd /home/<your login>

How to restore a wrongly deleted file?

No backup facility is provided by Grid'5000 platform. Please watch your fingers and do backup your data using external backup services.

What about disk quotas ?

You'll find that for each account and each site, disk quotas may be activated.

  • the soft limit is set to what the admins find a reasonable limit for an account on a more or less permanent basis. You can use more disk space temporarily, but you should not try and trick the system to keep that data on the shared file system.
  • the hard limit is set so as to preserve usability for other users if one of your scripts produces unexpected amounts of data. You'll not be able to override that limit.


How to increase my disk quota limitation?

Should you need higher quotas, please visit your user account settings page at https://api.grid5000.fr/ui/account (my storage tab), or send an email to support-staff@lists.grid5000.fr, explaining how much storage space you want and what for.

How do I unsubscribe from the mailing-list ?

Users' mailing-list subscription is tied to your Grid'5000 account. You can configure your subscriptions in your account settings:

How to unsubscribe from the mailing list

Alternate method, by configuring Sympa to stop receiving any email from the list (while still being subscribed):

  • If you haven't done it before, ask for a password on sympa.inria.fr from this form: https://sympa.inria.fr/sympa/firstpasswd/. Use the email address you used to register to Grid'5000.
  • Connect to https://sympa.inria.fr using your email address you used to register to Grid'5000 and your sympa.inria.fr password.
  • From the left panel, select users_grid5000. Then go to your subscriber options (Options d'abonné) and in the reception field (Mode de réception), select suspend (interrompre la réception des messages).

SSH related questions

How to avoid SSH host key checking?

With the StrictHostKeyChecking option, SSH host key checking can be turned off. This option can be set in the ~/.ssh/config file:

StrictHostKeyChecking no

Or it can be passed on the command line:

ssh -o StrictHostKeyChecking=no host

How not to get tons of SSH errors about Man-in-the-middle attacks while deploying images ?

If you get the following error when you try to connect a machine using ssh:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
6e:44:89:6d:ac:fc:d8:84:fd:2b:fb:22:e5:ba:5c:88.
Please contact your system administrator.

This is because SSH get worried by the fact that the machine answering to the connection is not the same from run to run. This is actually really logical if you just redeployed the image so it is not same system that is answering...

Technically speaking, the file /etc/ssh/ssh_host_dsa_key.pub is likely to be different in your own deployed image and in the default image. SSH will thus freak out since such replacement usually denote that someone is intercepting the communication and pretend to be the server to get informations from you.

If you don't want to care about this issues, there are several solutions:

  • Add StrictHostKeyChecking=no to your .ssh/config file to explain SSH to ignore about those errors.
  • Pass this option (StrictHostKeyChecking=no) on the command line to ssh (using -o)
  • Make sure that you have the same host_dsa_key in your own images than in defaults one. They can usually be found in the pre/post install scripts of your site.

Outside of Grid'5000 scope, the correct solution is to fix your ~/.ssh/known_hosts, either by hand or using the command ssh-keygen -R hostname.

Please have a look at the SSH page also.

What kind of public keys are supported on Grid'5000 ?

The only format of the public_keys allowed in Grid'5000 is the openSSH format.
You MUST provide and use ssh public keys in this format.

  • SSH2 like public key (NOT SUPPORTED) :
---- BEGIN SSH2 PUBLIC KEY ----
Comment: rsa-key-20090623
AAAAB3NzaC1yc2EAAAABJQAAAIEA1YO87ubDgjQmCEdyX98UZ1RaBNAEXNGUNX2t
D/lEw7MPShJKpVYpcj4JhrOqTc0QXIcLqefkucDaoAIlEAp7e5aShWhWFtYR5Mwn
qAF1hrMBMF0xJIqgZjUWUPxvvFVeQXkObUWQkRyj5AjlG9+qQDLOoD9GgBOqfLDV
edGCLoM=
---- END SSH2 PUBLIC KEY ----
  • OpenSSH like public key :
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEA1YO87ubDgjQmCEdyX98UZ1RaBNAEXNGUNX2tD/lEw7MPShJKpVYpcj4JhrOqTc0QXIcLqefkucDaoAIlEAp7e5aShWhWFtYR5MwnqAF1hrMBMF0xJIqgZjUWUPxvvFVeQXkObUWQkRyj5AjlG9+qQDLOoD9GgBOqfLDVedGCLoM=

To convert ssh public keys from SSH2 to OpenSSH, see this tutorial.

Software installation issues

What is the general philosophy ?

This is how things should work: a basic set of software is installed on the frontends and nodes' standard environment of each site. If you need some other software packages on nodes, you can create a Kadeploy image including then, and deploy it. You can also use at sudo-g5k. If you think those software should be installed by default, you can contact the support-staff.

Deployment related issues

My environment does not work on all clusters

It some rare occasion, an environment may not work on a given cluster:

  1. The kernel used does not support all hardware. You are advised to base your environment on one of the reference environments to avoid dealing with this, or to carefully read the hardware section of each site to see the list of kernel drivers that need to be compiled in your environment for it to be able to boot on all clusters. Of course, when a new cluster is integrated, you might need to update your kernel for portability.
  2. The post-installation scripts do not recognize your environment, and therefore network access, console access or site specific configurations are not taken into account. You can check the contents of the default post-installation scripts to see the variables set by kadeploy by looking at environment's description using kaenv.

Kadeploy fails with Image file not found!

This means that kadeploy is not able to read your environment's main archive. This can be caused by many reasons, i.e:

  • registered filename is wrong
  • extension is not right (for example .tar.gz does not work, whereas .tgz is OK)

Kadeploy is complaining about a node already involved in an other deployment

The waring you see is

node $node is already involved in another deployment

This error occurs

  • when 2 concurrent deployments are attempted on the same node. If you have 2 simultaneous deployments, make sure you have 2 distinct sets of nodes.
  • when there is a problem in the kadeploy database: typically when a deployment ended in a strange way, this can happen. The best is to wait for about 15 minutes and retry the deployment: kadeploy can correct its database automatically.

How to kill all my processes on a host ?

On the currently connected host (warning, it will disconnect you)

kill -KILL -1

How do I exit from kaconsole on cluster X from site Y

You can try '&.' sequence (french keyboard), but this may not work on all clusters.The Kaconsole page may give you more information.

Why are the network interfaces named eth2,eth3...ethn in my deployed environment?

This should be due to default udev rules on Debian based systems which allocate unique interface names to physical network devices. When you deploy an environment on an other node, it will detect new physical network devices and allocate them the next available interface names, incrementing it each time. Delete the appropriate rules in your environment to prevent udev from having this behaviour:

Terminal.png node:
rm /etc/udev/rules.d/*persistent-net.rules

Job submission related issues

What is the so called "best-effort" mode of OAR?

The best-effort was implemented to back-fill the cluster with jobs considered as less important without blocking "regular" jobs. To submit jobs under that policy, you simply have to select the besteffort type of job in your oarsub command.

oarsub -t besteffort script_to_launch

Jobs submitted that way will only get scheduled on processes when no other job use them (any regular job overtake besteffort jobs in the waiting queue, regardless of submission times). Moreover, these jobs are killed (as if oardel were called) when a regular job recently submitted needs the nodes used by a besteffort job.

By default, no checkpointing or automatic restart of besteffort jobs is provided. They are just killed. That is why this mode is best used with a tool which can detect the killed jobs and resubmit them. However OAR2 provides options for that. You may also have a look at tools like CiGri.

How to pass arguments to my script

When you do passive submission through oarsub, you must specify a script. This script can be a simple script name or a more complex command line with arguments.

To pass arguments, you have to quote the whole command line, like in the following example:

oarsub -l nodes=4,walltime=2 "/path/to/myscript arg1 arg2 arg3"

Note: to avoid random code injection, oarsub allows only alphanumeric characters ([a-zA-Z0-9_]), whitespace characters ([ \t\n\r\f\v]) and few others ([/.-]) inside its command line argument.

Why are /core and -t deploy or -t use_classic_ssh incompatible ?

Jobs with type deploy or type allow_classic_ssh imply the exclusive usage of a node. Therefore, specifying core information for your submission can only lead to some inconsistencies. It is therefore prohibited by an admission rule.

Why did my advance reservation start with less than all the resources I requested ?

Since OAR 2.2.12, an advance reservation is validated regardless of the state of resources being either:

  1. alive
  2. suspended
  3. absent

(but not dead) at the time the reservation is required to start and during the panned walltime (because those states are transitional).

Moreover, resources allocated to an advance reservation are definitely fixed upon this validation, which means that if any of those resources becomes dead, absent or suspected after the validation, that resource won't be replaced.

At the start time of the advance reservation then, OAR looks after any unavailable resources (absent or suspected), and whenever some exists, wait for them to return for 5 minutes, shall it append:

  • resource are in the absent state during the reboot after a kadeploy job, and then become alive again as soon as the boot complete
  • resource which good health is suspected by OAR might be fixed back by an admin or maintenance tool operation

If resources are not back yet at the time the job actually starts, these resources are lost for the job, which then provides less resources than expected indeed.

That is a price to pay for using advance reservation.

NB

Information about reduced number of resources or reduced walltime for a reservation due to this mechanism are available in the event part of the output of

oarstat -fj jobid

How can I check whether my reservations are respecting the Grid'5000 Usage Policy ?

You can use the script usagepolicycheck, present on all frontends. See if your current reservations are respecting the Policy with usagepolicycheck -t, use usagepolicycheck -h to see the other options.

Access to logs

OAR database logs

Grid'5000 gives the possibility to all users to use a read only access to OAR's database. You should be able to connect using PostgresSQL client as user oarreader with password read to database oar2 on all oardb.site.grid5000.fr. This gives you access to the complete history of jobs on all Grid'5000 sites. This gives you read-only access to the production database of OAR: please be careful with your queries to avoid overloading the testbed!

Note.png Note

Careful: Grid'5000 is not a computation grid, nor HPC center, nor a Cloud (Grid'5000 is a research instrument). That means that the usage of Grid'5000 by itself (OAR logs of Grid'5000 users' reservations) does not reflect a typical usage of any such infrastructure. It is therefore not relevant to analyze Grid'5000 OAR logs to that purpose. As a user, one can however use Grid'5000 to emulate a HPC cluster or cloud on reserved resources (in a job), possibly injecting a real load from a real infrastructure.

About OAR

How to know if a node is in energy saving mode or really absent ?

Nodes in energy saving mode are displayed with the state "Absent (standby)" by the oarnodes command.
The state "Absent (standby)" means that the node is shut down in order to save energy.
Nodes in this state will be automatically started by OAR when it will be needed.

Advanced users who check directly the OAR database can determine if a node is in energy saving mode or absent with the field "available_upto" in the resources table.
If energy saving is enabled on the cluster, the field "available_upto" provides a date (unix timestamp) until when the resource will be available.

  • A node "Absent" is in energy saving mode if the field "available_upto" is greater than the current unix timestamp

An example of SQL query listing absent nodes because of the energy saving mode:

SELECT distinct(network_address) FROM resources WHERE state="Absent" AND available_upto >= UNIX_TIMESTAMP()
  • A node "Absent" is really absent if :

- the field "available_upto" is equal to 0
- or the field "available_upto" is smaller than the current unix timestamp (this case should not occur upon Grid'5000)

An example of SQL query listing really absent nodes:

SELECT distinct(network_address) FROM resources WHERE state='Absent' AND (available_upto < UNIX_TIMESTAMP() OR available_upto = 0)

How to detect nodes in maintenance ?

Nodes in maintenance are nodes with a Dead state, a OAR maintenance property set to YES, or temporary in a really absent state (see above).

How to execute jobs within another one ?

This functionality is named container jobs. With this functionality it is possible to execute jobs within another one. So it is like a sub-scheduling mechanism.

  • First a job of the type container must be submitted, for example:
oarsub -I -t container -t cosystem -l nodes=10,walltime=2:10:00
...
OAR_JOB_ID=13542
...
  • Then it is possible to use the inner type to schedule the new jobs within the previously created container job:
oarsub -I -t inner=13542 -l nodes=7,walltime=0:30:00
oarsub -I -t inner=13542 -l nodes=3,walltime=0:45:00

More information in Advanced_OAR#Container_jobs

How to use MPI in Grid5000?

See also : The MPI Tutorial


MPI options to use

It is preferable to use the following options to avoid warnings or errors:

  • INFINIBAND clusters: Parapluie, Parapide, Griffon, Graphene, Edel, Genepi
Terminal.png node:
mpirun --mca orte_rsh_agent "oarsh" --mca btl openib,sm,self --mca pml ^cm -machinefile $OAR_NODEFILE $HOME/mpi_programm
  • For other clusters, you may use the following options:
    • --mca pml ob1 --mca btl tcp,self
    • --mca btl ^openib
    • --mca btl ^mx