Difference between revisions of "Storage Manager"

From Grid5000
Jump to: navigation, search
(Created page with "== How does it work == An user can share its home with other node on Grid'5000. Each time you start a job, an access for all the assigned nodes is automatically created. You m...")
 
m (Passage à debian10.)
 
(37 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== How does it work ==
+
{{Portal|Tutorial}}
An user can share its home with other node on Grid'5000. Each time you start a job, an access for all the assigned nodes is automatically created. You may want to use this tutorial if you want to allow additional addresses to access your home.
+
{{Portal|User}}
 +
{{StorageHeader}}
 +
{{TutorialHeader}}
  
{{Note|text=Only the home owner can interact with his accesses, so you need to be authenticated. It's done automatically on the froentend.}}
+
The Storage Manager manages access to NFS shares (in particular, users' '''home''' directories) on Grid'5000.
  
== Create access ==
+
This page describes its API, and focuses on accessing user's '''home''' directories.
Accesses creation is done by sending POST to the API.
 
Let's assume we want to share the <code class="replace">nancy</code> home of user <code class="replace">jpicard</code> to IP addresses <code class="replace">172.16.72.1</code> and <code class="replace">172.16.72.25</code>
 
  
We need a termination condition, it can be a specific date or a job termination.
+
== How it works ==
 +
The storage API works with '''access''' object. An access is defined by a list of addresses and a termination condition for a specific location on an NFS Server. Each user's home directory is located on <code>/export/home/</code><code class="replace">username</code> on the NFS server of each site: <code>nfs.</code><code class="replace">site</code><code>.grid5000.fr</code>
  
 +
'''Each time you start a job, an access to your local home directory is automatically created for all the assigned nodes. You may want to use this page if you want to allow additional addresses to access your home (like when you are using VLAN) or if you want to access your home over-site.'''
 +
You will need to use the [[API|Grid'5000 API]], the entry-point for the home of a '''user''' on a specific '''site''' is:
 +
<nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">site</code>/storage/home/<code class="replace">username</code>/access
 +
For [[Group Storage]], the entry-point of a '''storage''' on a server '''server''' of a '''site''' is:
 +
  <nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">site</code>/storage/<code class="replace">server</code>/<code class="replace">storage</code>/access
 +
For example, <code><nowiki>https://api.grid5000.fr/stable/sites/lille/storage/storage1/my-team-storage/access</nowiki></code>
 +
{{Note|text=Only the home owner can interact with his or her accesses, so you need to be authenticated. It's done automatically on the frontend. With curl, you can also use the <code>-u </code><code class="replace">username</code> option and you will be prompted for your password before the request is sent.}}
 +
 +
== Usage ==
 +
We assume that we want to manage the home of user <code class="replace">jpicard</code> on <code class="replace">nancy</code>, so the storage API entry-point will be:
 +
  <nowiki> https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access
 +
=== Create access ===
 +
Accesses creation is done by sending a POST request to the API with a JSON payload.
 +
We must provide a list of '''hosts''' to whom we want to give access to, for instance <code class="replace">dahu-1.grenoble.grid5000.fr</code> and <code class="replace">dahu-2.grenoble.grid5000.fr</code>. We also need a termination condition, it can be a specific date or a job termination.
 +
{{Note|text=To give access to a home from a specific VLAN, you can use its domain name, for example for dahu-1 on VLAN 2, you can add <code class="replace">dahu-1-kavlan-2.grenoble.grid5000.fr</code>.}}
 +
==== Until termination ====
 
If we want to use a specific time (for example <code class="replace">2018-12-25 19:38</code>), we send the following request to the Storage API:
 
If we want to use a specific time (for example <code class="replace">2018-12-25 19:38</code>), we send the following request to the Storage API:
   curl -X POST 'https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access' -H "Content-Type: application/json" -d '{"ipv4": ["172.16.72.1", "172.16.72.25"], "termination" : {"until":"2018-12-25 19:38"}}'
+
   curl -X POST <nowiki>'https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access'  
 +
  -H "Content-Type: application/json"  
 +
  -d '{"hosts": [<code class="replace">"dahu-1.grenoble.grid5000.fr", "dahu-2.grenoble.grid5000.fr"</code>],  
 +
      "termination" : {"until":"<code class="replace">2018-12-25 19:38"</code>}}'
 +
 
 +
==== Job termination ====
 +
If we want our access to be valid until the end of a Grid'5000 job, for example the job <code class="replace">4548</code> of  <code class="replace">grenoble</code>, we do:
 +
  curl -X POST <nowiki>'https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access'
 +
  -H "Content-Type: application/json"
 +
  -d '{"hosts": [<code class="replace">"dahu-1.grenoble.grid5000.fr", "dahu-2.grenoble.grid5000.fr"</code>],
 +
      "termination" : {"job": <code class="replace">4548</code>, "site": <code class="replace">"grenoble"</code>}}'
 +
If we want to give access to all the assigned nodes of a job, we don't need to give any hosts, the API will fill them for us:
 +
  curl -X POST <nowiki>'https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access'
 +
  -H "Content-Type: application/json"
 +
  -d '{"termination" : {"job": <code class="replace">4548</code>, "site": <code class="replace">"grenoble"</code>}}'
 +
 
 +
{{Warning|text= The access is created immediately and is valid until the termination of the job, even if the job has not started yet.}}
 +
 
 +
The '''nfs_address''' field will contain the address used to manually mount the home:
 +
{{Term|location=node|cmd=mount srv-data.nancy.grid5000.fr:/export/home/<code class="replace">jpicard</code> <code class="replace">/mnt/</code>}}
  
If we want to use a job termination, for example the job <code class="replace">4548</code> of  <code class="replace">grenoble</code>:
 
  curl -X POST 'https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access' -H "Content-Type: application/json" -d '{"ipv4": ["172.16.72.1", "172.16.72.25"], "termination" : {"job": 4548, "site": "grenoble"}}
 
{{Warning|text= The access is create immediately and until the termination of the job, even if the job has not yet started.}}
 
  
== List access ==
+
{{Note|text=<code class="env">debian10-x64-std</code>,<code class="env">debian10-x64-big</code> and <code class="env">debian10-x64-nfs</code> use autofs for the home directories of the local NFS server. So doing <code>cd /home/jpicard/</code> will automount the home of jpicard if you have the rights.}}
To list all access, you should do:
+
 
   curl https://api.grid5000.fr/stable/sites/<code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access
+
=== List access ===
You get a list of accesses
+
To list all accesses, you should do:
 +
   curl <nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access
 +
You get a list of accesses:
 
   {
 
   {
 
     "G5k-home_jpicard_j_1666466-nancy_1": {
 
     "G5k-home_jpicard_j_1666466-nancy_1": {
Line 29: Line 64:
 
         "job": 1666466,
 
         "job": 1666466,
 
         "site": "nancy"
 
         "site": "nancy"
       }
+
       },
 +
      "nfs_address": "srv-data.nancy.grid5000.fr:/export/home/jpicard"
 
     },
 
     },
 
     "G5k-home_jpicard_u_1535456240_1": {
 
     "G5k-home_jpicard_u_1535456240_1": {
Line 37: Line 73:
 
       "termination": {
 
       "termination": {
 
         "until": 1535456240,
 
         "until": 1535456240,
       }
+
       },
 +
      "nfs_address": "srv-data.nancy.grid5000.fr:/export/home/jpicard"
 
     }
 
     }
 
   }
 
   }
You can also ask for a specific access by providing its identifier, for instance :
+
You can also ask for a specific access by providing its identifier, for instance:
   curl https://api.grid5000.fr/stable/sites/<code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access/<code class="replace">G5k-home_jpicard_u_1535456240_1</code>
+
   curl <nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access/<code class="replace">G5k-home_jpicard_u_1535456240_1</code>
 
 
After that, you should be able to mount the home where you want:
 
{{Term|location=node|cmd=mount nfs.<code class="replace">nancy</code>.grid5000.fr:/home/<code class="replace">jpicard</code>}}
 
 
 
 
 
{{Note|text=<code class="env">debian9-x64-std</code>,<code class="env">debian9-x64-big</code> and <code class="env">debian9-x64-nfs</code> use autofs for the /home of the local NFS server. So <code>cd /home/jpicard/</code> will automount the home of jpicard if you have the right}}
 
 
 
== Delete an access ==
 
Accesses deletion is done by sending DELETE to the API.
 
To delete an access you specify it's identifier, for instance:
 
curl -X DELETE https://api.grid5000.fr/stable/sites/<code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access/<code class="replace">G5k-home_jpicard_u_1535456240_1</code>
 
  
 +
=== Delete an access ===
 +
Access deletion is done by sending a DELETE request to the API with the access identifier.
 +
curl -X DELETE <nowiki>https://api.grid5000.fr/stable/sites/</nowiki><code class="replace">nancy</code>/storage/home/<code class="replace">jpicard</code>/access/<code class="replace">G5k-home_jpicard_u_1535456240_1</code>
  
 
== API Specification: ==
 
== API Specification: ==
 
+
The Storage API uses the same status codes as the rest of [[API#Status_Codes|the Grid'5000 API]].
 
=== GET /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access ===
 
=== GET /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access ===
fetch a list of access.
+
Fetch a list of access.
  
 
=== GET /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access/<code class="replace">access_id</code> ===
 
=== GET /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access/<code class="replace">access_id</code> ===
fetch information of a specific access
+
Fetch information of a specific access.
  
 
=== DELETE /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access/<code class="replace">access_id</code> ===
 
=== DELETE /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access/<code class="replace">access_id</code> ===
delete a specific access
+
Delete a specific access.
  
 
=== DELETE /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access/?job=<code class="replace">job_id</code>&site=<code class="replace">job_site</code> ===
 
=== DELETE /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access/?job=<code class="replace">job_id</code>&site=<code class="replace">job_site</code> ===
delete all access link to job '''job_id''' of site '''job_site'''
+
Delete all access with a job termination '''job_id''' of site '''job_site'''.
  
 
=== POST /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access ===
 
=== POST /stable/sites/<code class="replace">site</code>/storage/home/<code class="replace">username</code>/access ===
Create new access. The payload must be in JSON format, and contains the following mandatory properties:
+
Create new access. The payload must be in JSON format, and contains the following properties:
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
 
! Parameter !! Type !! Description & Examples
 
! Parameter !! Type !! Description & Examples
 
|-
 
|-
| ipv4 || Mandatory || Array of ipv4 allowed to access NFS
+
| ipv4 || Optional || Array of ipv4 allowed to access NFS. eg:
 
   ["172.16.72.1",
 
   ["172.16.72.1",
     ""172.16.72.25]
+
     "172.16.72.25"]
 +
|-
 +
| hosts || Optional || Array of hosts allowed to access NFS. eg:
 +
  ["grisou-1.nancy.grid5000.fr",
 +
    "grisou-2.nancy.grid5000.fr"]
 
|-
 
|-
 
| termination || Mandatory || Termination condition can be of two types "Until" or "Job". The form is a structure/hash.
 
| termination || Mandatory || Termination condition can be of two types "Until" or "Job". The form is a structure/hash.
Until termination takes a date as an '''until''' parameter. The access will be deleted after the specify date. eg:
+
Until termination takes a date as an '''until''' parameter. The access will be deleted after the specify date. Eg:
 
   {"until" : "2018-12-25 19:38"}
 
   {"until" : "2018-12-25 19:38"}
Job terminations takes two parameters, '''job''' and '''site''', which are respectively the id and the site of a Grid'5000 job. When the job end, the export is deleted. eg:
+
Job terminations takes two parameters, '''job''' and '''site''', which are respectively the id and the site of a Grid'5000 job. When the job ends, the export is deleted. eg:
   {"job": "4548, "site":"nancy"}
+
   {"job": "4548, "site": "nancy"}
 +
If no hosts and no ipv4 are provided, the API will use the assigned nodes from the job.
 
|-
 
|-

Latest revision as of 15:40, 18 October 2019

Note.png Note

This tutorial is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

The Storage Manager manages access to NFS shares (in particular, users' home directories) on Grid'5000.

This page describes its API, and focuses on accessing user's home directories.

How it works

The storage API works with access object. An access is defined by a list of addresses and a termination condition for a specific location on an NFS Server. Each user's home directory is located on /export/home/username on the NFS server of each site: nfs.site.grid5000.fr

Each time you start a job, an access to your local home directory is automatically created for all the assigned nodes. You may want to use this page if you want to allow additional addresses to access your home (like when you are using VLAN) or if you want to access your home over-site. You will need to use the Grid'5000 API, the entry-point for the home of a user on a specific site is:

https://api.grid5000.fr/stable/sites/site/storage/home/username/access

For Group Storage, the entry-point of a storage on a server server of a site is:

 https://api.grid5000.fr/stable/sites/site/storage/server/storage/access

For example, https://api.grid5000.fr/stable/sites/lille/storage/storage1/my-team-storage/access

Note.png Note

Only the home owner can interact with his or her accesses, so you need to be authenticated. It's done automatically on the frontend. With curl, you can also use the -u username option and you will be prompted for your password before the request is sent.

Usage

We assume that we want to manage the home of user jpicard on nancy, so the storage API entry-point will be:

  https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access

Create access

Accesses creation is done by sending a POST request to the API with a JSON payload. We must provide a list of hosts to whom we want to give access to, for instance dahu-1.grenoble.grid5000.fr and dahu-2.grenoble.grid5000.fr. We also need a termination condition, it can be a specific date or a job termination.

Note.png Note

To give access to a home from a specific VLAN, you can use its domain name, for example for dahu-1 on VLAN 2, you can add dahu-1-kavlan-2.grenoble.grid5000.fr.

Until termination

If we want to use a specific time (for example 2018-12-25 19:38), we send the following request to the Storage API:

 curl -X POST 'https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access' 
 -H "Content-Type: application/json" 
 -d '{"hosts": ["dahu-1.grenoble.grid5000.fr", "dahu-2.grenoble.grid5000.fr"], 
      "termination" : {"until":"2018-12-25 19:38"}}'

Job termination

If we want our access to be valid until the end of a Grid'5000 job, for example the job 4548 of grenoble, we do:

 curl -X POST 'https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access'
 -H "Content-Type: application/json" 
 -d '{"hosts": ["dahu-1.grenoble.grid5000.fr", "dahu-2.grenoble.grid5000.fr"], 
      "termination" : {"job": 4548, "site": "grenoble"}}'

If we want to give access to all the assigned nodes of a job, we don't need to give any hosts, the API will fill them for us:

 curl -X POST 'https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access'
 -H "Content-Type: application/json"
 -d '{"termination" : {"job": 4548, "site": "grenoble"}}'
Warning.png Warning

The access is created immediately and is valid until the termination of the job, even if the job has not started yet.

The nfs_address field will contain the address used to manually mount the home:

Terminal.png node:
mount srv-data.nancy.grid5000.fr:/export/home/jpicard /mnt/


Note.png Note

debian10-x64-std,debian10-x64-big and debian10-x64-nfs use autofs for the home directories of the local NFS server. So doing cd /home/jpicard/ will automount the home of jpicard if you have the rights.

List access

To list all accesses, you should do:

 curl https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access

You get a list of accesses:

 {
   "G5k-home_jpicard_j_1666466-nancy_1": {
     "ipv4": [
       "172.16.64.97"
     ],
     "termination": {
       "job": 1666466,
       "site": "nancy"
     },
     "nfs_address": "srv-data.nancy.grid5000.fr:/export/home/jpicard"
   },
   "G5k-home_jpicard_u_1535456240_1": {
     "ipv4": [
       "172.16.64.16"
     ],
     "termination": {
       "until": 1535456240,
     },
     "nfs_address": "srv-data.nancy.grid5000.fr:/export/home/jpicard"
   }
 }

You can also ask for a specific access by providing its identifier, for instance:

 curl https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access/G5k-home_jpicard_u_1535456240_1

Delete an access

Access deletion is done by sending a DELETE request to the API with the access identifier.

curl -X DELETE https://api.grid5000.fr/stable/sites/nancy/storage/home/jpicard/access/G5k-home_jpicard_u_1535456240_1

API Specification:

The Storage API uses the same status codes as the rest of the Grid'5000 API.

GET /stable/sites/site/storage/home/username/access

Fetch a list of access.

GET /stable/sites/site/storage/home/username/access/access_id

Fetch information of a specific access.

DELETE /stable/sites/site/storage/home/username/access/access_id

Delete a specific access.

DELETE /stable/sites/site/storage/home/username/access/?job=job_id&site=job_site

Delete all access with a job termination job_id of site job_site.

POST /stable/sites/site/storage/home/username/access

Create new access. The payload must be in JSON format, and contains the following properties:

Parameter Type Description & Examples
ipv4 Optional Array of ipv4 allowed to access NFS. eg:
  ["172.16.72.1",
   "172.16.72.25"]
hosts Optional Array of hosts allowed to access NFS. eg:
  ["grisou-1.nancy.grid5000.fr",
   "grisou-2.nancy.grid5000.fr"]
termination Mandatory Termination condition can be of two types "Until" or "Job". The form is a structure/hash.

Until termination takes a date as an until parameter. The access will be deleted after the specify date. Eg:

  {"until" : "2018-12-25 19:38"}

Job terminations takes two parameters, job and site, which are respectively the id and the site of a Grid'5000 job. When the job ends, the export is deleted. eg:

  {"job": "4548, "site": "nancy"}

If no hosts and no ipv4 are provided, the API will use the assigned nodes from the job.