Storage5k: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
No edit summary
Line 74: Line 74:
|}
|}


= Purpose =
This page presents ways to use storage5k for the storage reservation in Grid'5000.


= Overview =


Each user has a default storage space (/home) exported via NFS. For larger storage needs, a user can book another storage space for a determined period.
= [[Storage5k]] =
storage5k allows users to reserve "storage" resources with OAR.
The idea is that reservation length for storage is in the order of weeks, and therefore storage5k can persist a dataset imported into Grid'5000 between experiments


= Usage =
== Overview ==
=== First check the chunk disk size of the oar storage resources  ===


malachheb@fsophia:~$ storage5k -a chunk_size
[[Storage5k]] is a tool for the reservation of large storage, providing much more space than the one available in a user's /home. Contrary to other resource reservations in Grid'5000, the duration of a storage5k reservation is not limited, allowing data to be used in long term experiments.
Chunk size: 10737418240 (10 GB)


In this example the chunk size is 10GB, so for a reservation of 100 GB, you should ask for 10 chunks.
The data reserved by storage5k is directly accessible on Grid'5000 nodes, using NFS or iSCSI mounts.  


=== Storage reservation  ===
A storage reservation is an OAR job with resources of type "storage". For each storage reservation you should specify the chunk number and the walltime (1 hour by default).
Storage5k supports two protocols for exporting the reserved volume; iscsi and NFS, you can choose the protocol you want to use with the "-p" option. By default NFS protocol is used. 


You can submit a storage reservation in two ways:
{{Warning|text=Storage5k is an experimental tool and data integrity is not guaranteed. You MUST backup your important data.}}


==== with storage5k command ====
{{Term|location=frontend|cmd=<code class="command">storage5k -a add -l chunks=2,walltime=2 -p iscsi</code>}}
  malachheb@fsophia:~$ storage5k -a add -l chunks=2,walltime=2 -p iscsi
  [ADMISSION RULE] Modify resource description with type constraints
  [ADMISSION RULE] Modify resource description as storage
  Generate a job key...
  OAR_JOB_ID=648
  Reservation mode : waiting validation...
  Reservation valid --> OK


==== with oarsub command ====
See the [[Storage5k]] page to check on which sites Storage5k is available.
{{Term|location=frontend|cmd=<code class="command">oarsub -t deploy -l chunks=2,walltime=2 -r "2011-12-09 11:57:00" -t iscsi</code>}}
  malachheb@fsophia:~$ oarsub -t deploy -l chunks=2,walltime=2 -r "2011-12-09 11:57:00" -t iscsi
  [ADMISSION RULE] Modify resource description with type constraints
  [ADMISSION RULE] Modify resource description as storage
  Generate a job key...
  OAR_JOB_ID=649
  Reservation mode : waiting validation...
  Reservation valid --> OK
{{Note|text= Charter allows an unlimited reservation walltime.
}}


{{Note|text= A reservation can take a few minutes if the space reserved is very large, due to the initialization of the reserved partition in the storage server.
== Usage ==
}}


{{Warning|text= Submitting a storage reservation simultaneously with compute nodes or other oar resources (vlan, subnet) '''is not supported''' (you would get an unexpected behaviour, this is a temporary limitation while we get things ironed out). }}
The Storage5k tool is available on Grid'5000 frontends. Check if the <code class="command">storage5k</code> command is installed with:


=== Check your storage reservation ===
<code class="host">frontend: </code><code class="command">storage5k</code> -v


Each storage reservation is identified by the job cpuset_name (e.g, malachheb_649). To display the details of your reservations, you can use "storage5k -a info".


{{Term|location=frontend|cmd=<code class="command">storage5k -a info</code>}}
An individual Storage5k resource is called a "chunk", and represents the smallest allocatable unit of storage. A typical chunk size is 10GB but it may vary among Grid'5000 sites. To display the chunks size, use the following command:


  malachheb@fsophia:~$ storage5k -a info
<code class="host">frontend: </code><code class="command">storage5k</code> -a chunk_size
  storage5k 1.0.0
  Job_Id: 648
    State: Running
    Reservation Id: malachheb_648
    Chunks: 2
    Chunks_size: 10737418240
    Space reserved: 21474836480
    Data Directory: /data/malachheb_648
    Free space: 20018364416
    Used space: 1119879168
    Source nfs: stock.sophia.grid5000.fr:/data/malachheb_648


See <code>storage5k --help</code> for the list of additional options you can pass. You can also use the <code>oarstat</code> command to display your job related to the storage reservation.


=== Use your reserved space ===
Let's say that you need 50GB of space during one day to carry out your experiment. To reserve this storage, use:


The reserved space is used to store the input/output of your experiments, therefore you need to mount this space in your computing nodes later on. There are two use cases here: nodes reserved by an '''interactive''' job, and nodes reserved by a '''deploy''' job.
  <code class="host">frontend: </code><code class="command">storage5k</code> -a add -l chunks=<code class="replace">number</code>,walltime=24


==== On nodes reserved by an interactive job ====


All your reserved storage spaces will be mounted/umounted automatically in these nodes.
You can get information on your reserved storage with:


  malachheb@fsophia:~$ oarsub -I
<code class="host">frontend: </code><code class="command">storage5k</code> -a info
  [ADMISSION RULE] Set default walltime to 3600.
  [ADMISSION RULE] Modify resource description with type constraints
  Generate a job key...
  OAR_JOB_ID=488925
  Interactive mode : waiting...
  Starting...
  Connect to OAR job 488925 via the node suno-7.sophia.grid5000.fr


  malachheb@suno-7:~$ df -ah
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/sda2            5.6G  2.3G  3.0G  44% /
  tmpfs                  16G    0  16G  0% /lib/init/rw
  proc                    0    0    0  -  /proc
  sysfs                    0    0    0  -  /sys
  udev                  16G  120K  16G  1% /dev
  tmpfs                  16G  4.0K  16G  1% /dev/shm
  devpts                  0    0    0  -  /dev/pts
  /dev/sda5            533G  198M  506G  1% /tmp
  kadeploy:/grid5000/    79G  62G  14G  83% /grid5000
  nfs.sophia.grid5000.fr:/export/home
                      12T  1.2T  11T  10% /home
  fusectl                  0    0    0  -  /sys/fs/fuse/connections
  none                    0    0    0  -  /dev/cpuset
  <span style="color:#009000">stock.sophia.grid5000.fr:/data/malachheb_488924
                      9.9G  23M  9.9G  1% /data/malachheb_488924</span>


Each reservation is mounted by default in the directory "/data/useruid_jobid" (e.g, /data/malachheb_488924). You can make a soft link for this directory in your home.
Note the '''Job_Id''' field. Actually, a Storage5k reservation is an OAR job ! You can use any of the OAR tools, such as:


==== On nodes reserved by a deploy job ====
<code class="host">frontend: </code><code class="command">oarstat</code> -f -j <code class="replace">storage_job_id</code>


before, you should make a OAR deploy reservation and deploy your system image with kadeploy3 in your reserved nodes.


{{Term|location=frontend|cmd=<code class="command">oarsub -t deploy -l nodes=2 -I</code>}}
An other important field is '''Source nfs'''. It displays the NFS mount point where your storage is exported. Note that your storage is already available from the frontend:
{{Term|location=frontend|cmd=<code class="command">kadeploy3 -f $OAR_NODE_FILE -k -e wheezy-x64-nfs</code>}}


<code class="host">frontend: </code><code class="command">ls</code> /data/<code class="replace">username</code>_<code class="replace">storage_job_id</code>
<code class="host">frontend: </code><code class="command">cp</code> <code class="replace">my_big_data</code> /data/<code class="replace">username</code>_<code class="replace">storage_job_id</code>/




{{Warning|text= <br> The deployed image should contain the <code>nfs-common</code> package, if not, you should install this package manually  on the reserved nodes.
Nodes you reserve will automatically have access to the storage. For instance, try this:
<br> The following reference images already contain <code>nfs-common</code>: wheezy-x64-nfs, wheezy-x64-big, wheezy-x64-prod and squeeze-x64-nfs, squeeze-x64-big.
}}


The user should mount/umount manually the reserved space in these nodes as follows.
<code class="host">frontend: </code><code class="command">oarsub</code> -l nodes=3 -I
You must specify the protocol that you used in your storage reservation (by default is the NFS protocol)
<code class="host">node: </code><code class="command">ls</code> /data/<code class="replace">username</code>_<code class="replace">storage_job_id</code>/
lost+found my_big_data
<code class="host">node: </code><code class="command">oarsh</code> <code class="replace">othernode</code>
<code class="host">othernode: </code><code class="command">ls</code> /data/<code class="replace">username</code>_<code class="replace">storage_job_id</code>/
lost+found my_big_data


* Mount/Umount ALL reserved space in ALL reserved nodes of the job ($OAR_JOBID is the deploy job which contains two nodes, suno-6 and suno-7):


{{Term|location=frontend|cmd=<code class="command">storage5k -a mount -j $OAR_JOBID -p {iscsi or nfs} </code>}} 
Though there is an exception: Deployed nodes do not mount your reserved storage by default. Let's try:


  malachheb@fsophia:~$ storage5k -a mount -j $OAR_JOBID -p iscsi
<code class="host">frontend: </code><code class="command">oarsub</code> -l nodes=3 -t deploy -I
  mount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success]
  mount the storage reservation malachheb_651 in node suno-7.sophia.grid5000.fr [ Success]
<code class="host">frontend: </code><code class="command">kadeploy3</code> -e wheezy-x64-nfs -f $OAR_NODE_FILE -k
<code class="host">frontend: </code><code class="command">ssh</code> <code class="replace">node</code>
<code class="host">node: </code><code class="command">ls</code> /data
ls: cannot access /data: No such file or directory
<code class="host">node: </code><code class="command">exit</code>


{{Term|location=frontend|cmd=<code class="command">storage5k -a umount -j 651 -p {iscsi or nfs} </code>}}


  malachheb@fsophia:~$ storage5k -a umount -j $OAR_JOBID  -p iscsi                                                                                                                                                         
For this situation, Storage5k provides you with a way to mount the reserved space into your nodes. This command will mount your storage in all the nodes belonging to the job <code class="replace">nodes_job_id</code>
  umount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success]
  umount the storage reservation malachheb_651 in node suno-7.sophia.grid5000.fr [ Success]


* Mount/Umount ALL reserved space in ONE reserved node:
<code class="host">frontend: </code><code class="command">storage5k</code> -a mount -j <code class="replace">nodes_job_id</code>
<code class="host">frontend: </code><code class="command">ssh</code> <code class="replace">node</code>
<code class="host">node: </code><code class="command">ls</code> /data/<code class="replace">username</code>_<code class="replace">storage_job_id</code>/
lost+found my_big_data


{{Term|location=frontend|cmd=<code class="command">storage5k -a mount -j 651 -m suno-6.sophia.grid5000.fr -p {iscsi or nfs} </code>}} 


  malachheb@fsophia:~$ storage5k -a mount -j 651 -m suno-6.sophia.grid5000.fr -p iscsi
You can umount storage with:  
  mount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success]


{{Term|location=frontend|cmd=<code class="command">storage5k -a umount -j 651 -m suno-6.sophia.grid5000.fr -p {iscsi or nfs} </code>}}
<code class="host">frontend: </code><code class="command">storage5k</code> -a umount -j <code class="replace">nodes_job_id</code>


  malachheb@fsophia:~$ storage5k -a umount -j 651 -m suno-6.sophia.grid5000.fr -p iscsi                                                                                                                                                         
  umount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success]


==== Access to your reserved space ====
With the knowledge of the '''Source nfs''' entry, you can manually mount your storage as well:


*Only the owner of the reservation can access to the reserved space.
<code class="host">frontend: </code><code class="command">ssh</code> root@<code class="replace">node</code>  #As root
*In the case of deploy reservation, you must be logged in as your grid5000 username on the nodes, if you want to be able to list and create files in the reserved storage spaces.  
 
root@<code class="host">node: </code><code class="command">mount</code> storage5k.lyon.grid5000.fr:/data/<code class="replace">username_storage_job_id</code> /mnt
{{Warning|text= All Grid5000 users and groups is preconfigured in the reference images (except the min images), the use of reference images is highly recommended.<br>
If you use another system image, you must add a user with the same uid/gid than your grid5000 username.
root@<code class="host">node: </code><code class="command">exit</code>
}}
 
<code class="host">frontend: </code><code class="command">ssh </code><code class="replace">node</code>        #As normal user
=== Delete your storage reservation ===
* storage5k -a del -j $jobid
<code class="host">node: </code><code class="command">ls</code> /mnt/
lost+found my_big_data
<code class="host">node: </code><code class="command">exit</code>


{{Term|location=frontend|cmd=<code class="command">storage5k -a del -j 648</code>}}


  malachheb@fsophia:~$ storage5k -a del -j 488787
  Deleting the job = 488787 ...REGISTERED.
  The job(s) [ 488787 ] will be deleted in a near future.


{{Warning|text= If you remove your storage reservation, you will lose all your data in the reserved space.<br>
{{Warning|text=Accessing storage in deployed nodes requires the <code>nfs-common</code> package. It is included by default in the <code>nfs</code>, <code>big</code> and <code>prod</code> environments}}
Remember to save your data before the deletion of your storage reservation.
And umount the reserved space from your nodes reserved by a deploy job, Otherwise the mounted space will be referenced to a partition that does not exist in the storage server.
}}

Revision as of 18:17, 1 February 2016

See also: Storage | Storage5k | DFS5k

Grid'5000 storage resources

Sites VM Name Disk partition Size status
Sophia stock.sophia.grid5000.fr /dev/sda 2TB Check.png
Grenoble Fail.png
Lille Fail.png
Lyon mizar.lyon.grid5000.fr 20TB Check.png
Luxembourg hera.luxembourg.grid5000.fr 1.80TB Check.png
Nancy gras.nancy.grid5000.fr 2.14TB Check.png
Reims Fail.png
Rennes srv-bigdata.rennes.grid5000.fr 6TB Check.png
Bordeaux Fail.png
Toulouse Fail.png


Storage5k

Overview

Storage5k is a tool for the reservation of large storage, providing much more space than the one available in a user's /home. Contrary to other resource reservations in Grid'5000, the duration of a storage5k reservation is not limited, allowing data to be used in long term experiments.

The data reserved by storage5k is directly accessible on Grid'5000 nodes, using NFS or iSCSI mounts.


Warning.png Warning

Storage5k is an experimental tool and data integrity is not guaranteed. You MUST backup your important data.


See the Storage5k page to check on which sites Storage5k is available.

Usage

The Storage5k tool is available on Grid'5000 frontends. Check if the storage5k command is installed with: 

frontend: storage5k -v


An individual Storage5k resource is called a "chunk", and represents the smallest allocatable unit of storage. A typical chunk size is 10GB but it may vary among Grid'5000 sites. To display the chunks size, use the following command: 

frontend: storage5k -a chunk_size


Let's say that you need 50GB of space during one day to carry out your experiment. To reserve this storage, use: 

frontend: storage5k -a add -l chunks=number,walltime=24


You can get information on your reserved storage with: 

frontend: storage5k -a info


Note the Job_Id field. Actually, a Storage5k reservation is an OAR job ! You can use any of the OAR tools, such as: 

frontend: oarstat -f -j storage_job_id


An other important field is Source nfs. It displays the NFS mount point where your storage is exported. Note that your storage is already available from the frontend: 

frontend: ls /data/username_storage_job_id

frontend: cp my_big_data /data/username_storage_job_id/


Nodes you reserve will automatically have access to the storage. For instance, try this: 

frontend: oarsub -l nodes=3 -I

node: ls /data/username_storage_job_id/
lost+found my_big_data

node: oarsh othernode

othernode: ls /data/username_storage_job_id/
lost+found my_big_data


Though there is an exception: Deployed nodes do not mount your reserved storage by default. Let's try: 

frontend: oarsub -l nodes=3 -t deploy -I

frontend: kadeploy3 -e wheezy-x64-nfs -f $OAR_NODE_FILE -k

frontend: ssh node

node: ls /data
ls: cannot access /data: No such file or directory

node: exit


For this situation, Storage5k provides you with a way to mount the reserved space into your nodes. This command will mount your storage in all the nodes belonging to the job nodes_job_id 

frontend: storage5k -a mount -j nodes_job_id

frontend: ssh node

node: ls /data/username_storage_job_id/
lost+found my_big_data


You can umount storage with: 

frontend: storage5k -a umount -j nodes_job_id


With the knowledge of the Source nfs entry, you can manually mount your storage as well: 

frontend: ssh root@node   #As root

root@node: mount storage5k.lyon.grid5000.fr:/data/username_storage_job_id /mnt

root@node: exit

frontend: ssh node        #As normal user

node: ls /mnt/
lost+found my_big_data

node: exit


Warning.png Warning

Accessing storage in deployed nodes requires the nfs-common package. It is included by default in the nfs, big and prod environments

Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=Storage5k&oldid=57323"