Storage5k
Grid'5000 storage resources
| Sites | VM Name | Disk partition | Size | status |
|---|---|---|---|---|
| Sophia | stock.sophia.grid5000.fr | /dev/sda | 2TB | 
|
| Grenoble | 
| |||
| Lille |
| |||
| Lyon | mizar.lyon.grid5000.fr | 20TB |
| |
| Luxembourg | hera.luxembourg.grid5000.fr | 1.80TB |
| |
| Nancy | gras.nancy.grid5000.fr | 2.14TB |
| |
| Reims |
| |||
| Rennes | srv-bigdata.rennes.grid5000.fr | 6TB |
| |
| Bordeaux |
| |||
| Toulouse |
|
Purpose
This page presents ways to use storage5k for the storage reservation in Grid'5000.
Overview
Each user has a default storage space (/home) exported via NFS. For larger storage needs, a user can book another storage space for a determined period. storage5k allows users to reserve "storage" resources with OAR. The idea is that reservation length for storage is in the order of weeks, and therefore storage5k can persist a dataset imported into Grid'5000 between experiments
Usage
First check the chunk disk size of the oar storage resources
malachheb@fsophia:~$ storage5k -a chunk_size Chunk size: 10737418240 (10 GB)
In this example the chunk size is 10GB, so for a reservation of 100 GB, you should ask for 10 chunks.
Storage reservation
A storage reservation is an OAR job with resources of type "storage". For each storage reservation you should specify the chunk number and the walltime (1 hour by default). Storage5k supports two protocols for exporting the reserved volume; iscsi and NFS, you can choose the protocol you want to use with the "-p" option. By default NFS protocol is used.
You can submit a storage reservation in two ways:
with storage5k command
|
frontend:
|
storage5k -a add -l chunks=2,walltime=2 -p iscsi |
malachheb@fsophia:~$ storage5k -a add -l chunks=2,walltime=2 -p iscsi [ADMISSION RULE] Modify resource description with type constraints [ADMISSION RULE] Modify resource description as storage Generate a job key... OAR_JOB_ID=648 Reservation mode : waiting validation... Reservation valid --> OK
with oarsub command
|
|
frontend:
|
oarsub -t deploy -l chunks=2,walltime=2 -r "2011-12-09 11:57:00" -t iscsi |
malachheb@fsophia:~$ oarsub -t deploy -l chunks=2,walltime=2 -r "2011-12-09 11:57:00" -t iscsi [ADMISSION RULE] Modify resource description with type constraints [ADMISSION RULE] Modify resource description as storage Generate a job key... OAR_JOB_ID=649 Reservation mode : waiting validation... Reservation valid --> OK
|
Note |
|---|---|
Charter allows an unlimited reservation walltime. | |
|
|
Note |
|---|---|
A reservation can take a few minutes if the space reserved is very large, due to the initialization of the reserved partition in the storage server. | |
|
Warning |
|---|---|
Submitting a storage reservation simultaneously with compute nodes or other oar resources (vlan, subnet) is not supported (you would get an unexpected behaviour, this is a temporary limitation while we get things ironed out). | |
Check your storage reservation
Each storage reservation is identified by the job cpuset_name (e.g, malachheb_649). To display the details of your reservations, you can use "storage5k -a info".
|
|
frontend:
|
storage5k -a info |
malachheb@fsophia:~$ storage5k -a info storage5k 1.0.0 Job_Id: 648 State: Running Reservation Id: malachheb_648 Chunks: 2 Chunks_size: 10737418240 Space reserved: 21474836480 Data Directory: /data/malachheb_648 Free space: 20018364416 Used space: 1119879168 Source nfs: stock.sophia.grid5000.fr:/data/malachheb_648
See storage5k --help for the list of additional options you can pass. You can also use the oarstat command to display your job related to the storage reservation.
Use your reserved space
The reserved space is used to store the input/output of your experiments, therefore you need to mount this space in your computing nodes later on. There are two use cases here: nodes reserved by an interactive job, and nodes reserved by a deploy job.
On nodes reserved by an interactive job
All your reserved storage spaces will be mounted/umounted automatically in these nodes.
malachheb@fsophia:~$ oarsub -I [ADMISSION RULE] Set default walltime to 3600. [ADMISSION RULE] Modify resource description with type constraints Generate a job key... OAR_JOB_ID=488925 Interactive mode : waiting... Starting... Connect to OAR job 488925 via the node suno-7.sophia.grid5000.fr
malachheb@suno-7:~$ df -ah
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 5.6G 2.3G 3.0G 44% /
tmpfs 16G 0 16G 0% /lib/init/rw
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
udev 16G 120K 16G 1% /dev
tmpfs 16G 4.0K 16G 1% /dev/shm
devpts 0 0 0 - /dev/pts
/dev/sda5 533G 198M 506G 1% /tmp
kadeploy:/grid5000/ 79G 62G 14G 83% /grid5000
nfs.sophia.grid5000.fr:/export/home
12T 1.2T 11T 10% /home
fusectl 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /dev/cpuset
stock.sophia.grid5000.fr:/data/malachheb_488924
9.9G 23M 9.9G 1% /data/malachheb_488924
Each reservation is mounted by default in the directory "/data/useruid_jobid" (e.g, /data/malachheb_488924). You can make a soft link for this directory in your home.
On nodes reserved by a deploy job
before, you should make a OAR deploy reservation and deploy your system image with kadeploy3 in your reserved nodes.
|
|
frontend:
|
oarsub -t deploy -l nodes=2 -I |
|
|
frontend:
|
kadeploy3 -f $OAR_NODE_FILE -k -e wheezy-x64-nfs |
|
|
Warning |
|---|---|
| |
The user should mount/umount manually the reserved space in these nodes as follows. You must specify the protocol that you used in your storage reservation (by default is the NFS protocol)
- Mount/Umount ALL reserved space in ALL reserved nodes of the job ($OAR_JOBID is the deploy job which contains two nodes, suno-6 and suno-7):
|
|
frontend:
|
storage5k -a mount -j $OAR_JOBID -p {iscsi or nfs} |
malachheb@fsophia:~$ storage5k -a mount -j $OAR_JOBID -p iscsi mount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success] mount the storage reservation malachheb_651 in node suno-7.sophia.grid5000.fr [ Success]
|
|
frontend:
|
storage5k -a umount -j 651 -p {iscsi or nfs} |
malachheb@fsophia:~$ storage5k -a umount -j $OAR_JOBID -p iscsi umount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success] umount the storage reservation malachheb_651 in node suno-7.sophia.grid5000.fr [ Success]
- Mount/Umount ALL reserved space in ONE reserved node:
|
|
frontend:
|
storage5k -a mount -j 651 -m suno-6.sophia.grid5000.fr -p {iscsi or nfs} |
malachheb@fsophia:~$ storage5k -a mount -j 651 -m suno-6.sophia.grid5000.fr -p iscsi mount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success]
|
|
frontend:
|
storage5k -a umount -j 651 -m suno-6.sophia.grid5000.fr -p {iscsi or nfs} |
malachheb@fsophia:~$ storage5k -a umount -j 651 -m suno-6.sophia.grid5000.fr -p iscsi umount the storage reservation malachheb_651 in node suno-6.sophia.grid5000.fr [ Success]
Access to your reserved space
- Only the owner of the reservation can access to the reserved space.
- In the case of deploy reservation, you must be logged in as your grid5000 username on the nodes, if you want to be able to list and create files in the reserved storage spaces.
|
|
Warning |
|---|---|
All Grid5000 users and groups is preconfigured in the reference images (except the min images), the use of reference images is highly recommended. | |
Delete your storage reservation
- storage5k -a del -j $jobid
|
|
frontend:
|
storage5k -a del -j 648 |
malachheb@fsophia:~$ storage5k -a del -j 488787 Deleting the job = 488787 ...REGISTERED. The job(s) [ 488787 ] will be deleted in a near future.
|
|
Warning |
|---|---|
If you remove your storage reservation, you will lose all your data in the reserved space. Remember to save your data before the deletion of your storage reservation. And umount the reserved space from your nodes reserved by a deploy job, Otherwise the mounted space will be referenced to a partition that does not exist in the storage server. | |