This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.
- 1 How it works
- 2 Usage
- 3 Show and use my reserved disks
- 4 Security issues
Disk reservation consists in reserving on nodes additional hard disks, which are otherwise not usable.
The table below shows the Grid'5000 clusters with such additional hard disks available for reservation.
|Site||Cluster||Number of nodes||Number of reservable disks per node|
How it works
Two use cases of the disk reservsation are possible:
- Long run reservations of disks only (job reserving no host, i.e. no processing power)
- disk-only reservations do not have to fit in the day vs. night&week-end host reservation policy, and can last up to many days (see Grid5000:UsagePolicy). The reserved disks can then be used by regular host jobs during the period of time of the disk reservation. In this use case, the goal is to get more persistence for the local storage of nodes, e.g. avoid the need to reformat disks and reimport dataset in each regular host job. Those long run jobs must use the
noopOAR job type.
- Regular jobs reserving both host and disks
- In this use case, the goal is to get access to the reservable disks within the experiment, just as if the disk were not to reserve separately.
In both cases, making use of the reserved disks requires to gain the root privileges, since disks are provided as bare metal hardware to be partitioned, formated, mounted, filled with no restriction but by the experimenter. As a result, the experimenter can use the reserved disk:
- either in a non-deploy job, in the standard environment but after enabling sudo with the
- or in a deploy job, in a kadeployed environment (use the
deployOAR job type, then
Technically speaking, when a deploy job starts, or whenever
sudo-g5k is called in a non-deploy job, the reserved disks stay available (shown by
lsblk) while the other disks are disabled and disappear.
Mind that some disks may show up in
Reserved disks can only be accessed by the user who reserved them.
Please note that reserved disks are not cleaned-up at the end of reservation. As a result:
- Data let on the disks can be accessed by user in later reservations.
- Reserved disk may first need to get cleaned-up before use (remove previous formating and partitioning)
See also Security issues.
The main commands to reserve disks are given below.
The maximum duration of a disk reservation is defined in the Usage Policy.
In the following example, add
Reserve disks and nodes at the same time
- How to reserve a node with only the main disk (none of the additional disks), on the grimoire cluster
(no change to the way a node was to be reserved in the past, before the disk reservation mechanism existed.)
- How to reserve a node with all its disks
- How to reserve nodes grimoire-1 and grimoire-2 with one reservable disk per node
Yes, the syntax of the last oarsub command is a bit awkward, so please be careful and mind having:
Reserve disks and nodes separately
You may, for example, decide to reserve some disks for one week, but the nodes where your disks are located only when you want to carry out an experiment.
First: reserve the disks
Since we want to reserve disks only in a first time, we use the noop job type: with this noop job type, OAR will not try to execute anything on the job resources (which is what we want since disk resources are not capable of executing programs).
(Please mind that Jobs of type noop cannot be interactive:
-I -t noop ... is not supported.)
Reserve two disks on grimoire-1 for one week, starting on 2018-01-01:
Or reserve the first two disks on grimoire-2:
Or reserve all disks on two nodes:
Second: reserve the nodes
You can then reserve nodes grimoire-1 and grimoire-2 for 3 hours, in the usual way:
You must respect this order : reserve the disks first, then reserve the nodes. Otherwise the disks you reserved will not be available on your nodes.
Show and use my reserved disks
Gantt diagrams with disk reservations
Reservations of both nodes (processors) and disks are displayed on the following Gantt diagrams:
Getting information about disk reservations from OAR and G5K APIs
- The OAR API shows the properties of each resource of a job. You can retrieve the properties of your reserved disks, such as disk or diskpath:
- The Grid'5000 API also provide some details about disk reservations under the "disks" key of the status and jobs APIs:
Use the disks once connected on the nodes
Connect to a node where you reserved one or more disks, either in the standard environment (thus using
sudo-g5k to get the root privileges) or in a kadeployed environment (and connected as root).
Then, several tools can be used to manage the disk(s):
ls -l /dev/disk/by-path/or equivalent
- shows you the block devices of your disks:
sdc, ... But careful again ! Find out which disk(s) you actually reserved ! It is not
sdawhich is the system disk, nor any of the non-reservable disks ! While you can modifiy any disk, only reservable disks are reserved !
For instance, assuming the
sdc disk is reserved on a
yeti machine in Grenoble,
lsblk will show:
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 447.1G 0 disk ├─sda1 8:1 0 3.7G 0 part [SWAP] ├─sda2 8:2 0 19.6G 0 part / ├─sda3 8:3 0 22.4G 0 part ├─sda4 8:4 0 1K 0 part └─sda5 8:5 0 401.5G 0 part /tmp sdc 8:32 0 1.8T 0 disk nvme0n1 259:0 0 1.5T 0 disk
And the matching by-path naming which is provided in Grenoble:Hardware#yeti is:
$ ls -l /dev/disk/by-path/ total 0 lrwxrwxrwx 1 root root 9 Oct 7 20:11 pci-0000:18:00.0-scsi-0:0:0:0 -> ../../sda lrwxrwxrwx 1 root root 10 Oct 7 20:11 pci-0000:18:00.0-scsi-0:0:0:0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 7 20:11 pci-0000:18:00.0-scsi-0:0:0:0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Oct 7 20:12 pci-0000:18:00.0-scsi-0:0:0:0-part3 -> ../../sda3 lrwxrwxrwx 1 root root 10 Oct 7 20:11 pci-0000:18:00.0-scsi-0:0:0:0-part4 -> ../../sda4 lrwxrwxrwx 1 root root 10 Oct 7 20:11 pci-0000:18:00.0-scsi-0:0:0:0-part5 -> ../../sda5 lrwxrwxrwx 1 root root 9 Oct 7 20:11 pci-0000:18:00.0-scsi-0:0:2:0 -> ../../sdc lrwxrwxrwx 1 root root 13 Oct 7 20:11 pci-0000:6d:00.0-nvme-1 -> ../../nvme0n1
Hence we notice that:
sdddo not show up: indeed they are reservable disks (see Grenoble:Hardware#yeti), but are not reserved !
sda(and the partitions
sda1, ...) show up, but that's the system disk, which is not researvable, and should mostly not be used more than it already is.
- Only some
sdapartitions are already mounted.
nvme0n1shows up, but it's the NVMe disk available on the yeti node, which is not a reservable disk (but can be exploited for a day or night/weekend experiment of course ! Feel free to partition/format/mount it)
- Note that
nvme0n1partitions could show up in additional lines, if any partition was created in a previous use of the disks!
If deploying an environment on the node with
partedor any other disk partionning tool
- must be used to partition (if no partition is defined) or re-partition a disk.
mkfsor any format utility
- must be used to format partitions with the desired filesystem.
- must be used to mount the partition.
pvremoveor other tools
- may have to be used before doing anything else (e.g. before any
fdisk) if a reserved disk was used before and let with unwanted signatures of the previous usage (partition table, LVM signature, etc).
- can be used for the lowest level block device operations. Use with care.
While, reserved disks can be used with quite basic mechanisms (partition + filesystem), they can of course also be used to set up more complex or specific usages, such as LVM, software raid, HDFS, Ceph, Beegfs, etc. Each of those specific usages requires specific tools which are not documented here.
Mind that the disk reservation system provides access to the block devices. It does not manage for you the partitioning nor the formatting nor the mounting of disks.
The mechanism used to enable/disable disks is designed to avoid mistakes from other users. However, a malicious user could take control of the RAID card, enable any disk, and access or erase your data. Please notify the Grid'5000 tech-team in case of such event, but first of all mind securing your data:
- Keep a copy (backup) in a safe place if relevant for your data ;
- If your data is sensitive, mind using cryptographic mechanisms to secure it.
Also, the data on reserved disks is not automatically erased at the end of your job. If you don't want the next user to access it, you have to erase it yourself.
Finally, no backup of data stored on the reserved disks is made.