Disk reservation: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
(40 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Maintainer|Florent Didier}}
{{Portal|User}}
{{Portal|User}}
{{Status|In production}}
{{Status|In production}}
{{StorageHeader}}
{{StorageHeader}}
{{TutorialHeader}}


'''Disk reservation''' consists in reserving nodes' hard disks, in order to locally store large datasets between your reservations, and avoid needing moving data to nodes at the beginning of each node reservation.
__TOC__


It takes benefit from nodes which feature a bunch of disks.
'''Disk reservation''' consists in reserving on nodes additional hard disks, which are otherwise not usable.


{{Note|text=During the beta testing phase, disks reservation will only be available on the ''grimoire'' cluster. The service will be extended to other clusters later. }}
The table below shows the Grid'5000 clusters with such additional hard disks available for reservation.
{{:Generated/DiskReservation}}


The table below shows the cluster compatibility with disk reservation.
= How it works =
{| class="wikitable"
Two use cases of the disk reservsation are possible:
!Site
* long run reservations of '''disks only''' (job reserving no host, i.e. no processing power): disk-only reservations do not have to fit in the day vs. night&week-end ''host'' reservation policy, and can last up to many days (see [[Grid5000:UsagePolicy]]). The reserved disks can then be used by regular ''host'' jobs during the period of time of the ''disk'' reservation. In this use case, the goal is to get more persistence for the local storage of nodes, e.g. avoid the need to reformat disks and reimport dataset in each regular ''host'' job. Those long run jobs must use the <code class='command'>noop</code> OAR job type.
!Cluster
* regular jobs reserving both '''host and disks'''. In this use case, the goal is to get access to the reservable disks within the experiment, just as if the disk were not to reserve separately.
!Number of nodes
 
!Number of reservable disks per node
In both cases, making use of the reserved disks requires to gain the '''root privileges''', since disks are provided as bare metal hardware to be partitioned, formated, mounted, filled with no restriction but by the experimenter. As a result, the experimenter can use the reserved disk:
|-
* either in a '''deploy''' job, in a kadeployed environment (use the <code class='command'>deploy</code> OAR job type, then <code class='command'>kadeploy</code>)
|rowspan="1"|[[Nancy:Hardware|Nancy]]
* or in a '''non-deploy''' job, in the standard environment but after enabling sudo with the <code class='command'>sudo-g5k</code> command.
|grimoire
|8
|5
|}


{{Warning|text=Disk reservation is a service for experimentation: data integrity is not guaranteed. You MUST backup your important data.}}
Technically speaking, when a ''deploy'' job starts, or whenever <code class='command'>sudo-g5k</code> is called in a ''non-deploy'' job, the reserved disks stay available while the other disks are disabled. Reserved disks can only be accessed by the user who reserved them.


= How it works =
Please note that reserved disks are '''not cleaned-up''' at the end of reservation. As a result:
Disk reservation works with jobs of type '''deploy''' and jobs of type '''noop'''. You can reserve some disks in the long run with a job of type noop. Then, you may reserve a job of type deploy on the same nodes as those where you reserved your disks.
* Data let on the disks can be accessed by user in later reservations.
* Reserved disk may first need to get cleaned-up before use (remove previous formating and partitioning)


When the job of type ''deploy'' starts, the disks you reserved are enabled by the RAID card of the node, and the other disks are disabled. Reserved disks can only be accessed by the user who reserved them (see also [[Disk_reservation#Security_issues|Security issues]]).
See also [[Disk_reservation#Security_issues|Security issues]].


= Usage =
= Usage =
Line 38: Line 36:


== Reserve disks and nodes at the same time ==
== Reserve disks and nodes at the same time ==
; Reserve a node with only the main disk (none of the additional disks), on the grimoire cluster:
{{Note|text=In the 3 following examples, remove the <code>-t deploy</code> if you plan to use the standard environment and <code class="command">sudo-g5k</code> in order to get the ''root privileges'', instead of <code class="command">kadeploy</code> and be ''root'' in you own deployed environment.}}
; How to reserve a node with only the main disk (none of the additional disks), on the grimoire cluster:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -p "cluster='grimoire'" -l /host=1</code>}}
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -p "cluster='grimoire'" -l /host=1</code>}}
(no change to the way a node was to be reserved in the past, before the disk reservation mechanism existed.)
(no change to the way a node was to be reserved in the past, before the disk reservation mechanism existed.)


; Reserve a node with all its disks:
; How to reserve a node with all its disks:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -l {"type='disk' or type='default' and disk_reservation_count > 0"}/host=1</code>}}
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -l {"(type='disk' or type='default') and cluster='grimoire'"}/host=1</code>}}
(disk_reservation_count is equal to the number of disks which can be reserved on a node.)


; Reserve nodes grimoire-1 and grimoire-2 with one reservable disk per node:
; How to reserve nodes grimoire-1 and grimoire-2 with one reservable disk per node:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -p "host in ('grimoire-1.nancy.grid5000.fr','grimoire-2.nancy.grid5000.fr')" -l /host=2+{"type='disk'"}/host=2/disk=1</code>}}
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -p "host in ('grimoire-1.nancy.grid5000.fr','grimoire-2.nancy.grid5000.fr')" -l /host=2+{"type='disk'"}/host=2/disk=1</code>}}
{{Note|text=Yes, the syntax of the last oarsub command is a bit awkward, so please be careful and mind having:
{{Note|text=Yes, the syntax of the last oarsub command is a bit awkward, so please be careful and mind having:
Line 74: Line 72:


=== Second: reserve the nodes ===
=== Second: reserve the nodes ===
{{Note|text=In the following example, remove the <code>-t deploy</code> if you plan to use the standard environment and <code class="command">sudo-g5k</code> in order to get the ''root privileges'', instead of <code class="command">kadeploy</code> and be ''root'' in you own deployed environment.}}
You can then reserve nodes grimoire-1 and grimoire-2 for 3 hours, in the usual way:
You can then reserve nodes grimoire-1 and grimoire-2 for 3 hours, in the usual way:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -l {"host in ('grimoire-1.nancy.grid5000.fr', 'grimoire-2.nancy.grid5000.fr')"}/host=2,walltime=3</code>}}
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I -t deploy -l {"host in ('grimoire-1.nancy.grid5000.fr', 'grimoire-2.nancy.grid5000.fr')"}/host=2,walltime=3</code>}}
Line 80: Line 79:


= Show and use my reserved disks =
= Show and use my reserved disks =
Reservations of both nodes (processors) and disks are displayed on the following [https://intranet.grid5000.fr/oar/Nancy/drawgantt-svg/disk/ Gantt diagram].
== Gantt diagrams with disk reservations ==
Reservations of both nodes (processors) and disks are displayed on the following Gantt diagrams:
{|
|-
|bgcolor="#ffffff" valign="top" style="border:1px solid #cccccc;padding:1em;padding-top:0.5em;"|
[https://intranet.grid5000.fr/oar/Grenoble/drawgantt-svg-disks/ Grenoble]
|bgcolor="#ffffff" valign="top" style="border:1px solid #cccccc;padding:1em;padding-top:0.5em;"|
[https://intranet.grid5000.fr/oar/Lille/drawgantt-svg-disks/ Lille]
|bgcolor="#ffffff" valign="top" style="border:1px solid #cccccc;padding:1em;padding-top:0.5em;"|
[https://intranet.grid5000.fr/oar/Nancy/drawgantt-svg-disks/ Nancy]
|bgcolor="#ffffff" valign="top" style="border:1px solid #cccccc;padding:1em;padding-top:0.5em;"|
[https://intranet.grid5000.fr/oar/Rennes/drawgantt-svg-disks/ Rennes]
|}


Once connected to a node in a deploy job with one or more disks reserved locally on the node, several tools can be used to manage the disk(s):
== Getting information about disk reservations from OAR and G5K APIs ==
* <code class="command">lsblk</code> will show you the block devices of your disks: ''sdb'', ''sdc'', ... (be careful: ''sda'' is the system disk) ;
* The OAR API shows the properties of each resource of a job. You can retrieve the properties of your reserved disks, such as disk or diskpath:
* Commands like <code class="command">fdisk</code> <code class="command">parted</code> can be used to partition the disk, if need ;
{{Term|location=fnancy|cmd=<code class="command">curl</code> <code>https://api.grid5000.fr/stable/sites/</code><code class="replace">site</code><code>/internal/oarapi/jobs/</code><code class="replace">job_id</code><code>/resources.json</code> (or <code>resources.yaml</code>)}}
 
* The Grid'5000 API also provide some details about disk reservations under the '''"disks"''' key of the status and jobs APIs:
{{Term|location=fnancy|cmd=<code class="command">curl</code> <code>https://api.grid5000.fr/stable/sites/</code><code class="replace">site</code><code>/status &#124; json_pp</code>}}
{{Term|location=fnancy|cmd=<code class="command">curl</code> <code>https://api.grid5000.fr/stable/sites/</code><code class="replace">site</code><code>/jobs/</code><code class="replace">job_id</code><code> &#124; json_pp</code>}}
 
== Show disks once connected on the machines ==
Once connected to a node where you reserved one or more disks, several tools can be used to manage the disk(s):
* <code class="command">lsblk</code> will show you the block devices of your disks: ''sdb'', ''sdc'', ... (be careful: ''sda'' is the system disk);
* Commands like <code class="command">fdisk</code> or <code class="command">parted</code> can be used to partition the disk, if needed;
* <code class="command">mkfs</code> can be used to format the disk.
* <code class="command">mkfs</code> can be used to format the disk.


Mind that the platform provides access to the block devices, but does not manage partitioning nor formatting.
Mind that the platform provides access to the block devices. It does not manage partitioning nor formatting nor mounting.
 
= Getting information about disk reservations from the G5K API =
The Grid'5000 API in version sid provide some details about disk reservations under the '''"disks"''' label of the status and jobs APIs:
{{Term|location=fnancy|cmd=<code class="command">curl</code> <code>-k https://api.grid5000.fr/sid/sites/nancy/status &#124; json_pp</code>}}
{{Term|location=fnancy|cmd=<code class="command">curl</code> <code>-k https://api.grid5000.fr/sid/sites/nancy/jobs/</code><code class="replace">job_id</code><code> &#124; json_pp</code>}}


= Security issues =
= Security issues =
The mechanism used to enable/disable disks is designed to avoid mistakes from other users. However, a malicious user could take control of the RAID card, enable any disk, and access or erase your data. Please mind securing your data:
The mechanism used to enable/disable disks is designed to avoid mistakes from other users. However, a malicious user could take control of the RAID card, enable any disk, and access or erase your data. Please notify the [mailto:support-staff@lists.grid5000.fr Grid'5000 tech-team] in case of such event, but first of all mind securing your data:
* Keep a copy (backup) in a safe place if relevant for your data ;
* Keep a copy (backup) in a safe place if relevant for your data ;
* If your data is sensitive, mind using cryptographic mechanisms to secure it.
* If your data is sensitive, mind using cryptographic mechanisms to secure it.


Also, the data on reserved disks is not automatically erased at the end of your job. If you don't want the next user to access it, you have to erase it yourself.
Also, the data on reserved disks is not automatically erased at the end of your job. If you don't want the next user to access it, you have to erase it yourself.
Finally, no backup of data stored on the reserved disks is made.

Revision as of 18:27, 19 March 2019

Note.png Note

This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Disk reservation consists in reserving on nodes additional hard disks, which are otherwise not usable.

The table below shows the Grid'5000 clusters with such additional hard disks available for reservation.

Site Cluster Number of nodes Number of reservable disks per node
Grenoble yeti 4 3
Lille chiclet 8 2
Lille chifflot 8 5
Lyon gemini 2 4
Nancy grappe 16 1
Nancy gros 124 1
Nancy grouille 2 1
Rennes parasilo 27 5

Last generated from the Grid'5000 Reference API on 2024-02-26 (commit 8c6cab022b)

How it works

Two use cases of the disk reservsation are possible:

  • long run reservations of disks only (job reserving no host, i.e. no processing power): disk-only reservations do not have to fit in the day vs. night&week-end host reservation policy, and can last up to many days (see Grid5000:UsagePolicy). The reserved disks can then be used by regular host jobs during the period of time of the disk reservation. In this use case, the goal is to get more persistence for the local storage of nodes, e.g. avoid the need to reformat disks and reimport dataset in each regular host job. Those long run jobs must use the noop OAR job type.
  • regular jobs reserving both host and disks. In this use case, the goal is to get access to the reservable disks within the experiment, just as if the disk were not to reserve separately.

In both cases, making use of the reserved disks requires to gain the root privileges, since disks are provided as bare metal hardware to be partitioned, formated, mounted, filled with no restriction but by the experimenter. As a result, the experimenter can use the reserved disk:

  • either in a deploy job, in a kadeployed environment (use the deploy OAR job type, then kadeploy)
  • or in a non-deploy job, in the standard environment but after enabling sudo with the sudo-g5k command.

Technically speaking, when a deploy job starts, or whenever sudo-g5k is called in a non-deploy job, the reserved disks stay available while the other disks are disabled. Reserved disks can only be accessed by the user who reserved them.

Please note that reserved disks are not cleaned-up at the end of reservation. As a result:

  • Data let on the disks can be accessed by user in later reservations.
  • Reserved disk may first need to get cleaned-up before use (remove previous formating and partitioning)

See also Security issues.

Usage

The main commands to reserve disks are given below.

Note.png Note

Note that accessing the data stored on reserved disks on nodes is only possible with reservation of type deploy (oarsub -t deploy ...). Non-deploy jobs do not give access to disks (only the system disk).

The maximum duration of a disk reservation is defined in the Usage Policy.

Reserve disks and nodes at the same time

Note.png Note

In the 3 following examples, remove the -t deploy if you plan to use the standard environment and sudo-g5k in order to get the root privileges, instead of kadeploy and be root in you own deployed environment.

How to reserve a node with only the main disk (none of the additional disks), on the grimoire cluster
Terminal.png fnancy:
oarsub -I -t deploy -p "cluster='grimoire'" -l /host=1

(no change to the way a node was to be reserved in the past, before the disk reservation mechanism existed.)

How to reserve a node with all its disks
Terminal.png fnancy:
oarsub -I -t deploy -l {"(type='disk' or type='default') and cluster='grimoire'"}/host=1
How to reserve nodes grimoire-1 and grimoire-2 with one reservable disk per node
Terminal.png fnancy:
oarsub -I -t deploy -p "host in ('grimoire-1.nancy.grid5000.fr','grimoire-2.nancy.grid5000.fr')" -l /host=2+{"type='disk'"}/host=2/disk=1
Note.png Note

Yes, the syntax of the last oarsub command is a bit awkward, so please be careful and mind having:

  • the -p option explicitly set the hosts you want (using "cluster='grimoire'" instead could not insure that you get the disks on the same nodes you will reserve) ;
  • both host= values in the -l option (2 in the example) exactly match the count of hosts in the list you provide in the -p option (grimoire-1.nancy.grid5000.fr and grimoire-2.nancy.grid5000.fr in the example).
  • we do not need to explicitly write "{type='default'}" in the -l option (before the /host=2+, because default is implicit is the type is not set.
See Advanced OAR for more explanation of the oarsub syntax.

Reserve disks and nodes separately

You may, for example, decide to reserve some disks for one week, but the nodes where your disks are located only when you want to carry out an experiment.

First: reserve the disks

Since we want to reserve disks only in a first time, we use the noop job type: with this noop job type, OAR will not try to execute anything on the job resources (which is what we want since disk resources are not capable of executing programs).

(Please mind that Jobs of type noop cannot be interactive: oarsub-I -t noop ... is not supported.)

3 examples:

Reserve two disks on grimoire-1 for one week, starting on 2018-01-01:

Terminal.png fnancy:
oarsub -r "2018-01-01 00:00:00" -t noop -l {"type='disk' and host='grimoire-1.nancy.grid5000.fr'"}/host=1/disk=2,walltime=168

Or reserve the first two disks on grimoire-2:

Terminal.png fnancy:
oarsub -r "2018-01-01 00:00:00" -t noop -l {"type='disk' and host='grimoire-2.nancy.grid5000.fr' and disk in ('sdb.grimoire-2', 'sdc.grimoire-2')"}/host=1/disk=2,walltime=168

Or reserve all disks on two nodes:

Terminal.png fnancy:
oarsub -r "2018-01-01 00:00:00" -t noop -l {"type='disk' and cluster='grimoire'"}/host=2/disk=ALL,walltime=168

Second: reserve the nodes

Note.png Note

In the following example, remove the -t deploy if you plan to use the standard environment and sudo-g5k in order to get the root privileges, instead of kadeploy and be root in you own deployed environment.

You can then reserve nodes grimoire-1 and grimoire-2 for 3 hours, in the usual way:

Terminal.png fnancy:
oarsub -I -t deploy -l {"host in ('grimoire-1.nancy.grid5000.fr', 'grimoire-2.nancy.grid5000.fr')"}/host=2,walltime=3

You must respect this order : reserve the disks first, then reserve the nodes. Otherwise the disks you reserved will not be available on your nodes.

Show and use my reserved disks

Gantt diagrams with disk reservations

Reservations of both nodes (processors) and disks are displayed on the following Gantt diagrams:

Grenoble

Lille

Nancy

Rennes

Getting information about disk reservations from OAR and G5K APIs

  • The OAR API shows the properties of each resource of a job. You can retrieve the properties of your reserved disks, such as disk or diskpath:
Terminal.png fnancy:
curl https://api.grid5000.fr/stable/sites/site/internal/oarapi/jobs/job_id/resources.json (or resources.yaml)
  • The Grid'5000 API also provide some details about disk reservations under the "disks" key of the status and jobs APIs:
Terminal.png fnancy:
curl https://api.grid5000.fr/stable/sites/site/status | json_pp
Terminal.png fnancy:
curl https://api.grid5000.fr/stable/sites/site/jobs/job_id | json_pp

Show disks once connected on the machines

Once connected to a node where you reserved one or more disks, several tools can be used to manage the disk(s):

  • lsblk will show you the block devices of your disks: sdb, sdc, ... (be careful: sda is the system disk);
  • Commands like fdisk or parted can be used to partition the disk, if needed;
  • mkfs can be used to format the disk.

Mind that the platform provides access to the block devices. It does not manage partitioning nor formatting nor mounting.

Security issues

The mechanism used to enable/disable disks is designed to avoid mistakes from other users. However, a malicious user could take control of the RAID card, enable any disk, and access or erase your data. Please notify the Grid'5000 tech-team in case of such event, but first of all mind securing your data:

  • Keep a copy (backup) in a safe place if relevant for your data ;
  • If your data is sensitive, mind using cryptographic mechanisms to secure it.

Also, the data on reserved disks is not automatically erased at the end of your job. If you don't want the next user to access it, you have to erase it yourself.

Finally, no backup of data stored on the reserved disks is made.