Ceph

See also: Storage overview | Storage Manager | Group Storage | Ceph | Local (on nodes) disks reservation

	Note
	This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability. - See more at: http://ceph.com/

This pages gives information with regards to the managed Ceph service provided by Grid'5000.

About Ceph

The following documentation show the architecture: https://docs.ceph.com/en/latest/architecture/

Grid'5000 Deployment

Sites	Size	Configuration	Rados	RBD	CephFS	RadosGW
Rennes	~ 9TB	16 OSDs on 4 nodes
Nantes	~ 7TB	12 OSDs on 4 nodes

Configuration

Generate your key

In order to access to the object store you will need a Cephx key. See : https://api.grid5000.fr/sid/sites/rennes/storage/ceph/ui/

Click on Generate a Cephx Key. A new key is generated.

Your key is now available from the frontends :

frontend:

curl -k https://api.grid5000.fr/sid/sites/rennes/storage/ceph/auths/jdoe.keyring

[client.jdoe]
  key = AQBwknVUwAPAIRAACddyuVTuP37M55s2aVtPrg==

Note : Replace jdoe by your login and rennes by nantes if your experiment is at Nantes.

Create/Update/Delete Ceph pools

Requirement : Generate your key

Manage your Ceph pools from the Grid'5000 Ceph frontend :

• https://api.grid5000.fr/sid/sites/rennes/storage/ceph/ui/
• https://api.grid5000.fr/sid/sites/nantes/storage/ceph/ui/

Currently each users are limited to 4 pools.

Replication size

• 1 : No replication (not secured, most efficient for write operations, not recommended : one disk crash and all datas are lost)
• n : One primary object + n-1 replicas (more security, less efficient for write operations)

	Note
	See https://ceph.io/wp-content/uploads/2016/08/weil-thesis.pdf (page 130)

You can edit the replication size by clicking on the replication size value. If the new value is greater than the old value, objects will automatically be replicated. Depending the amount of objects this operation will stress the Ceph cluster and take more or less time.

Quota

Quota is currently limited to 2TB per pools. You can change this quota at any time by clicking on the quota value, so prefer to use a small/realistic quota first.

Expiration date

The default expiration date is limited to 2 months. You can extend the expiration date at any time by clicking on the expiration date value.

Configure Ceph on clients

On the frontend or a node with standard environment

Create a ceph configuration file ~/.ceph/config :

• For Nantes :

[global]
mon host = ceph0,ceph1,ceph2
keyring = /home/<login>/.ceph/ceph.client.<login>.keyring

• For Rennes :

[global]
mon host = ceph0,ceph1,ceph2
keyring = /home/<login>/.ceph/ceph.client.<login>.keyring

Create a ceph keyring file ~/.ceph/ceph.client.$USER.keyring with your keyring :

frontend:

curl -k https://api.grid5000.fr/sid/sites/rennes/storage/ceph/auths/$USER.keyring > ~/.ceph/ceph.client.$USER.keyring

Note : Replace rennes by nantes if your are at Nantes.

export CEPH_CONF=~/.ceph/config
export CEPH_ARGS="--id $USER"

Note : maybe a good idea to export CEPH_CONF and CEPH_ARGS in your ~/.bashrc or ~/.zshrc file.

Test

$ ceph health
HEALTH_OK
$ ceph osd tree
# id	weight	type name	up/down	reweight
-1	8.608	root default
-6	8.608		datacenter rennes
-2	2.152			host ceph0
0	0.538				osd.0	up	1
1	0.538				osd.1	up	1
2	0.538				osd.2	up	1
3	0.538				osd.3	up	1
-3	2.152			host ceph1
4	0.538				osd.4	up	1
5	0.538				osd.5	up	1
6	0.538				osd.6	up	1
7	0.538				osd.7	up	1
-4	2.152			host ceph2
8	0.538				osd.8	up	1
9	0.538				osd.9	up	1
10	0.538				osd.10	up	1
11	0.538				osd.11	up	1
-5	2.152			host ceph3
12	0.538				osd.12	up	1
14	0.538				osd.14	up	1
15	0.538				osd.15	up	1
13	0.538				osd.13	up	1

Usage

Rados Object Store access

Requierement : Create a Ceph pool • Configure Ceph on client

From command line

Put an object into a pool

node:

rados --pool jdoe_objects put <objectname> <infile>

List objects of a pool

node:

rados --pool jdoe_objects ls

Get object from a pool

node:

rados --pool jdoe_objects get <objectname> <outfile>

Remove an object

node:

rados --pool jdoe_objects rm <objectname>

Usage informations

node:

rados --pool jdoe_objects df

pool name       category                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
jdoe_objects -                    1563027            2            0            0           0            0            0          628      2558455
  total used       960300628       295991
  total avail     7800655596
  total space     9229804032

From your application (C/C++, Python, Java, Ruby, PHP...)

See : https://docs.ceph.com/en/latest/rados/api/librados-intro/?highlight=librados

Rados benchmarks

On a 1G ethernet client

frontend:

oarsub -I -p parapluie

Write (16 concurrent operations)

node:

rados --pool pmorillo_bench bench 60 write --no-cleanup

  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    0       0         0         0         0         0         -         0
    1      16        39        23   91.9669        92   0.91312  0.429879
    2      16        70        54   107.973       124  0.679348  0.458774
    3      16        96        80   106.644       104  0.117995   0.48914
    4      16       126       110    109.98       120  0.207923  0.520003
    5      16       154       138    110.38       112  0.370721  0.521431
    6      16       180       164   109.314       104  0.293496  0.533345
...
Total time run:        60.321552
Total reads made:     1691
Read size:            4194304
Bandwidth (MB/sec):    112.132 

Average Latency:       0.569955
Max latency:           4.06499
Min latency:           0.055024

Read

node:

rados --pool pmorillo_bench bench 60 seq

Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds.
Object prefix: benchmark_data_parapluie-31.rennes.grid5000._25776
  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    0       0         0         0         0         0         -         0
    1      16        39        23     91.98        92  0.553582  0.434956
    2      16        66        50   99.9807       108  0.471024  0.486303
    3      16        95        79   105.314       116  0.654954  0.510875
    4      16       124       108   107.981       116  0.522723  0.535851
    5      16       151       135   107.981       108  0.244908  0.547624
    6      16       180       164   109.314       116  0.239384   0.54942
...
Total time run:         60.478634
Total writes made:      1694
Write size:             4194304
Bandwidth (MB/sec):     112.040 

Stddev Bandwidth:       15.9269
Max bandwidth (MB/sec): 124
Min bandwidth (MB/sec): 0
Average Latency:        0.571046
Stddev Latency:         0.354297
Max latency:            2.62603
Min latency:            0.165878

On a 10G ethernet client

frontend:

oarsub -I -p paranoia

Write (16 concurrent operations)

• Pool replication size : 1

node:

rados --pool pmorillo_bench bench 60 write --no-cleanup

Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds.
Object prefix: benchmark_data_paranoia-3.rennes.grid5000.fr_5626
  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    0       0         0         0         0         0         -         0
    1      16       146       130   519.915       520  0.488794  0.103537
    2      16       247       231   461.933       404  0.164705  0.126076
    3      16       330       314   418.612       332  0.036563  0.148763
    4      16       460       444   443.941       520  0.177378  0.141696
...
Total time run:         60.579488
Total writes made:      5965
Write size:             4194304
Bandwidth (MB/sec):     393.863 

Stddev Bandwidth:       83.789
Max bandwidth (MB/sec): 520
Min bandwidth (MB/sec): 0
Average Latency:        0.162479
Stddev Latency:         0.195071
Max latency:            1.7959
Min latency:            0.033313

Read

node:

rados --pool pmorillo_bench bench 60 seq

  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    0       0         0         0         0         0         -         0
    1      16       288       272   1087.78      1088  0.028679 0.0556937
    2      16       569       553   1105.82      1124  0.049299 0.0569641
    3      16       850       834   1111.85      1124  0.024369 0.0566906
    4      16      1127      1111   1110.86      1108  0.024855 0.0570628
    5      16      1406      1390   1111.86      1116  0.182567 0.0568681
...
Total time run:        21.377795
Total reads made:     5965 
Read size:            4194304
Bandwidth (MB/sec):    1116.111 

Average Latency:       0.0573023
Max latency:           0.270447
Min latency:           0.011937

RBD (Rados Block Device)

Requierement : Create a Ceph pool • Configure Ceph on client

Create a Rados Block Device

frontend:

rbd --pool jdoe_rbd create <rbd_name> --size <MB>

	Warning
	Ceph server release is older than frontend. For the moment we have to force "--image-format=1". Which create the warning "rbd: image format 1 is deprecated"

frontend:

rbd --pool jdoe_rbd ls

frontend:

rbd --pool jdoe_rbd info <rbd_name>

Exemple :

$ rbd -p pmorillo_rbd create datas --size 4096
$ rbd -p pmorillo_rbd ls
datas
$ rbd -p pmorillo_rbd info datas
rbd image 'datas':
	size 4096 MB in 1024 objects
	order 22 (4096 kB objects)
	block_name_prefix: rb.0.604e.238e1f29
	format: 1

Note : rbd -p pmorillo_rbd info datas is equivalent to rbd info pmorillo_rbd/datas

Create filesystem and mount RBD

frontend:

oarsub -I -t deploy

frontend:

kadeploy3 -e debian11-x64-base -f $OAR_NODEFILE -k

frontend:

ssh root@deployed_node

node:

apt install ceph

node:

modprobe rbd

node:

modinfo rbd

Create Ceph configuration file /etc/ceph/ceph.conf :

[global]
mon host = ceph0,ceph1,ceph2

Get CephX key from the frontend :

frennes $ curl -k https://api.grid5000.fr/sid/sites/rennes/storage/ceph/auths/$USER.keyring | ssh root@deployed_node "cat - > /etc/ceph/ceph.client.$USER.keyring"

node:

rbd --id jdoe --pool jdoe_pool map <rbd_name>

Exemple :

rbd --id pmorillo map pmorillo_rbd/datas

node:

rbd --id jdoe --pool jdoe_pool showmapped

Exemple :

rbd --id pmorillo showmapped
id pool         image snap device
0  pmorillo_rbd datas -    /dev/rbd0

node:

mkfs.ext4 /dev/rbd0

node:

mkdir /mnt/rbd

node:

mount /dev/rbd0 /mnt/rbd

node:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        15G  1.6G   13G  11% /
...
/dev/sda5       525G   70M  498G   1% /tmp
/dev/rbd0       3.9G  8.0M  3.6G   1% /mnt/rbd

Note : To map rbd at boot time, use /etc/ceph/rbdmap file :

# RbdDevice		Parameters 
#poolname/imagename	id=client,keyring=/etc/ceph/ceph.client.keyring

Resize, snapshots, copy, etc...

See :

• http://ceph.com/docs/master/rbd/rados-rbd-cmds/
• http://ceph.com/docs/master/man/8/rbd/#examples
• http://ceph.com/docs/master/rbd/rbd-snapshot/
• rbd -h

QEMU/RBD

Requierement : Create a Ceph pool • Configure Ceph on client

Convert a qcow2 file into RBD

frennes:

oarsub -I

node:

qemu-img convert -f qcow2 -O raw /grid5000/virt-images/debian11-x64-base.qcow2 rbd:pool_name/debian11:id=jdoe

node:

rbd --id jdoe --pool pool_name ls

debian11

Start KVM virtual machine from a Rados Block Device

node:

screen kvm -m 1024 -drive format=raw,file=rbd:pool_name/debian11:id=jdoe -nographic