Deploying and Using IaaS Clouds on Grid'5000

From Grid5000
Jump to: navigation, search


Contact for Nimbus:

  • Pierre Riteau, PhD student, Université de Rennes 1, IRISA, Myriads Team | mail | Webpage

Contact for OpenNebula:

  • Houssem Eddine Chihoub, PhD student, INRIA Rennes - Bretagne Atlantique, KerData team | mail | Webpage


Contents

Introduction

The goal of this practical session is to introduce users to the deployment and usage of popular IaaS toolkits on Grid'5000. In the first part, users will be presented with a general overview of two IaaS toolkits, Nimbus and OpenNebula. They will learn how to deploy these two systems on Grid'5000, using scripts developed by members of the Grid'5000 community. After interacting with each cloud separately using their native tools, they will provision resources programmatically using the EC2 API.

General overview of Nimbus and OpenNebula

Nimbus and OpenNebula are both Infrastructure-as-a-Service toolkits. They allow users to build computing infrastructures from virtualized resources. In practice, users contact the cloud service and request virtual machines on which they have full administrative access, allowing them to run any kind of software they want. Both Nimbus and OpenNebula are architectured similarly. A frontend node runs the main cloud service. This service receives user requests, schedules virtual machines, does authentication and accounting, etc. Other nodes in the cloud are dedicated to execute virtual machines: we refer to them as hypervisors or VMMs (Virtual Machine Managers).

Nimbus Architecture

As explained above, the main service of Nimbus runs on a dedicated node. Two components implement this service: the IaaS service, which answers user requests to create and terminate virtual machines, and the Cumulus storage service, which handles all requests related to virtual machine images (download & upload). Users interact with these two components using HTTP. The Nimbus IaaS service exposes its own web service API, or an Amazon EC2-compatible API. Cumulus implements the Amazon S3 API.

VMM nodes run a program called workspace-control, which interacts with the libvirt library to manage Xen and KVM virtual machines. They also run a DHCP server managed by Nimbus in order to serve IP addresses to virtual machines.

(thumbnail)
Figure 1: Nimbus Architecture

OpenNebula Architecture

OpenNebula provides a set of Tools. These are management tools developed using the interfaces provided by the OpenNebula Core, such as the CLI (command line interface), the scheduler, the libvirt API implementation or the Cloud RESTful interfaces.

The OpenNebula Core, which include the main virtual machine, storage, virtual network and host management components, controls and monitors virtual machines, virtual networks, storage and hosts. The core performs its actions (e.g. monitor a host, or cancel a VM) by invoking a suitable driver.

Drivers, are used to plug-in different virtualization, storage and monitoring technologies and Cloud services into the core.

(thumbnail)
Figure 2: OpenNebula Architecture

Grid'5000 specificities

When we deploy Nimbus and OpenNebula on Grid'5000, we are in charge of configuring the IP network used by virtual machines. We are not using the network isolation capabilities of Grid'5000, so we have to make sure all simultaneous cloud deployments use different IP networks. To solve this problem, a subnet reservation system is available. We can reserve one or several IP networks, ranging from /22 to /16 subnets, allocated inside the Grid'5000 virtual network.

Nimbus

Nimbus deployment

First, we connect to a frontend, for instance Rennes :

(local)$ ssh rennes.g5k

Pre-requisites: Complementary Ruby Gems

Several Ruby gems (gosen, json, net-ssh, net-scp and restfully) are required by the Nimbus deployment script.

To do that as a regular user, one approach is to alter the GEM_HOME variable to point to a directory in your homedir.

$ export GEM_HOME=$HOME/.gem 
$ gem install gosen json net-ssh net-scp restfully

Note: you can also use gem install --user-install gosen json net-ssh net-scp restfully

Pre-requisites: SSH internal key

The script also assumes that you have an SSH key internal to Grid'5000, which can be used to connect to the frontend. If it is not the case, create a new SSH key by running the following (don't enter any passphrase, just type Enter):

$ ssh-keygen -q
Enter file in which to save the key (/home/login/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again:

Then, make it possible to connect to the frontend using this SSH key:

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Initial deployment

Now that our environment is ready, we do a resource reservation. We ask OAR for 4 nodes with hardware virtualization capabilities, and one /22 subnet (1022 IP addresses).

$ oarsub -l {"virtual!='none'"}nodes=4+slash_22=1,walltime=2 -t deploy 'sleep 86400'
[ADMISSION RULE] Modify resource description with type constraints
[ADMISSION RULE] Modify resource description as subnet
Generate a job key...
OAR_JOB_ID=381465

Take a note of the OAR job ID that was given to you, in this case 381465.

The Nimbus deployment script supports multiple job reservations. To describe these reservations, it gets its input from a YAML file provided by the user. We are going to use the script with only one reservation, but we still need to create the file. Open a text editor and create a file containing this content, replacing highlighted values by the ones matching your job. For the rest of this tutorial, we suppose the file is named jobs.yml and located in your home directory.

--- 
- hypervisor: kvm
  uid: 381465
  site: rennes

Now, we are ready to run the deployment script:

$ /grid5000/code/staging/nimbus-deploy/deploy_nimbus_cloud.rb ~/jobs.yml
I, [2011-04-07T14:36:42.910772 #7094]  INFO -- : Kadeploy run 1 with 4 nodes (0 already deployed, need 3 more)
I, [2011-04-07T14:42:14.814027 #7094]  INFO -- : Nodes deployed: paradent-46.rennes.grid5000.fr paradent-48.rennes.grid5000.fr paradent-49.rennes.grid5000.fr paradent-5.rennes.grid5000.fr
I, [2011-04-07T14:42:14.814537 #7094]  INFO -- : Had to run 1 kadeploy runs, deployed 4 nodes
I, [2011-04-07T14:42:28.132897 #7094]  INFO -- : Broadcasting node configurations
I, [2011-04-07T14:42:30.073163 #7094]  INFO -- : Configuring and installing Nimbus (please wait)
I, [2011-04-07T14:45:15.575224 #7094]  INFO -- : Installation finished

Nimbus cloud in rennes:
- service node is paradent-46.rennes.grid5000.fr
- client node is paradent-48.rennes.grid5000.fr (connect with "ssh nimbus@paradent-48.rennes.grid5000.fr")

The script deploys and configures a Nimbus cloud. When it is finished it prints details about the deployment: what is the service node, and what is the client node where the Nimbus cloud client is installed and ready to use.

Nimbus usage

There are several ways to interact with a Nimbus cloud: the cloud client (a simple and high level client), the reference client (a client offering low level access to the WSRF frontend), and the EC2 API. For this part of the tutorial, we are going to use the cloud client.

This client is installed and pre-configured on a dedicated node, listed at the end of the deployment. First, connect to it using the nimbus account (replace the hostname by the one used in your deployment)

$ ssh nimbus@paradent-48.rennes.grid5000.fr

The cloud client (version 019) is installed in /nimbus/nimbus-cloud-client-019. Change to this directory:

$ cd nimbus-cloud-client-019

Nimbus relies on the Globus Java WS Core for its service implementation. As such, we need to create a certificate proxy to be authenticated with the service stack. Run the following command:

$ ./bin/grid-proxy-init.sh
Your identity: C=US,ST=Several,L=Locality,O=Example,OU=Operations,CN=Pierre.Riteau@irisa.fr
Creating proxy, please wait...
Proxy verify OK
Your proxy is valid until Fri Apr 08 23:58:05 CEST 2011

We can now use the Nimbus cloud client. First, let's list the virtual machine images available in the Cumulus repository:

$ ./bin/cloud-client.sh --list
No files.

Of course, there is no available images: our cloud is brand new! We are going to upload a KVM image using the Debian squeeze operating system.

$ ./bin/cloud-client.sh --transfer --sourcefile /home/priteau/squeeze-vm.raw

Transferring
  - Source: squeeze-vm.raw
  - Destination: cumulus://Repo/VMS/620c4a30-90ff-11df-b85b-002332c9497e/squeeze-vm.raw

Preparing the file for transfer:
  1255MB [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 100% 

Transferring the file:
  1255MB [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 100% 

Done.

If we list available images again, we can see our image was uploaded successfully.

$ ./bin/cloud-client.sh --list
[Image] 'squeeze-vm.raw'                   Read/write
        Modified: Apr 8 2011 @ 12:00   Size: 1316044800 bytes (~1255 MB)

We are now ready to start our first virtual machine. This is done using the cloud client too:

$ ./bin/cloud-client.sh --run --hours 2 --name squeeze-vm.raw

Launching workspace.

Workspace Factory Service:
    https://parapluie-18.rennes.grid5000.fr:8443/wsrf/services/WorkspaceFactoryService

Creating workspace "vm-001"... done.


       IP address: 10.158.12.1
         Hostname: rennes-vm1
       Start time: Fri Apr 08 13:45:16 CEST 2011
    Shutdown time: Fri Apr 08 15:45:16 CEST 2011
 Termination time: Fri Apr 08 15:55:16 CEST 2011

Waiting for updates.


"vm-001" reached target state: Running

Running: 'vm-001'

Take note of the VM identifier that was assigned, in this case vm-001. It will be used later to interact with the service. The cloud client also reports the IP address that was allocated to your VM, in this case 10.158.12.1. We use this IP address to connect to the VM, using SSH. No password is required since Nimbus takes care of uploading an SSH key during the VM provisioning.

$ ssh root@10.158.12.1
Warning: Permanently added '10.158.12.1' (RSA) to the list of known hosts.
Linux nimbusvm-1 2.6.26-2-amd64 #1 SMP Wed Aug 19 22:33:18 UTC 2009 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Mar 30 15:24:21 2010 from 10.156.0.1

We are now connected to a VM with full administrative access: we can install packages, run any applications, etc.

To destroy the VM, we run the following command:

$ ./bin/cloud-client.sh --terminate --handle vm-001

Terminating workspace.
  - Workspace handle (EPR): '/nimbus/nimbus-cloud-client-019/history/vm-001/vw-epr.xml'

Destroying vm-001... destroyed.

OpenNebula

OpenNebula deployment

The scripts to automatically deploy OpenNebula using the provided images can be found here:

/home/priteau/bsScripts/

The content of the openNebula directory is the following:

  • config: directory that contains the configuration scripts for deploying openNebula.
  • env.sh: script with all the global variables necessary for OpenNebula deployment.

In the config directory you will find all the necessary scripts for deploying both OpenNebula Frontend and nodes. The main script is auto-config.sh and it will be called by the launching script.

The ONConfigureFrontend.sh, ONConfigureCloud.sh, and ONConfigureNode.sh will copy and then call internal scripts on and from the newly deployed ON Frontend and ON nodes in order to make the necessary settings. The configuration of your SSH keys is needed for this operation.

Do a reservation on G5K

Copy the scripts in your home directory:

$ cp -R /home/priteau/bsScripts .

Reserve nodes with hardware virtualization capabilities, and one /22 subnet (1022 IP addresses):

$ oarsub -l {"virtual!='none'"}nodes=3+slash_22=1,walltime=2 -t deploy 'sleep 86400'
[ADMISSION RULE] Modify resource description with type constraints
[ADMISSION RULE] Modify resource description as subnet
Generate a job key...
OAR_JOB_ID=1186545

Connect to your job:

$ oarsub -C OAR_JOB_ID

Specify the number of ON nodes (this does not include the ON frontend) in the global env.sh of OpenNebula scripts (in the directory ~/bsScripts/openNebula). If you reserved and got exactly three nodes, you don't need to change anything:

# The number of machines that need to be deployed
export NO_ON_NODES=2
Warning.png Warning

As seen in env.sh file, only one node will be the frontend. It is required to specify the number of the nodes that you want to deploy also in the env.sh variable. Make sure that the number of deployed nodes is not > number of nodes from the reservation - 1 (the frontend). Else, the deployment will fail.

Run the configuration and deploy scripts

Go to the directory ~/bsScripts/launchDepl and run the config-all.sh script:

$ sh config_all.sh

Your cloud should be deployed in a few minutes!

OpenNebula usage

Further configuration

Configuration scripts will be copied and run on the newly deployed images, you can find them in the following directories:

  • /srv/cloud/on_utils on the OpenNebula Frontend image (there is a script for configuring the Frontend, doing the necessary settings and another one for configuring and setting the cloud).
  • /srv/cloud/on_utils on the OpenNebula Node image.

You can now login to your opennebula cloud frontend :

$ ssh oneadmin@frontend

Build the Cloud Virtual Network

On the oneadmin home on the frontend:

$ cd $HOME/one-templates

You can find OpenNebula templates (configuration files) for the virtual network, the virtual machine image, and the virtual machine.

There are many ways to build a virtual network in OpenNebula (for more information please see [1]). Our virtual network uses fixed leases. The template file (on-network.net) should contain something similar to the following content:

NAME = "OpenNebula network"
TYPE = FIXED

#Now we'll use the cluster private network (physical)
BRIDGE = br0
LEASES = [ IP="10.156.0.51"]
LEASES = [ IP="10.x.x.x"]
LEASES = [ IP="10.x.x.x"]
...

Note here that the IP adresses are gotten from the subnet reservation and written in the template file.

The NAME is the name of the fixed virtual network, the TYPE is 'fixed' in our case. The bridge br0 is the brigde configured on OpenNebula node. The LEASES are a list of IPs to be assigned to the deployed virtual machines.

Create the virtual network :

$ onevnet create $HOME/one-templates/on-network.net

The output of onevnet list should look like this:

ID USER     NAME              TYPE BRIDGE P #LEASES
 0 oneadmin on-network  Fixed    br0 N       0

Deploy a virtual machine on our OpenNebula cloud

Now we want to deploy an instance of our virtual machine image "squeeze-vm". The image is already registered in the OpenNebula image repository. The template for squeeze-vm is squeeze-vm.one and looks something like :

NAME   = squeeze-vm
CPU    = 2
MEMORY = 512

DISK   = [ image     = "squeeze-vm-image" ]

NIC    = [ NETWORK = "OpenNebula network" ]

FEATURES=[ acpi="no" ]

CONTEXT = [
   hostname    = "$NAME",
   ip_public   = "$NIC[IP, NETWORK=\"Opennebula network\"]",
   files      = "/srv/cloud/one/one-templates/init.sh /srv/cloud/one/one-templates/id_rsa.pub /srv/cloud/one/one-templates/test.txt",
   target      = "hdc",
   root_pubkey = "id_rsa.pub",
   username    = "opennebula",
   user_pubkey = "id_rsa.pub"
]

The variables provide informations about the virtual machine in the OpenNebula Cloud context (name, CPU used, maximum memory, from where to take the image or what virtual network to use). For more information about the template file setting, please see [2].

Create and deploy a squeeze-vm virtual machine:

$ onevm create $HOME/one-templates/squeeze-vm.one

The output of onevm list should look like this:

ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
 9 oneadmin squeeze-vm runn   0      0K      paradent-2 00 00:00:24

That means we have a running squeeze-vm virtual machine on host paradent-2.

To show more information about the virtual machine run:

$ onevm show ID

Among these informations note the IP address! You can access your virtual machine by simply running (the password should be grid5000):

$ ssh root@IP-address-of-the-VM

To delete the VM:

$ onevm delete ID

Connect to your virtual machine using vnc client

Add the following line in the template (squeeze-vm.one)

GRAPHICS = [ type = "vnc", listen = "0.0.0.0", port = "5938"]

Create and deploy a new squeeze-vm virtual machine instance:

$ onevm create $HOME/one-templates/squeeze-vm.one

The output of onevm list should look like this:

  ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
   3 oneadmin squeeze-vm runn   0    512M griffon-38.nanc 00 00:11:04


It means that we have a running squeeze-vm virtual machine on host griffon-38.nancy.grid5000.fr.

To connect to your VM using your local vnc client you have to establish a ssh tunnel. So, from your local machine type

$ ssh -L 15938:griffon-38.nancy.grid5000.fr:5938 access.lyon.grid5000.fr

Now you could connect to your VM using vnc client on localhost:15938

Using clouds programmatically

To use clouds without manual interaction with the system, we can take advantage of APIs offered by clouds. In this section, as an example, we are going to script the deployment of a cluster of machines on Nimbus, using the EC2 API.

The boto library

boto is a Python library to easily access the AWS APIs, including EC2. First, we need to install it on a frontend:

Installing boto

$ mkdir -p $HOME/local/lib/python2.5/site-packages/
$ tar xzf /home/priteau/boto-2.0b4.tar.gz
$ cd boto-2.0b4
$ export PYTHONPATH=$HOME/local/lib/python2.5/site-packages/
$ python setup.py install --prefix=$HOME/local

Writing a boto script to access Nimbus

Connecting to the cloud

To connect to the cloud, we set up our credentials and initiate connection objects:

access_id = "vIKOIQa5tduXorYRxctyh"
access_secret = "TOZcDmPNFYPii8UTcGXrGiRaW4RNfzm2TcTvqZaNCz"
canonical_id = "620c4a30-90ff-11df-b85b-002332c9497e"
host = options.service
port = 8444
cumulusport = 8888

region = RegionInfo(name="nimbus", endpoint=host)
ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=port)
cf = OrdinaryCallingFormat()
s3conn = S3Connection(access_id, access_secret, host=host, port=cumulusport, is_secure=False, calling_format=cf)

Uploading a VM image to the image repository

This code shows how to upload a VM image to an S3-compatible image repository such as Cumulus:

# upload the file according to the Nimbus naming convention
bucket = s3conn.get_bucket("Repo")

print "Uploading the VM image to Cumulus"
k = boto.s3.key.Key(bucket)
k.key = "VMS/%s/%s" % (canonical_id, 'squeeze-vm.raw')
if not k.exists():
    k.set_contents_from_filename("/home/priteau/%s" % 'squeeze-vm.raw')

Uploading a public key to the cloud

We can also upload a private key to be able to connect to the VMs without password:

# Upload our SSH public key to the cloud
f = open(os.path.expanduser('~/.ssh/id_rsa.pub'), 'r')
ssh_public_key = f.read()
f.close()
ec2conn.create_key_pair("gsg-keypair||%s" % ssh_public_key)

Starting a VM cluster

Finally, we can start a VM cluster (options.number is the number of VMs to run):

# now run the VMs
reservation_nodes = ec2conn.run_instances('squeeze-vm.raw', min_count=options.number, max_count=options.number, key_name="gsg-keypair")
instances_nodes = reservation_nodes.instances

The full script

Now that we've seen how to interact with the EC2 API, let's try a full example. Copy the following script to ~/ec2.py. Don't forget to run chmod +x ~/ec2.py. Then you can run it using the following command (replace the node name by your own Nimbus service node):

~/ec2.py -s SERVICE_NODE_NAME -n 2

The -n option specifies the number of VMs to start.

#!/usr/bin/env python

import boto
from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection
from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo

import os, sys
from optparse import OptionParser
import shlex, subprocess
import tempfile
import time

parser = OptionParser()
parser.add_option("-s", "--service", dest="service",
                  help="hostname of the Nimbus service", metavar="HOST")
parser.add_option("-n", "--number", dest="number",
                  help="number of VMs to deploy", metavar="NUMBER")

(options, args) = parser.parse_args()

if not options.service or not options.number:
    print "usage: ec2.py --service HOST --number NUMBER"
    sys.exit(1)

access_id = "vIKOIQa5tduXorYRxctyh"
access_secret = "TOZcDmPNFYPii8UTcGXrGiRaW4RNfzm2TcTvqZaNCz"
canonical_id = "620c4a30-90ff-11df-b85b-002332c9497e"
host = options.service
port = 8444
cumulusport = 8888

region = RegionInfo(name="nimbus", endpoint=host)
ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=port)
cf = OrdinaryCallingFormat()
s3conn = S3Connection(access_id, access_secret, host=host, port=cumulusport, is_secure=False, calling_format=cf)

# upload the file according to the Nimbus naming convention
bucket = s3conn.get_bucket("Repo")

print "Uploading the VM image to Cumulus"
k = boto.s3.key.Key(bucket)
k.key = "VMS/%s/%s" % (canonical_id, 'squeeze-vm.raw')
if not k.exists():
    k.set_contents_from_filename("/home/priteau/%s" % 'squeeze-vm.raw')

# Upload our SSH public key to the cloud
f = open(os.path.expanduser('~/.ssh/id_rsa.pub'), 'r')
ssh_public_key = f.read()
f.close()
ec2conn.create_key_pair("gsg-keypair||%s" % ssh_public_key)

# now run the VMs
reservation_nodes = ec2conn.run_instances('squeeze-vm.raw', min_count=options.number, max_count=options.number, key_name="gsg-keypair")
instances_nodes = reservation_nodes.instances

for instance in instances_nodes:
    instance.update()
    while instance.state == 'pending':
        print 'instance is %s' % instance.state
        time.sleep(10)
        instance.update()
    if instance.state != 'running':
        print 'error: instance is in state %s' % instance.state
        sys.exit(1)

for instance in instances_nodes:
   print instance.ip_address

nodes = tempfile.mkstemp()
f = os.fdopen(nodes[0], 'w')
for instance in instances_nodes:
     f.write("%s\n" % instance.ip_address.encode())
f.close()

print "Sleeping for 60 seconds for the VMs to boot"
time.sleep(60)
Personal tools
Namespaces

Variants
Actions
Public Portal
Users Portal
Admin portal
Wiki special pages
Toolbox