Booting and Using Virtual Machines on Grid'5000: Difference between revisions

From Grid5000
Jump to navigation Jump to search
mNo edit summary
No edit summary
 
(8 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Maintainer|None}}
{{See also| [[https://www.grid5000.fr/mediawiki/index.php/Hemera:SC:Virtualization | Large Scale Management of Virtual Machines(HEMERA)]] }}
 
{{Maintainer|Daniel Balouek}}
{{Status|In production}}
{{Author|Daniel Balouek}}
{{Author|Daniel Balouek}}
{{Portal|Admin}}
{{Portal|Admin}}
Line 24: Line 27:
= Deployment =
= Deployment =


== Booking the ressources ==
== Booking the resources ==


The first step is to retrieve the last version of the scripts
The first step is to retrieve the last version of the scripts (the dynVM__TEMPLATE__ is located on sophia site)
{{Term|location=frontend|cmd=<code class="command">scp -r /home/alebre/dynVM__TEMPLATE__/ '''DESTINATION_FOLDER''' </code>}}
{{Term|location=frontend|cmd=<code class="command">scp -r /home/alebre/dynVM__TEMPLATE__/ '''DESTINATION_FOLDER''' </code>}}


Line 40: Line 43:


This script will return you an OAR that includes the reservation of the nodes, a virtual network and a subnet
This script will return you an OAR that includes the reservation of the nodes, a virtual network and a subnet
{{Term|location=frontend|cmd=<code class="command"> ./getmaxgridnodes.sh  2012-24-07 19:00:00 03:00:00 </code>}}
{{Term|location=frontend|cmd=<code class="command"> ./getmaxgridnodes.sh  2012-09-04 19:00:00 03:00:00 </code>}}




Execute the oar request returned by getmaxgridnodes.sh (don't forget to redirect it as shown in the example)
Execute the oar request returned by getmaxgridnodes.sh (don't forget to redirect it as shown in the example)
The master site is sophia in the following example.
The master site is sophia in the following example.
Warning : Take care of using clusters/site that offers hardware virtualization support (https://api.grid5000.fr/sid/ui/quick-start.html)
{{Term|location=frontend|cmd=<code class="command"> oargridsub -t deploy -w 3:00:00 -s "2012-24-07 19:00:00" sophia:rdef="{\\\\\\\"type='kavlan-global'\\\\\\\"}/vlan=1+/nodes=5+/slash_18=1",rennes:rdef=/nodes=5,nancy:rdef=/nodes=5,lille:rdef=/nodes=5 > ~/oargrid.out</code>}}
{{Term|location=frontend|cmd=<code class="command"> oargridsub -t deploy -w 3:00:00 -s "2012-24-07 19:00:00" sophia:rdef="{\\\\\\\"type='kavlan-global'\\\\\\\"}/vlan=1+/nodes=5+/slash_18=1",rennes:rdef=/nodes=5,nancy:rdef=/nodes=5,lille:rdef=/nodes=5 > ~/oargrid.out</code>}}


== Deploying and configuring the physical machines ==
== Deploying and configuring the physical machines ==
Line 58: Line 62:




You are now connected to the grid OAR job
You are now connected to the OAR job




Line 66: Line 70:


Move to the Flauncher directory
Move to the Flauncher directory
{{Term|location=frontend|cmd=<code class="command"> cd Flauncher</code>}}
{{Term|location=frontend|cmd=<code class="command"> cd DESTINATION_FOLDER/Flauncher</code>}}




Line 223: Line 227:
== Communication ==
== Communication ==


* About the Saturation of ARP tables
* About the Saturation of ARP tables, see [[ARP_table_overflow_tuning]]
[[ARP_table_overflow_tuning]]
 
[[TCP_bandwidth_tuning]]
 


== Tools ==
== Tools ==
Line 233: Line 233:
* Network isolation over Grid'5000 : [[KaVLAN|KaVLAN]]
* Network isolation over Grid'5000 : [[KaVLAN|KaVLAN]]
* Booking a range of IP adresses : [[Subnet reservation|Subnet Reservation System]]
* Booking a range of IP adresses : [[Subnet reservation|Subnet Reservation System]]
* Deployment of nodes : [[Kadeploy|Kadeploy3]]
* Deployment of nodes : Kadeploy
* Execution Remote commands : [[TakTuk|TakTuk]]
* Execution Remote commands : [[Using TakTuk|TakTuk]]




Line 240: Line 240:


* Adrien Lèbre,  Associate Professor at Mines Nantes, Ecole des mines de Nantes, Ascola Team | [mailto:adrien.lebre@inria.fr mail]
* Adrien Lèbre,  Associate Professor at Mines Nantes, Ecole des mines de Nantes, Ascola Team | [mailto:adrien.lebre@inria.fr mail]
* Daniel Balouek, Ingénieur Héméra, ENS Lyon, Inria, Avalon Team | [mailto:daniel.balouek@inria.fr mail]
* Daniel Balouek, Hemera Engineer, ENS Lyon, Inria, Avalon Team | [mailto:daniel.balouek@inria.fr mail]
* Flavien Quesnel, PhD student, Ecole des mines de Nantes, Ascola Team | [mailto:flavien.quesnel@inria.fr mail]
* Flavien Quesnel, PhD student, Ecole des mines de Nantes, Ascola Team | [mailto:flavien.quesnel@inria.fr mail]

Latest revision as of 11:56, 17 April 2019

Overview

The goal of this tutorial is to introduce the deployment of a large amount of virtual machines over the Grid'5000 platform.

After a short precision about Grid'5000 specifications and the requirements of this session, users will be presented a set of scripts and tools designed over the Grid'5000 software stack to deploy and interact with a significant number of virtual machines. Our largest experiment and a few related tips are also described.

Once deployed, these instances can then be used at the user convenience in order to investigate particular concerns such as the impact of migrating a large amount of VMs simultaneously or the study of new proposals dealing with VM images management.

Grid'5000 specifications

When booting KVM instances over the platform, we need physical hosts that supports hardware virtualisation.

During the deployment, we are in charge of configuring the IP network used by virtual machines throught network isolation capabilities of Grid'5000 and the subnet reservation system. It enables the use of an IP network ranging from /22 to /16 subnets, and ensures the communications between all the instances.


Deployment

Booking the resources

The first step is to retrieve the last version of the scripts (the dynVM__TEMPLATE__ is located on sophia site)

Terminal.png frontend:
scp -r /home/alebre/dynVM__TEMPLATE__/ DESTINATION_FOLDER


Move to the folder containing the code

Terminal.png frontend:
cd DESTINATION_FOLDER


Book the g5k resources according to the desired time and duration of your experiment.

We will now consider as master site the site from which you book the ressources and executes the scripts.


This script will return you an OAR that includes the reservation of the nodes, a virtual network and a subnet

Terminal.png frontend:
./getmaxgridnodes.sh 2012-09-04 19:00:00 03:00:00


Execute the oar request returned by getmaxgridnodes.sh (don't forget to redirect it as shown in the example) The master site is sophia in the following example.

Warning : Take care of using clusters/site that offers hardware virtualization support (https://api.grid5000.fr/sid/ui/quick-start.html)

Terminal.png frontend:
oargridsub -t deploy -w 3:00:00 -s "2012-24-07 19:00:00" sophia:rdef="{\\\\\\\"type='kavlan-global'\\\\\\\"}/vlan=1+/nodes=5+/slash_18=1",rennes:rdef=/nodes=5,nancy:rdef=/nodes=5,lille:rdef=/nodes=5 > ~/oargrid.out

Deploying and configuring the physical machines

Move to the Flauncher directory

Terminal.png frontend:
cd Flauncher


Get the list of nodes and then connect the user to job

Terminal.png frontend:
./grid_get_nodes.sh


You are now connected to the OAR job


Display the list of nodes (optional)

Terminal.png frontend:
less ~/gridnodes-uniq


Move to the Flauncher directory

Terminal.png frontend:
cd DESTINATION_FOLDER/Flauncher


Deploy the nodes

Terminal.png frontend:
./grid_deploy_nodes.sh


Deploy the vlan and set the hypervisor (Warning : Use lowercase letter and repeat the master site as the first site)

Terminal.png frontend:
./vlan_auto_launch.sh sophia "sophia nancy lille rennes"


We use a service node during the process to perform the remote operation on the nodes of the infrastructure.

Retrieve the service node

service_node=$(sed -n '/sophia/p' ./log/machines-list.txt | head -n1)

Connect to the service node

Terminal.png frontend:
ssh root@$service_node


Figure 1: FLauncher Process

Usage

Retrieving the infrastructure informations

The service node contains all the informations about the deployed infrastructure.

Those informations can be accessed through various getters functions according to the desired informations.

Example : Retrieving the mapping between the reserved IP and the nodes

Terminal.png service node:
./get_names_and_ip_from_hosts.sh


Please consult the README file for complete description of the available getters.

Terminal.png service node:
less README


Creation of the Virtual Machines

Create a single virtual machine on a host

It is possible to create a single virtual machine by using the following script

The creation needs an IP adress(IP) , an amount of RAM memory(MEMORY), the current site and node and a arbitrary ID.

Terminal.png on physical host:
./create_vm_local.sh -i IP -m MEMORY -n SITE -s NODE -v ID


Create all the virtual machines from a service node

When the infrastructure require a large amount of virtual machines with a specific mapping, it is necessary to describe the desired infrasctructure to enable an automatic and remote deployment.

#list of nodes
suno-3.sophia.grid5000.fr 4 20 8
suno-6.sophia.grid5000.fr 4 20 8
suno-9.sophia.grid5000.fr 4 20 8
#list of VMs
vm00 1 1 1 2147483647
vm01 1 1 1 2147483647
vm02 1 1 1 2147483647
vm03 1 1 1 2147483647
vm04 1 1 1 2147483647
vm05 1 1 1 2147483647
#initial configuration
suno-3.sophia.grid5000.fr vm00 vm03
suno-6.sophia.grid5000.fr vm01 vm04
suno-9.sophia.grid5000.fr vm02 vm05
#end of configuration

In this particular configuration file, we declare 3 physical nodes and 6 virtual machines.

The last part of the configuration describes a simple mapping of 2 virtual machines on each node.

(The nodes must designate the physical hostnames of your infrastructure)


We can now create virtual machines on each host according to the specified configuration file, from the service node.

Terminal.png service node:
./remote_create_vms.sh CONF_FILE


All the virtual instances are started at the same time, using a hierarchical structure among the physical nodes.The correlation between names and IPs is stored in a dedicated file propagated on each physical node. This allows us to identify and communicate with all the virtual machines.

At the end of the operation, the instances previously described are booted and available for experiments.


Interaction with the Machines

We need to be able to control,monitor and communicate with both the host OSes and the guest instances spread across the infrastructure at any time.

For that purpose, the following scripts are based on hierarchical communication structures to ensure a large-scale communication with the physical and/or virtual instances.

Execute remote commands

Execute a command on a list of nodes using a tree distribution

Terminal.png service node:
./mykash.sh nodefile command timeout


Copy files

Copy a file on a list of node using a tree distribution


Terminal.png service node:
./mykaput.sh nodefile file_to_upload remote_path tree_arity


In Practice : 10240 Virtual Machines on 512 Physical Hosts

Considering that physical machines must support hardware virtualization to start KVM instances, the largest experiment that has been conducted up to now involved 10240 KVM instances upon 512 nodes through 4 sites and 10 clusters. The whole setup is performed in less than 30 minutes with about 10 minutes spent on the deployment of the nodes, 5 minutes for the installation and configuration of the required packages on the physical hosts, while the rest is dedicated to the booting of the virtual machines. The result of that work opens doors to the manipulation of virtual machines thoughout a distributed infrastructure like traditionnal operating systems handle process on a local node.


Useful Tips

Booking grid resources

  • Provide the request that retrieves the maximum number of nodes that is available during the time slot defined on specific clusters
Terminal.png frontend:
disco -s '2012-07-24 19:00:00' -w 1 griffon graphene

Deployments

  • Ensure a minimum amount of deployed nodes

To ensure that 95% (rounded down) of the reserved nodes are correctly deployed (3 attempts max), instead of running:

Terminal.png frontend:
kadeploy3 -e squeeze-x64-prod -f $OAR_NODEFILE -k


Run:

NB_NODES=$(sort -u $OAR_NODE_FILE | wc -l)
MIN_NODES=$(($NB_NODES * 95/100))
/grid5000/code/bin/katapult3 --deploy-env squeeze-x64-prod --copy-ssh-key --min-deployed-nodes $MIN_NODES --max-deploy-runs 3


Communication

Tools


Contact

  • Adrien Lèbre, Associate Professor at Mines Nantes, Ecole des mines de Nantes, Ascola Team | mail
  • Daniel Balouek, Hemera Engineer, ENS Lyon, Inria, Avalon Team | mail
  • Flavien Quesnel, PhD student, Ecole des mines de Nantes, Ascola Team | mail