Virtualization in Grid'5000

Purpose

This page presents how to use KVM on the production environment (with a "non-deploy" reservation). The aim is to permit the execution of virtual machines on the nodes, along with a subnet reservation, which will give you a range of routed IP for your experiment.

In the first part, you will learn the basics of g5k-subnets, which is a prerequisite for the rest of this tutorial. The Quick start explains how to run a VM on the production environment in the minimal number of steps. The next part is optional, it explains in details the contextualization mechanism, which allows you to customize your virtual machines. Finally, in the Multi-site experiment section, we will deploy 2 VMs on 2 sites, and we will measure the network bandwidth between them with iperf.

Prerequisite: Network subnets reservation with g5k-subnets

Users deploying VMs on Grid'5000 need to attribute IP address to them. Each site of Grid'5000 is allocated a /14 block for this purpose, divided in 4 smaller blocks.

OAR can be used to reserve a range of IPs. OAR permits to share the IP resources among users, and avoid the potential IP conflicts at the same time.

Reservation

Subnet reservation through OAR is similar to normal resource reservation.

To reserve 4 /22 subnets and 2 nodes, just type:

frontend:

oarsub -l slash_22=4+nodes=2 -I

You can of course have more complex request. To obtain 4 /22 on different /19 subnets, you can type:

frontend:

oarsub -l slash_19=4/slash_22=1+nodes=2/core=1 -I

Usage

The simplest way to get the list of your allocated subnets is to use the g5k-subnets script provided on the head node of the submission.

# g5k-subnets
10.8.0.0
10.8.8.0

Several other printing options are available (-p option to display the CIDR format, -b to display broadcast address, -n to see the netmask, and -a is equivalent to -bnp):

# g5k-subnets -a
10.8.0.0/21	10.11.255.255	255.255.252.0	10.11.255.254
10.8.8.0/21	10.11.255.255	255.255.252.0	10.11.255.254

You can also summarize the subnets into a larger one if they are contiguous:

# g5k-subnets -sp
10.8.0.0/20

You can display all the available IP in your reservation, and their associated unique mac addresses, with the following command.

# g5k-subnets -im
10.158.16.1     00:16:3E:9E:10:01
...

	Note
	For detailed information, see the Subnet reservation page. The Grid5000:Network page also describes our organization of the virtual IP space inside Grid'5000.

Quick start

In this part, we will create a virtual machine in a few steps, and ssh to it.

Job submission

In order to test easily the kvm environment, we use an interactive job, and we reserve one subnet and one node with hardware virtualization capabilities.

frontend:

oarsub -I -l "slash_22=1+{virtual!='none'}/nodes=1"

Disk image, virtual machine

A disk image containing debian wheezy is available at the following path: /grid5000/images/KVM/wheezy-x64-base.qcow2

It can be used as a base for more advanced work. For the next steps of this tutorial, copy the disk image to /tmp on the node:

node:

cp /grid5000/images/KVM/wheezy-x64-base.qcow2 /tmp/

Network settings

As seen before, g5k-subnets maintains a correspondence between MAC addresses and IP addresses. The Debian system provided on the disk image is configured to use DHCP, the DHCP server will assign the IP corresponding to the MAC address of the virtual machine.

Consequently, you have to choose an IP in the range you have reserved, and set the MAC address of the VM to the associated MAC address.

You can get the list of available IP, and an associated unique MAC address with the following command.

node:

g5k-subnets -im

10.172.0.1      00:16:3E:AC:00:01
10.172.0.2      00:16:3E:AC:00:02
10.172.0.3      00:16:3E:AC:00:03
10.172.0.4      00:16:3E:AC:00:04
10.172.0.5      00:16:3E:AC:00:05
10.172.0.6      00:16:3E:AC:00:06
10.172.0.7      00:16:3E:AC:00:07
10.172.0.8      00:16:3E:AC:00:08
10.172.0.9      00:16:3E:AC:00:09
10.172.0.10     00:16:3E:AC:00:0A
...

Run the guest OS using libvirt

Libvirt is a toolkit for managing virtualization servers. Libvirt is also an abstraction layer for different virtualization solutions, including KVM but also Xen and VMWare ESX.

In our case, we use libvirt on top of KVM.

• Create a domain file in XML, describing a virtual machine.

eg : domain.xml

 <domain type='kvm'>
  <name>wheezy</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <os>
    <type arch="x86_64">hvm</type>
  </os>
  <clock offset="localtime"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver type='qcow2'/>
      <source file='/tmp/wheezy-x64-base.qcow2'/>
      <target dev='vda' bus='virtio'/>
     <shareable/>
    </disk>
    <interface type='bridge'>
      <mac address='AA:BB:CC:DD:EE:FF'/>
      <source bridge='br0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/ttyS0'/>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <source path='/dev/ttyS0'/>
      <target port='0'/>
    </console>
  </devices>
 </domain>

	Note
	- Adapt this file to your case, you must change the "mac address" field with one of the g5k-subnet addresses - The password for the root account is grid5000

• Now, the guest OS can be started.

node:

virsh create domain.xml

• You can also use virsh to manage your guest OS:
  • list the running virtual machines: virsh list
  • open a console on the "wheezy" virtual machine: virsh console wheezy

	Note
	Use CTRL+AltGr+] on a french keyboard to disconnect from virsh console

• At this point, you can repeat the full process and launch several VMs in parallel.

Run the guest OS using the kvm command

You can also use kvm to start the virtual machine. However, to use the network with kvm, a Tun/Tap interface must be created manually for each virtual machines.

This virtual interface will be attached to your virtual machine, and bridged on the physical machine to the production network. Therefore, the virtual machine will be able to get an IP from the DHCP server and access the network.

A script is available to create automatically this interface on the node: create_tap:

node:

sudo create_tap

• Tun/Tap interfaces are listed by issuing the command /sbin/ifconfig.

node:

/sbin/ifconfig

tap0      Link encap:Ethernet  HWaddr 00:16:3e:db:c6:41
          inet6 addr: fe80::58ff:a4ff:fe97:c6a8/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:29435 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

	Note
	- Create one Tun/Tap interface per guest OS. - Use tunctl if you need to delete a Tun/Tap device. node: /usr/sbin/tunctl -d tap0

Copy the disk image to /tmp on the node:

node:

cp /grid5000/images/KVM/wheezy-x64-base.qcow2 /tmp/kvm_wheezy-x64-base.qcow2

Start your virtual machine using kvm (this is an example command, feel free to adapt it to your use case: The kvm process is launched in a screen session, if you are not familiar with screen, read its documentation):

node:

screen kvm -m 512 -hda /tmp/kvm_wheezy-x64-base.qcow2 -net nic,model=virtio,macaddr=AA:BB:CC:DD:EE:FF -net tap,ifname=tap0,script=no -nographic

	Note
	- tap0 is the name of our Tun/Tap interface. Adapt it with the Tun/Tap name on which you want to attach your guest OS. - Adapt "macaddr" option with one of the g5k-subnet addresses - The password for the root account is grid5000

SSH to your virtual machine

Finally, you can ssh directly to your VM from anywhere in Grid'5000:

node:

ssh root@g5k-subnet_ip_addr

	Note
	The password for the root account is grid5000

Use contextualization to configure your VMs

This part describes the basic usage of a contextualization, an easy way to dynamically configure your VM.

Mechanism

Contextualization can be used to do any kind of configuration upon VMs boot, in order to configure the virtual machines on boot.. It works like the following :

• Test for the presence of a CD in the CD drive of the VM
• if it exists, mount the CD, test the presence of a script post-install, and run it as root

Create a contextualization script

In this example, we will create a simple contextualization script to add your SSH key to enable password-less connection.

• Create a directory to store your contextualization script:

node:

mkdir kvm-context/

• Copy your SSH public key into it:

node:

cp ~/.ssh/id_rsa.pub kvm-context/

• Create a file named "post-install" and add execution rights to it:

node:

touch kvm-context/post-install

node:

chmod 755 kvm-context/post-install

• Edit the "post-install" script and write the command needed to add the SSH key to root's authorized_keys file :

#!/bin/sh
mkdir -p /root/.ssh
cat /mnt/id_rsa.pub >> /root/.ssh/authorized_keys

Note that contextualization files are accessible in the virtual machine inside /mnt directory

• Once you have prepared the content of your contextualization script, you can generate an ISO image :

node:

genisoimage -r -o kvm-context.iso kvm-context/

The file kvm-context.iso is ready to be attached to a VM, the script included in the iso will be executed and will configure the first network interface at boot time.

Start a VM with contextualization

Use this domain file to start your VM using libvirt:

domain.xml

 <domain type='kvm'>
  <name>wheezy</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <os>
    <type arch="x86_64">hvm</type>
  </os>
  <clock offset="localtime"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver type='qcow2'/>
      <source file='/tmp/wheezy-x64-base.qcow2'/>
      <target dev='vda' bus='virtio'/>
     <shareable/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file='/tmp/kvm-context.iso'/>
      <target dev='vdb' bus='virtio'/>
      <readonly/>
    </disk>
    <interface type='bridge'>
      <mac address='AA:BB:CC:DD:EE:FF'/>
      <source bridge='br0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/ttyS0'/>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <source path='/dev/ttyS0'/>
      <target port='0'/>
    </console>
  </devices>
 </domain>

	Note
	- Note the second disk section, you must adapt it with the path of the contextualization iso file (don't forget to also change the mac address - To include contextualization ISO to a VM started with the kvm command, add -cdrom /tmp/kvm-context.iso option

Start the guest OS and connect it using ssh :

node:

virsh create domain.xml

node:

ssh root@g5k-subnet_ip_addr

If the contextualization ran correctly, you should not need to enter a password.

Add contextualization to your own VM

The contextualization mechanism is not standard, if you want to use it on your VM, you must adapt it to your image.

• Here is an example of a contextualization script :

#!/bin/bash

DEVICE=
[ -b /dev/hdb ] && DEVICE=/dev/hdb
[ -b /dev/sdb ] && DEVICE=/dev/sdb
[ -b /dev/vdb ] && DEVICE=/dev/vdb
[ -b /dev/xvdb ] && DEVICE=/dev/xvdb
[ -b /dev/sr0 ] && DEVICE=/dev/sr0

if [ -b "$DEVICE" ];then
    /bin/mount -t iso9660 $DEVICE /mnt 2> /dev/null

    if [ -f /mnt/post-install ]; then
      bash /mnt/post-install
    fi

    umount /mnt 2> /dev/null
fi
exit 0

• This contextualization script must be executed during the boot sequence. For instance,
  • copy it to /usr/local/bin/init
  • add execution rights to /usr/local/bin/init
  • to launch the script at VM startup, add:

# KVM contextualization script
/usr/local/bin/init

inside /etc/rc.local, at the end of the file (before the exit 0 if any).

Multi-site experiment

In this part, to illustrate what can be done using Virtual machines on the production environment, we will start two virtual machines on two sites, and make them communicate using the virtualization network.

Reservation

Open 2 terminals, and ssh to the frontends of 2 sites, in this example, it will be the frontend of Luxembourg, and the frontend of Nancy. Then, reserve two virtualization-capable nodes and two subnets on two different sites.

frontend: oarsub -I -l "slash_22=1+{virtual!='none'}/nodes=1"

frontend: oarsub -I -l "slash_22=1+{virtual!='none'}/nodes=1"

Network configuration

In this part, we will choose an IP for the 2 virtual machines.

Choose a couple of IP & MAC for each VM, in the output of g5k-subnets -im. Note that g5k-subnets returns completely different information on each site. In the following, we assume that you chose 10.144.8.1 (00:16:3e:90:08:01) in Nancy, and 10.172.0.1 (00:16:3e:ac:00:01) in Luxembourg.

node(nancy): g5k-subnets -im | head

node(luxembourg): g5k-subnets -im | head

Instantiate your VMs

Copy a standard virtual machine image

Copy the default virtual machine image from /grid5000/images/KVM/wheezy-x64-base.qcow2 to /tmp

node(nancy): cp /grid5000/images/KVM/wheezy-x64-base.qcow2 /tmp/

node(luxembourg): cp /grid5000/images/KVM/wheezy-x64-base.qcow2 /tmp/

Create the domain.xml file

The domain.xml file contains the description of your virtual machine. You must adapt it, in order to use a mac address provided by g5k-subnets -im. The virtual machine will get the IP associated to its mac address.

<domain type='kvm'> <name>wheezy</name> <memory>362144</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/wheezy-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <interface type='bridge'> <mac address='00:16:3e:90:08:01'/> <source bridge='br0'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain>

<domain type='kvm'> <name>wheezy</name> <memory>362144</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/wheezy-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <interface type='bridge'> <mac address='00:16:3e:ac:00:01'/> <source bridge='br0'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain>

Launch the two VMs

node(nancy): virsh create domain.xml

node(luxembourg): virsh create domain.xml

Enjoy !

SSH in your VMs

node(nancy): ssh root@10.144.8.1

node(luxembourg): ssh root@10.172.0.1

	Note
	The password for the root account is grid5000

Install and run iperf

Finally, we will install iperf and measure the bandwidth between the two VMs:

• install iperf with apt-get ;
• then, run iperf in server mode (-s parameter) on one node, and in client mode (-c parameter) on the other.

vm(nancy): apt-get install iperf vm(nancy): iperf -s root@vm-1:~# iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.144.8.1 port 5001 connected with 10.172.0.1 port 52389 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.09 GBytes 938 Mbits/sec

vm(luxembourg): apt-get install iperf vm(luxembourg): iperf -c 10.144.8.1 root@vm-1:~# iperf -c 10.144.8.1 ------------------------------------------------------------ Client connecting to 10.144.8.1, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.172.0.1 port 52389 connected with 10.144.8.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.09 GBytes 938 Mbits/sec