Advanced KaVLAN

From Grid5000
Jump to: navigation, search
See also: KaVLAN | Network reconfiguration tutorial | Advanced KaVLAN | KaVLAN VPN
Note.png Note

This tutorial is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.


The goal of Kavlan is to provide network isolation for Grid'5000 users. KaVLAN allow users to manage VLAN on their Grid'5000 nodes. The benefits is complete level 2 isolation. It can be used together with OAR and Kadeploy to do some experimentations on the grid.

The first step is to read the KaVLAN introduction to understand what kind of VLANs you can configure.

If you want a more concrete example of what you can do with VLANs on Grid'5000, you can go through the Network_reconfiguration_tutorial.

Reserve VLANs and deploy nodes inside

In the first part of the tutorial, we will use kadeploy and kavlan together on a single site, with a routed vlan (we could also use a local vlan).

Once connected on a frontend, in order to obtain nodes and a VLAN you must reserve a kavlan resource with oarsub. There are 3 kinds of resources: kavlan, kavlan-local, kavlan-global. Here, we will use 3 nodes and a routed VLAN, let's say in Sophia on cluster Suno:

Terminal.png frontend:
oarsub -t deploy -l {"type='kavlan'"}/vlan=1,{"cluster='suno'"}/nodes=3 -I

A shell is now opened on the frontend (like any regular deploy job) You can get the id of your VLAN using the kavlan command:

Terminal.png frontend:
kavlan -V

If you run this command outside the shell started by OAR for your reservation, you must add the oar JOBID.

Terminal.png frontend:
kavlan -V -j JOBID

You should get an integer in the <4-9> range for this routed VLAN (the range for local vlan is <1-3>, and there is one global VLAN per OAR server, i.e. one per site).

For our example, let's say we got suno-2, suno-30 and suno-31, and kavlan #4.

You can get all the options of the command using --help:

# kavlan --help
Usage: kavlan [options]
Specific options:
    -i, --vlan-id N                  set VLAN ID (integer or DEFAULT)
    -C, --ca-cert CA                 CA certificate
    -c, --client-cert CERT           client certificate
    -k, --client-key KEY             client key
    -l, --get-nodelist               Show nodenames in the given vlan
    -e, --enable-dhcp                Start DHCP server
    -d, --disable-dhcp               Stop DHCP server
    -V, --show-vlan-id               Show vlan id of job (needs -j JOBID)
    -g, --get-vlan                   Show vlan of nodes
    -s, --set-vlan                   Set vlan of nodes
    -j, --oar-jobid JOBID            OAR job id
    -m, --machine NODE               set nodename (several -m are OK)
    -f, --filename NODEFILE          read nodes from a file
    -u, --user USERNAME              username
    -v, --[no-]verbose               Run verbosely
    -q, --[no-]quiet                 Run quietly
        --[no-]debug                 Run with debug output
    -h, --help                       Show this message
        --version                    Show version

Once you have a kavlan reservation running, you can put your nodes in your VLAN (and back into the default VLAN) at anytime during the lifetime of your job; we will not use this for now.

Instead we will change the VLAN with kadeploy directly. The next step is to deploy the nodes with an environment image, for instance jessie-x64-big.

Enable the dhcp server of the VLAN

Before deploying, if you don't install your own DHCP server, you should start the default DHCP server of the VLAN. Do this with the kavlan command (add -j JOBID if needed) :

Terminal.png frontend:
kavlan -e

(You can disable the DHCP server with kavlan -d)

Deploy nodes and change VLAN in one step

Terminal.png frontend:
kadeploy3 -f $OAR_NODEFILE -k -e jessie-x64-big --vlan `kavlan -V`

Once the deployment is done, you will be able to connect on your nodes. They are now inside the VLAN, therefore they are not reachable with their default IP:

Terminal.png frontend:
ping suno-30 -c1
PING ( 56(84) bytes of data.
From ( icmp_seq=1 Destination Host Unreachable

--- ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

You can get the list of new hostnames of your nodes in the VLAN with kavlan -l. For the next part of this tutorial, let's create a nodefile and copy it on the first node:

Terminal.png frontend:
kavlan -l | tee nodefile
Terminal.png frontend:
scp nodefile root@`head -1 < nodefile`:/tmp

The password for user root on Grid'5000 environments is "grid5000".

You can see that you can ping these new hostnames:

Terminal.png frontend:
ping suno-30-kavlan-4 -c1
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=63 time=0.151 ms

--- ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.151/0.151/0.151/0.000 ms

Setup a DHCP server on your nodes

Configure DHCP

If you need to run your own DHCP server (for example if you want to run a cluster distribution inside kavlan or test kadeploy ), you can use a script to generate the configuration file:

Then, go back the the frontend, and download the script that will generate your dhcp configuration:

Create this file (gen_dhcpd_conf.rb) on the frontend :


# Author: Nicolas Niclausse
# Copyright 2010-2011: INRIA

# script specific to grid5000:
# generate dhcpd config files for kavlan

require 'rubygems'
require 'restfully' # gem install restfully --source
require 'ip' # gem install ruby-ip
require 'getoptlong'
require 'optparse'
require 'ostruct'

headers = "ddns-update-style none;
option space pxelinux;
option pxelinux.magic      code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;
option vendorinfo          code 43  = string;

conf = File.expand_path('~/.restfully/')
options = if FileTest.exists?(conf) then YAML.load_file(conf) else {} end
options[:base_uri] = ''

def parseopts(args)
  options =
  options.debug = false
  options.verbose = false
  options.quiet = false
  options.nodes = []
  opts = do |opts|
    opts.banner = "Usage: gen_dhcpd_conf.rb [options]"
    opts.separator ""
    opts.separator "Specific options:"
    opts.on("-s","--site SITE",  "generate only DHCP conf for site SITE") do |site| = site
    opts.on("-i","--vlan-id N", Integer , "generate only DHCP conf for vlan N") do |vlan|
      options.vlan = vlan
    opts.on("-q", "--[no-]quiet", "Run quietly") do |q|
      options.quiet = q
    opts.on("-v", "--[no-]verbose", "Run verbosely") do |v|
      options.verbose = v
    opts.on_tail("-h", "--help", "Show this message") do
      puts opts

$opts = parseopts(ARGV) do |root, session|
  options = {:query => {:version => root['version']}}
  root.sites(options).each do |site|
    next if not $ and mysite != $
    # optionaly, read mac address from external yaml file
    ref = if FileTest.exists?(mysite+".yaml") then
            puts mysite +": no yaml file for macs" unless $opts.quiet
    if $opts.vlan.nil? then
      vlans = (1..9).to_a
      # try to guess global vlan assigned to current site
      (10..21).each do |gvlan|
          puts "global vlan found for site %s: " % mysite unless $opts.quiet
      vlans = [$opts.vlan]
    vlans.each do |vlan|
      filename = "dhcpd-kavlan-"+vlan.to_s+"-"+mysite+".conf"
      open(filename, 'w') do |f|
        puts "generating "+filename unless $opts.quiet
        f.puts headers
          gateway = IPSocket::getaddress("gw-kavlan-"+vlan.to_s+"."+mysite+"")
          puts "WARN: Get address error: probably no kavlan DNS setup for site " + mysite + " , skip" if $opts.verbose;
        # /20 for local vlans (1..3) and /18 for routed vlan (4..9)
        if vlan < 4
          ip ="/20")
          ns = gateway
          ntp = gateway
          tftp = gateway
          ip ="/18")
          ntp = IPSocket::getaddress("ntp."+mysite+"")
          ns = IPSocket::getaddress("dns."+mysite+"")
          tftp = IPSocket::getaddress("kadeploy-server."+mysite+"")
        netmask = ip.netmask.to_addr
        broadcast = ip.broadcast.to_addr
        network =
        f.puts "subnet %s netmask %s {" %  [network , netmask]
        f.puts "    default-lease-time 86400;
    max-lease-time 604800;"
        #f.puts "    option domain-name \"\"; " % mysite
        f.puts "    option domain-name-servers %s;" % ns
        f.puts "    option ntp-servers %s; " % ntp
        f.puts "    option routers %s;" % gateway
        f.puts "    option subnet-mask %s; " % netmask
        f.puts "    option broadcast-address %s;" % broadcast
        f.puts "    filename  \"pxelinux.0\";"
        f.puts "    next-server %s;" % tftp

        sites_for_vlan = if vlan < 10
                           [ site ]
        sites_for_vlan.each do |currentsite|
          currentsite.clusters(options).each do |cluster|
            cluster.nodes(options).each do |node|
              device = node['network_adapters'].find{|s| s['network_address'] =~ /^\w+-\d+\.\w+\.grid5000\.fr/}
              next if device.nil?
              hostname = device['network_address']
              next if hostname.nil?
              hostname_vlan = hostname.gsub(/^(\w+-\d+)(\..*)$/){$1+"-kavlan-"+vlan.to_s+$2}
              shortname_vlan = hostname_vlan.gsub(/^(\w+-\d+-\w+-\d+)(\..*)$/){$1}
              shortname = hostname.gsub(/^(\w+-\d+)(\..*)$/){$1}
              realsite = hostname.split(".")[1]
                vlan_ip = IPSocket::getaddress(hostname_vlan)
                puts "WARN: Get address error: probably no DNS setup for vlan " +vlan.to_s+" on  site " + sitename + " , skip" if $opts.verbose;
              if device['mac'].nil? then
                if ref[shortname].nil? then
                  puts "WARN: mac undefined for host %s, skip" % hostname unless $opts.quiet
                  mac = ref[shortname]['mac_eth0']
                mac = device['mac']
              f.puts "   host %s {" % hostname_vlan
              f.puts "     hardware ethernet %s; " % mac
              f.puts "     option host-name \"%s\";" %  shortname_vlan
              f.puts "     option domain-name \"%s\\";" % realsite
              f.puts "     fixed-address %s;" % vlan_ip
              if vlan > 9
                # for global vlan, we need the local tftp server
                currenttftp = IPSocket::getaddress("kadeploy-server."+sitename+"")
                f.puts "     next-server %s;" % currenttftp
              f.puts "   }"
        f.puts "}"

(this script use restfully and ruby-ip gems)

Then, generate the configuration (replace VLANID and SITE by your current site and VLAN id), and copy it on the node:

Terminal.png frontend:
chmod +x ./gen_dhcpd_conf.rb
Terminal.png frontend:
gem install ruby-ip --no-ri --no-rdoc --user-install
Terminal.png frontend:
./gen_dhcpd_conf.rb --site SITE --vlan-id VLANID
Terminal.png frontend:
scp dhcpd-kavlan-VLANID-SITE.conf root@`head -1 < nodefile`:

For user accounts, you need to specify your GEM_HOME directory because in the classical one, you won't be able to install the "ruby-ip". To make it possible, type :

Terminal.png frontend:
export GEM_HOME=~/.gem/ruby/2.1.0/

You have to disable the default DHCP server of the VLAN:

On the frontend
Terminal.png frontend:
kavlan -d

Now you have to install a DHCP server on the node (we assume the node is not yet in the job VLAN, or the vlan is routed and have access to internet for apt):

Terminal.png node:
apt-get install dhcp3-server
Terminal.png node:
cp /root/dhcpd.conf /etc/dhcp/

On the node chosen as a DHCP server, start the server:

Terminal.png node:
/etc/init.d/dhcp3-server start

Then, in another shell, connect as root on a second node (or use kaconsole):

Terminal.png frontend:
ssh root@node-xx-kavlan-yy

And restart the network configuration:

Terminal.png node-dhcp-client:
/etc/init.d/networking restart
DHCPDISCOVER on eth0 to port 67 interval 6
[ 5185.656817] bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
[ 5185.670596] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
DHCPDISCOVER on eth0 to port 67 interval 8
DHCPREQUEST on eth0 to port 67
Stopping NTP server: ntpd.
Starting NTP server: ntpd.
bound to -- renewal in 37174 seconds.

on the dhcp server, check the logs:

Terminal.png node-dhcp-server:
tail /var/log/daemon.log
Apr 12 17:32:24 suno-6-kavlan-4 dhcpd: DHCPDISCOVER from 00:26:b9:3f:43:a1 via eth0
Apr 12 17:32:24 suno-6-kavlan-4 dhcpd: DHCPOFFER on to 00:26:b9:3f:43:a1 via eth0
Apr 12 17:32:24 suno-6-kavlan-4 dhcpd: DHCPREQUEST for ( from 00:26:b9:3f:43:a1 via eth0
Apr 12 17:32:24 suno-6-kavlan-4 dhcpd: DHCPACK on to 00:26:b9:3f:43:a1 via eth0

In the four last lines, you see that your own dhcp server has given an address to the other node.


For your information, if you need to do a PXE boot, you must change the tftp server in the generated dhcpd configuration file:

Terminal.png node:
IP=`hostname -i`
Terminal.png node:
perl -i -pe "s/next-server .*/next-server $IP;/" /etc/dhcp/dhcpd.conf

(if there is no next-server configured, you must edit the file by hand and add a line like this:

next-server XX.XX.XX.XX ;

where XX.XX.XX.XX is the IP of your node (echo $IP).

Change the VLAN of your nodes manually

Put your nodes into the reserved VLAN

If you really want to change the VLAN manually, you can, but it's much simpler to change the vlan with kadeploy.

In order to change the VLAN of the nodes manually, you must reconfigure the network after the vlan has changed; but once the VLAN has changed, you can't connect to the node! An easy way to do this is to use the 'at' command (apt-get install at if it's not installed in your nodes)

We will use Taktuk to start remote commands on several nodes at once. In this example, we will use all the nodes. Since taktuk does not handle duplicate names in the nodefile, we must first remove duplicates.

First, we will use taktuk to install at on all nodes, then the taktuk command will simply launch the network reconfiguration in one minute. Finally, we set the VLAN of all our nodes.

As we will change the network configuration of nodes, we will use an isolated kavlan (a.k.a. kavlan-local) to not interfer with the rest of Grid'5000 network.

Terminal.png frontend:
oarsub -t deploy -l {"type='kavlan-local'"}/vlan=1,walltime=2 -I
Terminal.png frontend:
kavlan -V > myvlan
Terminal.png frontend:
oarsub -t deploy -l nodes=2 -I
Terminal.png frontend:
kadeploy3 -e wheezy-x64-base -k -f $OAR_FILE_NODES
Terminal.png frontend:
taktuk -s -l root -f $OAR_FILE_NODES broadcast exec [ "apt-get update; apt-get --yes install at" ]
Terminal.png frontend:
taktuk -s -l root -f $OAR_FILE_NODES broadcast exec [ "echo '/etc/init.d/networking restart'| at now + 1 minute " ]
Terminal.png frontend:
kavlan -i `cat myvlan` -s -f $OAR_FILE_NODES

All nodes are configured in the vlan 2. In one minute, your nodes will renegotiate their IP addresses and will be available inside the VLAN, you can connect to each of them using kaconsole or ssh (as we use a kavlan-local, you must connect to the gateway of that kavlan first):

Terminal.png frontend:
ssh kavlan-`cat myvlan`
Terminal.png kavlan-VLANID:
ssh root@suno-30-kavlan-2

You can use the ip neigh command to see the known hosts in your LAN; you should only see IPs in the subnet

Terminal.png node:
ip neigh dev eth0  INCOMPLETE dev eth0 lladdr 02:00:00:00:01:02 REACHABLE

You should be able to ping another of your host inside your VLAN

Terminal.png node:
ping -c 3 suno-42-kavlan-2
64 bytes from icmp_req=1 ttl=64 time=0.141 ms
64 bytes from icmp_req=2 ttl=64 time=0.166 ms
64 bytes from icmp_req=3 ttl=64 time=0.165 ms

--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.141/0.157/0.166/0.015 ms

Put your nodes back into the default VLAN

First, get put the list of your nodes name with vlan in a file:

Terminal.png frontend:
uniq $OAR_NODEFILE > mynodes
Terminal.png frontend:
sed "s/\([^.]*\)\(.*\)/\1-kavlan-`cat myvlan`\2/" mynodes > mynodes-vlan

Don't forget to first start the network restarting command with taktuk:

Terminal.png frontend:
taktuk -s -l root -f ./mynodes-vlan broadcast exec [ "echo '/etc/init.d/networking restart' | at now + 1 minute " ]

Then you can put your nodes back in the default VLAN:

Terminal.png frontend:
kavlan -s -i DEFAULT -f $OAR_NODEFILE

You should be able to ping your nodes:

Terminal.png frontend:
for i in `uniq $OAR_NODEFILE`; do ping -c 1 $i; done

Another way to put back nodes into the default VLAN is to change the vlan and then kareboot the nodes.

Terminal.png frontend:
kavlan -s -i DEFAULT -f $OAR_NODEFILE
Terminal.png frontend:
kareboot3 -f $OAR_NODEFILE -r simple_reboot

Other usage

Using the API

Kavlan is also available through the API. Using the job and deploy API, you can, as with the command line tools, reverve nodes with vlan and deploy nodes into a vlan. If you want to manipulate VLAN directly through the API, you can do several things:

You can get the vlans you have reserved:


You can get all the vlans available on a site:


You can get the VLAN of nodes on a site:


You can print the VLAN of a list of nodes

POST {\"nodes\": [ <list of node names>]}

You can change the VLAN of a list of nodes:

POST {\"nodes\": [ <list of node names>]}

You can start the dhcp server for the vlan

PUT {"action":"start"}

You can stop the dhcp server for the vlan

PUT {"action":"stop"}

Use a global VLAN

With a global VLAN, you can put nodes from several sites in the same VLAN

First reserve a global vlan on one site (here sophia) and 2 nodes on lille,sophia and lyon:

Terminal.png frontend:
oargridsub -t deploy -w 2:00:00 sophia:rdef="{\\\\\\\"type='kavlan-global'\\\\\\\"}/vlan=1+/nodes=2",lille:rdef=/nodes=2,lyon:rdef=/nodes=2 > oargrid.out

Get the oargrid Id and Job key from the output of oargridsub:

Terminal.png frontend:
export OAR_JOB_KEY_FILE=`grep "SSH KEY" oargrid.out | cut -f2 -d: | tr -d " "`
Terminal.png frontend:
export OARGRID_JOB_ID=`grep "Grid reservation id" oargrid.out | cut -f2 -d= | cut -d ' ' -f2`

Get the node list using oargridstat:

Terminal.png frontend:
oargridstat -w -l $OARGRID_JOB_ID | grep grid> ~/gridnodes

Then use kadeploy3 to deploy your image on all sites and change the VLAN:

Terminal.png frontend:
kadeploy3 -f gridnodes -a -k --multi-server -o ~/nodes.deployed --vlan 18

If you want to manipulate directly VLAN of a node, you have to run the kavlan command on the site where the node is, e.g. if you have reserved the global vlan located at sophia and want to put some nodes of lille into this vlan, you have to run kavlan -m nodename -i VLAN_GLOBAL_ID -s on lille site (or use the API with lille site in the URL).

How to use a local VLAN

In this section, we will describe the specificity of the local VLANs.

If you want to use local VLAN, you have to first connect on the gateway of the vlan. For this, once you have a running reservation on a local VLAN, you have a ssh accces to the gateway:

Terminal.png frontend:
ssh kavlan-<vlanid>

Then you can reach your nodes inside the VLAN. Another option is to use the kaconsole command.

(You can still use kadeploy to put your nodes in the VLAN in one step.)

Configure ssh to easily connect to nodes in a local VLAN

You can configure ssh to make the connection through the gateway transparent:

In order to transparently use ssh to acces to isolated nodes (local VLAN), you should add this to your .ssh/config file on the frontend:

Host *-*-kavlan-1 *-*-kavlan-1.*
    ProxyCommand ssh -a -x kavlan-1 nc -q 0 %h %p
Host *-*-kavlan-2 *-*-kavlan-2.*
    ProxyCommand ssh -a -x kavlan-2 nc -q 0 %h %p
Host *-*-kavlan-3 *-*-kavlan-3.*
    ProxyCommand ssh -a -x kavlan-3 nc -q 0 %h %p

Then you can simply use ssh <cluster>-<nodeid>-kavlan-<vlanid> to access the node , for example:

Terminal.png frontend:
ssh root@NODE-kavlan-VLANID