KaVLAN: Difference between revisions

From Grid5000
Jump to navigation Jump to search
Line 63: Line 63:


KaVLAN only works with ''deploy'' reservations; to obtain nodes and a VLAN, simply add the '''-t kavlan''' option to <code class="command">oarsub</code>. For example, if you need 3 nodes and a VLAN:
KaVLAN only works with ''deploy'' reservations; to obtain nodes and a VLAN, simply add the '''-t kavlan''' option to <code class="command">oarsub</code>. For example, if you need 3 nodes and a VLAN:
oarsub -t kavlan -t deploy -l /nodes=3 -I
{{Term|location=frontend|cmd=<code class="command">oarsub</code> -t kavlan -t deploy -l /nodes=3 -I}}


Then you can get the id of your VLAN using the <code class="command">kavlan</code> command
Then you can get the id of your VLAN using the <code class="command">kavlan</code> command
kavlan -V
{{Term|location=frontend|cmd=<code class="command">kavlan</code> -V}}


If you run this command outside the shell started by OAR for your reservation, you must add the oar JOBID. Due to a bug in oarstat, you may get a few lines of Perl on stderr.
If you run this command outside the shell started by OAR for your reservation, you must add the oar JOBID. Due to a bug in oarstat, you may get a few lines of Perl on stderr.
kavlan -V -j JOBID
{{Term|location=frontend|cmd=<code class="command">kavlan</code> -V -j <code class="replace">JOBID</code>}}


You should get an integer in the <1-8> range.
You should get an integer in the <1-8> range.


You can get all the options of the command using --help:
You can get all the options of the command using --help:
 
<pre class="brush: bash">
  # kavlan --help
  # kavlan --help
  Version 1.0rc2
  Version 1.0rc2
Line 92: Line 92:
       -h|--help                    print this help
       -h|--help                    print this help
       -v|--verbose                  verbose mode
       -v|--verbose                  verbose mode
</pre>


Once you have a kavlan reservation running, you are allowed to connect to the VLAN gateway named <code class='hostname'>kavlan-<ID></code> where ID is your vlan ID, and you can also put your nodes in your VLAN (and back into the default VLAN) at anytime during the lifetime of your job.
Once you have a kavlan reservation running, you are allowed to connect to the VLAN gateway named <code class='hostname'>kavlan-<ID></code> where ID is your vlan ID, and you can also put your nodes in your VLAN (and back into the default VLAN) at anytime during the lifetime of your job.
Line 98: Line 99:


Let's say you want to deploy all nodes:
Let's say you want to deploy all nodes:
kadeploy -f $OAR_NODEFILE -e <myenvironment>
{{Term|location=frontend|cmd=<code class="command">kadeploy</code> -f $OAR_NODEFILE -e <code class="replace">myenvironment</code>}}


== Enable/disable the dhcp server of the gateway ==
== Enable/disable the dhcp server of the gateway ==

Revision as of 16:51, 16 March 2010


Overview

The goal of Kavlan is to allow people to manage VLAN on Grid'5000 nodes. The benefits is complete level 2 isolation. It can be used together with OAR and Kadeploy to do some experimentations on the grid.

The following figure shows two jobs running with KaVLAN: each job has it's nodes isolated in a VLAN (purple and green). The other nodes are all in the default VLAN (red). The only way to reach the isolated nodes is to use a gateway node (kavlan-1 and kavlan-2 in the figure). The gateway has two Ethernet interfaces: one in the default VLAN and one is the dedicated VLAN. This way, you can use ssh to reach your nodes (an other way to reach an isolated node is to use the kaconsole command).

KaVLAN architecture: 2 jobs running KaVLAN
Note.png Note

The gateways are NOT doing any routing: they are only used as ssh gateways.

Currently, KaVLAN can be used on one site only. The Technical team is investigating the possibility of using QinQ in Grid'5000 to allow Grid-wide VLANs.

Installation status on sites :

Sites Version Status
Bordeaux Fail.png
Grenoble Fail.png
Lille 1.0rc3 Check.png
Lyon Fail.png
Nancy InProgress.png
Orsay Fail.png
Rennes Fail.png
Sophia 1.0rc5 Check.png
Toulouse Fail.png

Usage

How to reserve a VLAN

KaVLAN only works with deploy reservations; to obtain nodes and a VLAN, simply add the -t kavlan option to oarsub. For example, if you need 3 nodes and a VLAN:

Terminal.png frontend:
oarsub -t kavlan -t deploy -l /nodes=3 -I

Then you can get the id of your VLAN using the kavlan command

Terminal.png frontend:
kavlan -V

If you run this command outside the shell started by OAR for your reservation, you must add the oar JOBID. Due to a bug in oarstat, you may get a few lines of Perl on stderr.

Terminal.png frontend:
kavlan -V -j JOBID

You should get an integer in the <1-8> range.

You can get all the options of the command using --help:

 # kavlan --help
 Version 1.0rc2
 USAGE : kavlan [options]
       -r|--get-network-range
       -g|--get-network-gateway
       -l|--get-nodelist
       -V|--get-vlan-id              print VLAN ID of job (needs -j JOBID)
       -d|--disable-dhcp
       -e|--enable-dhcp
       -i|--vlan_id <VLANID>
       -s                            set vlan for given node(s)
       -f|--filenode <NODEFILE>
       -j|--oar-jobid=<JOBID>
       -m|--machine <nodename>
       -q|--quiet                    quiet mode
       -h|--help                     print this help
       -v|--verbose                  verbose mode

Once you have a kavlan reservation running, you are allowed to connect to the VLAN gateway named kavlan-<ID> where ID is your vlan ID, and you can also put your nodes in your VLAN (and back into the default VLAN) at anytime during the lifetime of your job.

Since KaVLAN works only with deploy jobs, the next step is to deploy at least one node (otherwise, you won't have root acces on it and therefore can't restart it's network configuration).

Let's say you want to deploy all nodes:

Terminal.png frontend:
kadeploy -f $OAR_NODEFILE -e myenvironment

Enable/disable the dhcp server of the gateway

Once the deployment is over, you are now able to change the VLAN of your nodes. First check that the DHCP server is running on the gateway, run on the frontend (add -j JOBID if needed) :

kavlan -e 

You can disable the DHCP server with kavlan -d

Change the VLAN of your nodes

In order to change the VLAN of the nodes, you must reconfigure the network after the vlan has changed; but once the VLAN has changed, you can't connect to the node! An easy way to do this is to use the 'at' command (apt-get install at if it's not installed in your nodes)

We will use Taktuk to start remote commands on several nodes at once. In this example, we will use all the nodes. Since taktuk does not handle duplicate names in the nodefile, we must first remove duplicates.

The taktuk command simply launch the network reconfiguration in one minute. Then we set the VLAN of all our nodes.

# uniq $OAR_NODEFILE > ./mynodes
# taktuk -l root -f ./mynodes broadcast exec [ "echo '/etc/init.d/networking restart'| at now + 1 minute " ]
# kavlan -s
Take node list from OAR nodefile: /var/lib/oar/387465
 ... node azur-25.sophia.grid5000.fr changed to vlan KAVLAN-7
 ... node azur-28.sophia.grid5000.fr changed to vlan KAVLAN-7
 ... node azur-30.sophia.grid5000.fr changed to vlan KAVLAN-7
all nodes are configured in the vlan 7

In one minute, your nodes will renegotiate their IP addresses and will be available inside the VLAN. To get the name of your nodes in the VLAN, use the -l option:

kavlan  -l
azur-25-kavlan-7.sophia.grid5000.fr
azur-28-kavlan-7.sophia.grid5000.fr
azur-30-kavlan-7.sophia.grid5000.fr

You can connect to each of them using kaconsole or ssh:

ssh kavlan-<vlanid>
kavlan-7@sophia> ssh root@azur-25-kavlan-7
azur-25-kavlan-7:~#

You can configure ssh to make this transparent:

Configure ssh to easily connect to nodes in a VLAN

In order to transparently use ssh to acces to isolated nodes, you should add this to your .ssh/config file on the frontend:

Host *-*-kavlan-1
   ProxyCommand ssh -q -a -x kavlan-1 nc %h %p
Host *-*-kavlan-2
   ProxyCommand ssh -q -a -x kavlan-2 nc %h %p
Host *-*-kavlan-3
   ProxyCommand ssh -q -a -x kavlan-3 nc %h %p
Host *-*-kavlan-4
   ProxyCommand ssh -q -a -x kavlan-4 nc %h %p
Host *-*-kavlan-5
   ProxyCommand ssh -q -a -x kavlan-5 nc %h %p
Host *-*-kavlan-6
   ProxyCommand ssh -q -a -x kavlan-6 nc %h %p
Host *-*-kavlan-7
   ProxyCommand ssh -q -a -x kavlan-7 nc %h %p
Host *-*-kavlan-8
   ProxyCommand ssh -q -a -x kavlan-8 nc %h %p

Then you can simply use ssh <cluster>-<nodeid>-kavlan-<vlanid> to access the node , for ex:

ssh root@azur-25-kavlan-7
azur-25-kavlan-7:~#

Put your nodes back into the default VLAN

kavlan -s -i DEFAULT -f $OAR_NODEFILE

Advance usage

Setup a DHCP server on your nodes

If you need to run your own DHCP server (for example if you want to run a cluster distribution inside kavlan or test kadeploy ), you can use the configuration file available on the VLAN's gateway.

Let's say that you want to install dhcpd on azur-25-kavlan-7. You first have to install a dhcp server (azur-25-kavlan-7$ apt-get install dhcp3-server on debian) on this node.

Then, copy the configuration file from the gateway to the node:

scp kavlan-7:/etc/dhcp3/dhcpd.conf root@azur-25-kavlan-7:/etc/dhcp3/

You need to change the tftp server in order to do PXE boot:

azur-25-kavlan-7# IP=`hostname -i`
azur-25-kavlan-7# perl -i -pe "s/next-server .*/next-server $IP;/" /etc/dhcp3/dhcpd.conf

And then you can start the server, once you have disabled the gateway's DHCP server (frontend$ kavlan -d).

azur-25-kavlan-7# /etc/init.d/dhcp3-server start