From Grid5000

Contents 1 IP networks in use 1.1 Public Networks 1.2 Local Networks 2 Network 3 Security local policy 3.1 Network equipments 3.1.1 Ethernet network 3.1.1.1 Cisco Catalyst 6509 3.1.1.2 Cisco Catalyst 3750 3.1.1.3 Nodes NICs 3.1.2 Myrinet high performance network 3.1.2.1 Switch 3.1.2.2 Nodes NICs 4 Logical networks 4.1 Production VLAN 4.2 Myrinet network

IP networks in use

You have to use a public network range to run an experiment between several Grid5000 sites.

Public Networks

• computing : 172.16.80.0/20
• virtual-1 : 10.148.0.0/14
• virtual-2 : 10.152.0.0/14

Local Networks

• admin : 172.17.80.0/20

Network

The equipments are physically connected in the experiment networks as described in this diagram:

The main switch is a Cisco Catalyst 6509 which interconnects the whole site with the rest of the Grid'5000 platform on a dedicated Ethernet 10Gb network.

The site is divided into two clusters:

• gdx: The gdx cluster is composed of :
  • 14 Cisco Catalyst 3750 24 ports Gigabit Ethernet switches, directly connected to the Cisco Catalyst 6509 through aggregated links 3x1Gb. 18 nodes are connected to each of those switches, through Ethernet 1Gb links.
  • 2 HP Procurve 24 ports Megabits Ethernet switches, directly connected to the Cisco Catalyst 6509 through aggregated links 2x1Gb. 18 nodes are connected to each of those switches, through Ethernet 1Gb links.
  • 1 HP procurve 48 ports Gigabit Ethernet switches, directly connected to the Cisco Catalyst 6509 through aggregated links 3x1Gb. 22 nodes are connected to each of those switches, through Ethernet 1Gb links.

Moreover, all nodes are connected to a Myrinet 10Gb switch, on a high speed and low latency interconnect network, using Myrinet and MX protocols.

• netgdx

The netgdx cluster is composed of 30 nodes, directly connected to the Cisco Catalyst 6509 through 3 Ethernet Gigabit links.

Security local policy

Port and addresses filtering policy

	Client side

Server side	Remote GRID'5000	Local GRID'5000	Local front gateway	Local laboratory	Rest of the world
Remote GRID'5000	N/A	G	G	D	D
Local GRID'5000	G	G	G	D	D
Local front gateway	D	D	D	FW	FW
Local laboratory	D	D	D	?	?
Rest of the world	D	D	D	?	?

• N/A: should not be seen at the local routing device;
• G: granted within the rule of thumb;
• D: denied;
• FW: FireWall filtering (incoming ssh port only, known IP only);
•  ?: independant from Grid'5000.

Network equipments

Ethernet network

Cisco Catalyst 6509

The Cisco Catalyst 6509 is a 9 slots chassis which provides switching service between site equipments and routing service with the rest of the Grid'5000 platform.

• Supervision board

The supervision board is Supervisor Engine 720 (WS-SUP720-BASE) with MSFC3 Daughterboard. It runs IOS s72033_rp-IPSERVICESK9-VM, version 12.2(33)SXI1.

• Modules

Slot	Card type	Model
1	empty	none
2	empty	none
3	empty	none
4	CEF720 8 port 10GE with DFC	WS-X6708-10GE
5	Supervisor Engine 720 (Active)	WS-SUP720-BASE
6	CEF720 48 port 10/100/1000mb Ethernet	WS-X6748-GE-TX
7	CEF720 48 port 10/100/1000mb Ethernet	WS-X6748-GE-TX
8	CEF720 48 port 10/100/1000mb Ethernet	WS-X6748-GE-TX
9	CEF720 48 port 10/100/1000mb Ethernet	WS-X6748-GE-TX

• 10Gb Ethernet Link

The 10Gb Ethernet link is connected on port 4/1 of the switch (port 1 of card WS-X6708-10GE in slot 4). It uses a X2 module Cisco X2-10GB-LR, with a monomode optical fiber directly connected to the POP Renater in Orsay.

Cisco Catalyst 3750

The 18 switches Cisco Catalyst 3750 are WS-C3750G-24TS, with 24 Ethernet 10/100/1000 ports. They provide 4 additionnal SFP ports, with Cisco 1000BASE-T SFP.

In general, 18 nodes are connected on each of those switches, on port range from 7 to 24. An aggregated link (EtherChannel, w/o LACP/IEEE 802.3ad) composed of 3 SFP ports is used to connect the switches to the Cisco Catalyst 6509.

Nodes NICs

• gdx nodes provide 2 NICs:
  • onboard Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet, pci bus 04:04.0, named eth0 in production environment
  • onboard Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet, pci bus 04:04.1, named eth1 in production environment (unavailable for use, not connected)

• netgdx nodes provide 3 NICs:
  • onboard Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet, pci bus 04:04.0, named eth0 in production environment
  • onboard Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet, pci bus 04:04.1, named eth1 in production environment
  • additional PCI-Express Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet, pci bus 03:02.0, named eth2 in production environment

Myrinet high performance network

A copper based Myrinet 10Gb high speed and low latency network interconnects all 310 gdx nodes together. It uses the Myrinet protocol, with the low-level message passing system MX (Myrinet Express). For convenience, it provides an Ethernet emulation over MX (EoMX) with allows the use of IP protocol among others, at least in the production environment.

Switch

The switch is a 21U enclosure Myricom 10G-21U-CLOS-ENCL which provides 512 10Gb ports, in 41 slots with the following line cards:

Slot range	Model	uc microcode version	uc rom version	firmware
M	10G-MONITOR-B	10G Switch Firmware v2.2	9S08 Monitor v2.0	v0.9.9.18
0-15	10G-SW32LC-16M	10G Switch Firmware v2.4	9S08 Monitor v2.0	NA
16-23	10G-25W32LC	10G Switch Firmware v2.4	9S08 Monitor v2.0	NA
24-39	10G-SW32LC-16M	10G Switch Firmware v2.4	9S08 Monitor v2.0	NA

Nodes NICs

All gdx nodes provide additional PCI-Express Myricom 10G-PCIE-8A-C cards, at PCI bus 07:00.0. In the production environment, cards run with MX kernel module version 1.2.8. Those cards have a CX4 connector, with copper cables directly connected to the switch.

Logical networks

The networks are distributed as described in this diagram:

At the data link layer, there are 3 networks:

• Backbone VLAN (id 550), which is used for traffic with the rest of Grid'5000 platform,
• Production VLAN (id 100), which contains IP production networks,
• Myrinet network, which contains Ethernet network emulated over Myrinet among classical Myrinet network.

The Cisco Catalyst 6509 routes traffic between the rest of Grid'5000 platform and IP production network.

Production VLAN

The production VLAN contains the IP production network at the network layer. The network is 172.16.80.0/20, and it is divided into the multiple subnets:

• 172.24.[80.1 - 81.56] , which belongs to gdx-[1 - 310]-eth0,
• 172.24.82.[1 - 30], which belongs to netgdx-[1 - 30]-eth0,
• 172.24.82.[101 - 130], which belongs to netgdx-[1 - 30]-eth1,
• 172.24.82.[201 - 230], which belongs to netgdx-[1 - 30]-eth2.

All nodes use the Cisco Catalyst 6509 VLAN 100 IP address (172.16.95.254) as their default gateway.

Myrinet network

As seen in the physical network overview, Myrinet network is physically isolated from Ethernet networks. Therefore, Ethernet network emulated over Myrinet is isolated as well. There isn't any interconnexion, neither at the data link layer nor at the network layer.

A subnet of the IP production network is used to address all gdx nodes emulated ethernet interfaces (myri0 in production environment): 172.18.80.0/20.