Strasbourg Core-Network XP Infra

The following describes the technical team's projects on the Grid'5000 site of Strasbourg to allow core network experimentations.

This is a work in progress.

Core network topology

Testbed physical topology

The Strasbourg site is made up of nodes with numerous network interfaces. The aim is to enable the user to design complex and varied network topologies between server (fleckenstein, future GPU cluster) and switch (engelbourg, ramstein). This is also made possible by the mesh infrastructure being set up to interconnect all the nodes in the site's clusters.

Testbed topology emulation

The goal is to let users design topologies of interconnection between core network experimentation switches and other server nodes.

Thanks to an infrastructure switch (sw-core), users will be able to create on-demand L2 links between experimentation switches (named "engelbourg-[1-8]" and "ramstein-1") and server nodes (named "fleckenstein-[1-10]" pr "gpucluser-[1-4]").

The figure below on the right shows an example of an emulated network topology. On the left, the figure shows the associated real network topology.

Users will be able to fully reinstall/reconfigure all experimental pieces of equipment in the green boxes thanks to Kadeploy.

Users will be able to configure VLANs on the "mesh" network switch (rose box) thanks to KaVLAN, thus creating the L2 links between the experimental equipment.

A tool such as distem may be installed on servers to emulate latency/failures.

Switch links description

All nodes in all clusters have several interfaces connected to the sw-core switch. This architecture enables users to reconfigure the network and create L2 links on demand between all nodes (engelbourg, ramstein, fleckenstein, gpucluster) on the Strasbourg site.

A	Link	B
sw-core	3x100G	engelbourg-[1-8]
	8x100G	ramstein-1
	2x100G	fleckenstein-[1-10]
	1x25G
	?	gpucluster

Node-to-node direct interconnection description

In addition to the configurable links to sw-core, a direct interconnection between the nodes described in the table below is also implemented.

A	Link	B
engelbourg-[1-8]	1x100G	engelbourg-[1-8]
ramstein-1	1x100G	engelbourg-[1-8]
fleckenstein-[1-5]	4x25G	engelbourg-[1-4]
fleckenstein-[6-10]	4x25G	engelbourg-[5-8]

Instrumentation of the NOS

The goal of the project is to allow users to master the Operating System of network switches, just like already possible on experimentation servers.

With the arrival of "white box switches" on the market, the following assumptions can be taken:

• Nowadays, many switches actually run a "GNU/Linux-based" operating system (e.g., OpenSwitch, Sonic, Cumulus), which is a derivative of general-purpose server distributions (e.g., Debian).
• From the hardware viewpoint, they are very server-like, because assembled from standard commodity parts. For example, the engelbourg experimental switches (Edge-core Wedge100BF) have the following specs:
  • x86 CPU
  • Hard-drive (SSD)
  • Management NIC is just a classical CPU-attached NIC
  • They have a BMC (out-of-band power-on/off, serial console, monitoring, ...)
• Of course, they also feature an ASIC (e.g., Tofino) for the offload of network functions:
  • That ASIC requires a dedicated Linux kernel and/or driver.
  • But it can be assimilated to any accelerator, just like GPUs or FPGAs installed in servers. Thus, it can be managed like accelerator cards, with no impact on the bare-metal OS deployment mechanism.

The conclusion is that a cluster of white-box switches can be managed like a cluster of servers from the operating system and hardware perspective (apart from the network cabling, of course, discussed above).

In particular, that means that while ONIE is nowadays the standard mechanism to install NOS, it is replaceable by other server installation mechanisms, which allow more flexibility for experimentation (frequent reinstallation, customization, configuration traceability).

Grid'5000 uses the Kadeploy software to achieve that role on any hardware of the platform. Work is in progress to allow users to "kadeploy" on-the-shelf NOS images in the ONIE format, thanks to one of the two options:

• using an image converter (E.g, a Kameleon recipe) allowing to create kadeploy environments from ONIE images.
• Adding the ONIE NOS image support in the Kadeploy installer miniOS.

But right now, it is already possible to deploy a vanilla Debian with Intel P4studio software suite and do some experiments on the Tofino ASIC.

In conclusion, the engelbourg cluster of experimental switches will be managed just like the other Grid'5000 clusters of servers, benefiting from all the powerful features provided by the Grid'5000 stack (resource reservation, bare-metal parallel deployment tool, recipe-based OS image builder, hardware reference repository, hardware sanity certification, system regression tests, and so on).