Networking overview

MetalSoft is capable of of automatically deploying complex network configurations on a multitude of switch brands including Cisco Nexus, Cisco ACI, Juniper, Dell, Cumulus, HP and others.

The approach that we take is “intent-based” in the sense that the user fully describes the desired network “services” and the system then translates those into actual switch and OS configurations. This is a service-provider-inspired approach where the end-users are “consumers” of network services offered by the “provider”. As such they don’t control which subnets to allocate from, which VLANs will be used etc but they can specify the size of the subnet they need for example.

The user defines via the infrastructure editor or the API (or terraform, ansible etc) how the desired network setup looks like.

The concept of networks and network profiles

Both L2 and L3 configurations on both switches, servers and storage pools are controlled at the same time by using Networks and Network Profiles.

• By connecting an instance port to a Network the user expresses the intent to consume the services of that network from the respective instance.
• By selecting a Network Profile other than the default the user requests some alterations to the default service offered by the respective network.

Types of Networks

There are currently 3 types of networks:

1. LAN networks

   LAN networks typically provide connectivity between servers of the same infrastructure.

   1. An L2 broadcast domain that is shared between ports of the same infrastructure connected to this network but isolated from LAN networks provisioned for other Infrastructures and from other LAN networks. This means that users can use their own IPs if they want to to connect for example the nodes of a Kubernetes cluster. In most implementations this will be achieved by using a unique VLAN ID, VXLAN VNI or EPG.
   2. One or more subnets of sufficient size (taken together) to provide one IP for each port connected to the network. These subnets are provisioned from the default subnet pools configured by admins in the Subnets section. These IPs are usually private IPv4 and IPv6 IPs.
   3. Link aggregation support on the switch side if two or more ports of the same instance are connected to the same LAN network. If the physical ports are connected to two different switches an M-LAG or equivalent concept will be used to create the switch-side aggregated port. See the LACP section for more details.

   More than one LAN networks can be created on the same infrastructure.

2. WAN networks

   WAN networks typically provide connectivity not only between servers but also with the outside world. WAN networks typically provide the following services:

   1. An L2 broadcast domain that is shared between ports of the same infrastructure connected to this network but isolated from WAN networks provisioned for other Infrastructures. This means that users can use their own IPs if they want to to connect for example the nodes of a Kubernetes cluster. In most implementations this will be achieved by using a unique VLAN ID, VXLAN VNI or EPG.
   2. One or more subnets of sufficient size (taken together) to provide one IP for each port connected to the network. These subnets are provisioned from the default subnet pools configured by admins in the Subnets section. These IPs are usually public (routable) IPv4 and IPv6 IPs but not always, depending on the design of the service.
   3. A gateway for all allocated subnets that provides connectivity to outside of the infrastructure. MetalSoft provides a mechanism to provision L3 switch virtual interfaces (SVIs) to act as gateways. These L3 interfaces will be private to this network. Each infrastructure will get their own gateways.
   4. Link aggregation support on the switch side if two or more ports of the same instance are connected to the same WAN network. If the physical ports are connected to two different switches an M-LAG or equivalent concept will be used to create the switch-side aggregated port. See the LACP section for more details. Note that more than one gateway SVIs will be created in this case to ensure redundancy in case a switch fails.

3. SAN networks

   A SAN network is a special network dedicated to SAN traffic. It provides one of the following services depending on the type of storage pool and storage protocols used:

   1. An isolated L2 broadcast domain that is shared between the servers on the same infrastructure connected to the SAN network. This L2 broadcast domain is terminated on the storage pools from which volumes are offered to the instances of the same infrastructure.
   2. An isolated L3 network where the gateway resides on the leaf switch to which the server port is connected to, isolated from other SAN networks with ACLs.
   3. One or more subnets of sufficient size (taken together) to provide one IP for each port connected to the network. These subnets are provisioned from the default subnet pools configured by admins in the Subnets section of type SAN. These IPs are usually private IPv4 and IPv6 IPs but not always, depending on the design of the service.
   4. ISCSi/FC Multi-path support will be enabled if two ports of the same instance are connected to the same SAN network. In this case, different subnets will be allocated for each pair of switch port and storage pool virtual port.

Network profiles

To define the behavior of a network, a network profile can alter:

1. Choose a specific native (access mode) VLAN IDinstead of one that is automatically allocated
2. Specify a series of allowed VLAN IDs on the same ports
3. Choose a specific subnet pool to use for the native VLAN or for the other allowed VLANs.
4. Choose if the SVIs should not be provisioned on the WAN network
5. Choose if a specific network should be extended on a border device to connect to a External Connection.

Refer to Managing network profiles for more details.

Details on link-aggregation

If the user chooses to connect two or more ports to the same WAN or LAN network a link aggregation will be created on the switches and a bond interface will be created on the server.

1. If the two ports lead to different switches defined as a switch-pair then an M-LAG or equivalent multi-chassis link aggregation will be created. LACP type 4 will be used as negotiation protocol. Refer to Managing switch pairs for more details.
2. If the two ports lead to two different switches that are configured in “stacked” configuration in which case a single switch record is registered in MetalSoft, the system will create regular link-aggregations, using LACP type 4.
3. If the two ports lead to the same switch a simple link-aggregation will be created on the respective switch. The link will be configured using LACP type 4.

To ensure consistent behavior, the LACP priority will be higher on the port with the lower index so that if the LACP protocol cannot negotiate with the server (such as during the boot process) the fallback port will always be port with index #0.

How server interface indexes are allocated

To ensure consistent behavior the network ports of the servers will always have indexes based on the switch hostname and then the switch port in lexicographic order irrespective of the actual order in the server. For example if a server is connected to two switches with hostnames sw1 and sw2 and connected to Ethernet10 on both, the port connected to sw1 will always be port index #0 and the port connected to sw2 will always be port #1. If the same server is connected to sw1 Ethernet11 and sw1 Ethernet10 the second port will be index #0 and the first one will be #1.