Deploying Metalsoft

This guide will cover setting up a MetalSoft-managed datacenter. In general lines setting up a datacenter involves the following steps:

  1. Install Equipment, Setup cabling [DCOps]
  2. Setup a MetalSoft VM [DevOps]
  3. (optional) Install MetalSoft controller software [DevOps]
  4. Setup a router/VPN concentrator VM [DevOps]
  5. Allocate IP ranges (subnets) [Networking]
  6. Configure OOB switch (if not already configured) [Networking]
  7. Create new Datacenter in MetalSoft [DevOps]
  8. Register IP Ranges (subnets) in MetalSoft [DevOps]
  9. Install DC Agents in DC Agents VM [DevOps]
  10. Register ToR switch in MetalSoft [DevOps]
  11. Register servers [DevOps]
  12. Install & Register storage (optional) [DevOps/StorageOps]

Note that this tutorial uses the CLI but the admin interface can also be used.

Installing the MetalSoft controller

Follow this guide to setup the controller.

Required Hardware

This reference architecture uses the following hardware:

  • 5x servers with dedicated BMC ports and a minimum of 2 ethernet ports, 4 recommended. A local disk is recommended. These will be the usable servers.
  • 1x ESXi server - Min 8GB RAM, 4 ethernet ports. 100GB HDD Example: This will host two VMs: the datacenter agents machine and the “router/VPN” VM.
  • 1x Top-of-Rack (ToR) switch
  • 1x Out-of-Band (OOB) switch
  • 1x Terminal (Console) server (Example: Avocent)

It is possible to build a reference architecture without a Terminal server but it’s presence is recommended for remote troubleshooting.

Cabling

The cabling is shown as follows:

../_images/preparing_to_setup_a_datacenter_01.svg

Note: In this setup each server is connected with 4 ports to the ToR switch and an extra dedicated BMC port to the OOB network. MetalSoft doesn’t need an extra provisioning network and thus doesn’t need shared BMC ports. If present MetalSoft will simply ignore the extra port in the OS. Provisioning is done using the In-band network by reconfiguring the switch during the setup.

In fact, our recommendation is also NOT to use the shared port for security reasons. A rogue tenant could potentially use it to access the OOB network.

IP Allocation

MetalSoft uses several IP ranges with various purposes such as OOB IPs, In-band IPs, storage IPs etc. The following IP ranges are the recommended defaults:

../_images/preparing_to_setup_a_datacenter_02.svg

  1. WAN subnet - IPv4

    This subnet will be used to reach tenant servers from outside the managed datacenter. For a baremetal-as-a-service IaaS product this would be a routable IP address subnet.

    Example: 10.255.226.0/24

  2. WAN subnet - IPv6:

    This subnet will be used to reach tenant servers from outside the managed datacenter. For a baremetal-as-a-service IaaS product this would be a routable IP address subnet.

    Example: fd1f:8bbb:56b3:2000:0000:0000:0000:0000/53)

  3. SAN client subnet - IPv4

    This subnet is assigned to servers on the SAN interface.

    Example: 100.64.0.0/21

  4. SAN storage subnet - IPv4

    This subnet is used for the storage arrays. Servers will get (via DHCP) a static route that pushes storage traffic (towards this subnet) via the SAN interface on the server.

    Example: 100.96.0.0/16

  5. OOB Subnet - IPv4

    This subnet will be used to allocate IPs to server’s BMCs and potentially to switch BMCs.

    Example: 10.255.227.0/24

  6. Quarantine subnet - IPv4

    This subnet will be used by MetalSoft to allocate IPs to servers during server registration and before the server’s identity has been established. All unused ports on the TOR switch are configured to be part of the Quarantine VLAN (by convention VLAN ID 5). This allows zero touch enrollment of servers but does not allow a physical attack to access tenant networks.

    Example: 172.16.0.0/24

  7. Dummy subnet (”Primary” WAN IP Subnet)

    This is only required for HP older switches. These IPs are permanently allocated on the interfaces as “Primary” IPs in order to avoid a downtime when manipulating secondary IP addresses on the switch interfaces. Traffic should not be routed to them in any way.

    Example: 172.24.4.0/22

  8. Router ‘public’ IP Address

    This is the IP configured on the router, reachable from the outside world. It is the entry point towards the managed datacenter. It is also the endpoint for VPN connections if the VPN has been configured on this machine.

    Example: 83.246.0.140

  9. Router ‘in-band’ IP Address

    This is the IP configured on the router, reachable from the in-band network. This will be used as a gateway towards the datacenter agents as well as towards the internet or the rest of the network.

    Example: 172.16.10.1

  10. Router ‘out-of-band’ IP Address

    This is the IP configured on the router, reachable from the out-of-band network. This will be used as a gateway towards the datacenter agents for on-boarding of zero-touch servers and BMC configuration.

    Example: 10.255.227.1

  11. DC Agents machine IP Address

    This is the IP configured on the datacenter agents machine.

    Example: 172.16.10.6

  12. Top of Rack (ToR) 01 switch in-band IP Address

    This is the switch in-band IP address. In this example setup, it is used for:

    1. As the gateway towards the Quarantine
    2. As the gateway towards the SAN in-band network
    3. As the in-band management interface

    Example IP 172.16.10.2

  13. Quarantine L3 interface

    This virtual switch interface (VSI) interface is used as a gateway for the quarantine network. This IP must be configured on one of the ToR switches.

    Example: 172.16.0.1

Router configuration

The reference architecture uses a router machine that bridges the in-band, out-of-band and the uplink links.

The netplan configuration should look like this:

edit the /etc/netplan/50-cloud-init.yaml

network:
  version: 2
  ethernets:
    ens160:
      addresses: [83.246.0.140/24]
      gateway4: 83.246.0.1
      nameservers:
        addresses:
        - 8.8.8.8
        - 1.1.1.1
    ens192:
      addresses: [10.255.227.1/24]
    ens224:
      addresses: [172.16.10.1/30]
      routes:
        #this is a static route that takes DHCP traffic coming from the agents machine
        #to the quarantine network via the in-band link.
        - to: 172.16.0.0/16
          via: 172.16.10.2
        #this is a static route that allows a connection between a datacenter agent and the SAN network 
        #via one of the ToR switches. This is only required to initiate volume replication processes via SSH.
        - to: 100.96.0.0/16
          via: 172.16.10.2
        #this is a static route that allows a connection between a datacenter agent and the SAN L3 interfaces
        #used as gateways on the ToR switches. This is required to allow the server to perform diskless boot.
        - to: 100.64.0.0/21
          via: 172.16.10.2
        #this is a static route that allows a connection between the agent and the WAN L3 interfaces
        #used on ToR switches
        - to: 10.255.226.0/24
          via: 172.16.10.2
        #this is a static route that allows DHCP traffic between the agent and the WAN gateway (VLAN VSI) interface's  'primary' (dummy) ip address
        #used on ToR switches
        - to: 172.24.4.0/22
          via: 172.16.10.2
    ens256:
      addresses: [172.16.10.5/30]

Apply the configuration:

root@router:~# netplan --debug generate
root@router:~# netplan --debug apply

Check that the routes have been added properly. You should see routes for: Quarantine, OOB, In-band towards the WAN interfaces and in-band towards the DC agents machine and towards the in-band SAN network.

root@router:~# ip route
default via 83.246.0.1 dev ens160 proto static  #default gw
10.255.226.0/24 via 172.16.10.2 dev ens224 proto static #in-band towards the ip ranges configured on the server's WAN ports
10.255.227.0/24 dev ens192 proto kernel scope link src 10.255.227.1  #OOB subnet
83.246.0.0/24 dev ens160 proto kernel scope link src 83.246.0.140  #uplink
100.64.0.0/21 via 172.16.10.2 dev ens224 proto static  #in-band towards the ip range configured on the server's SAN ports. This allows DHCP relay traffic to WAN.
100.96.0.0/16 via 172.16.10.2 dev ens224 #in-band SAN, configured on the SAN array
172.16.0.0/16 via 172.16.10.2 dev ens224 #quarantine subnet
172.16.10.0/30 dev ens224 proto kernel scope link src 172.16.10.1  #in-band towards the rest of the network and WAN IP addresses
172.16.10.4/30 dev ens256 proto kernel scope link src 172.16.10.5  #towards the DC agent machine
172.24.4.0/22 via 172.16.10.2 dev ens224 proto static #in-band towards the WAN gateway L3 primary ip address on the ToR. This allows DHCP relay traffic to WAN.

The VPN configuration is beyond the scope of this tutorial. Visit How to Set Up an OpenVPN server on Ubuntu 18.04 for more details.

Create the datacenter record in MetalSoft

Create MetalSoft datacenterrecord. Create a file with the following contents:

#DC Agents primary IP
BSIVRRPListenIPv4: 172.16.10.6
#DC Agents secondary IP list (comma separated)
BSIMachineListenIPv4List: 
    - 172.16.10.6
#DC Agents IPs subnet, in CIDR format (x.x.x.x/x)
BSIMachinesSubnetIPv4CIDR: 172.16.10.6/24

#This is the ip from which the controller will
#see the incoming connections from this datacenter's DC agents
BSIExternallyVisibleIPv4:  83.246.0.140

#HTTP(S) root URL for the general purpose HTTP repository 
#(package manager resources, deploy setup files, etc.). It does not end in a slash.
#Make sure the DNSServers property below includes a server that will be authoritative for the domainname listed here otherwise the agent will not resolve the DNS record.
repoURLRoot: http://repo.bigstepcloud.com

#Repo URL root for the quarantine network (installation network) where DNS is not available yet. 
#Must not contain a DNS entry. it is usually the ip to which the above resolves 
#to that is reachable from the quarantine network
repoURLRootQuarantineNetwork: http://repo.bigstepcloud.com

#this is the TFTP DC Agent's ip. This is usually the same as BSIVRRPListenIPv4
TFTPServerWANVRRPListenIPv4: 172.16.10.6

#the IP range of the storage arrays in your network
SANRoutedSubnet: 100.96.0.0/16

NTPServers:
    - 84.40.58.44
    - 84.40.58.45

DNSServers:
    #make sure this resolves the url at repoURLRoot
    - 1.1.1.1
    - 8.8.8.8

#This setting controls the tenant isolation mechanism

switchProvisioner:    
    #must be one of LANProvisioner, VPLSProvisioner, VLANProvisioner
    type: VPLSProvisioner    
    #The ACL number for SAN ports
    ACLSAN: 3999
    #The ACL number for WAN ports
    ACLWAN: 3399
    #The ACL range for ACL rules on the SAN ports. Per switch
    SANACLRange: 3700-3998
    #the VLAN Range for LAN networks, per ToR switch. Make sure these don't overlap.
    ToRLANVLANRange: 400-699
    #the VLAN Range for SAN networks, per ToR switch. Make sure these don't overlap.
    ToRSANVLANRange: 700-999
    #the VLAN Range for WAN networks, per ToR switch. Make sure these don't overlap.
    ToRWANVLANRange: 100-300
    #The VLAN to use for quarantine network
    quarantineVLANID: 5
    #VLAN Range for WAN ports 'North' switches.
    NorthWANVLANRange: 1001-2000
    
#switchProvisioner:    
#    type: VLANProvisioner
#    #The VLAN to use for quarantine network
#    quarantineVLANID: 5
#    #VLAN range for WAN networks
#    WANVLANRange: 100-199
#    #VLAN range for LAN networks
#    LANVLANRange: 200-299

Create the record using the CLI:

$ metalcloud-cli datacenter create --config examples/datacenter.yaml --format yaml --id uk-london --title 'UK, London 01'

$ metalcloud-cli datacenter get --id uk-london --show-config --format yaml

DATACENTER OVERVIEW
-------------------
Datacenter name (label): uk-london
Datacenter display name (title): London, UK
User: 
Flags: 
Parent: 
Type: metal_cloud
Created: 2020-07-31T20:03:55Z
Updated: 2020-08-07T12:11:12Z

Consult Adding a New Datacenter for more details on available options.

Switch configurations

Out-of-band (OOB) Switch configuration

Normally the OOB network is already configured and no special requirements apart form reachability between all the hosts is required. If a new OOB switch is being added at the minimum:

  1. Configure serial console
  2. Reset admin password
  3. Configure maximum number of VLANs to 256

Example configuration:

Top-of-Rack (ToR) switch configuration

MetalSoft supports automatic configuration on certain switches via the ‘Zerotouch’ configuration mechanism (ONIE). The following is an example of a manual configuration of a ToR switch:

  1. Configure management IPs (in this scenario we use in-band management IP for the switch but we could have used out-of-band management as well. IP used in this case is 172.16.10.2)
  2. Configure telnet
  3. Enable SSH console access
  4. Create quarantine VLAN 5
  5. Add all ports into the quarantine VLAN
  6. Configure DHCP relay with the DCAgents machine as a destination
  7. Activate Restconf if not automatically activated and if the device supports it.

Example configuration for a HP5900 switch:

sysname HP5900-H1060
#
telnet server enable
telnet server acl 2000
telnet server ipv6 acl ipv6 2000
#
irf mac-address persistent timer
irf auto-update enable
undo irf link-delay
irf member 1 priority 1
#
link-aggregation global load-sharing mode destination-ip source-ip destination-port source-port
#
undo link-aggregation load-sharing mode local-first
#
ip ttl-expires enable
#
ipv6 option drop enable
#
dhcp enable
#
lldp global enable
#
cut-through enable
burst-mode enable
#
system-working-mode standard
password-recovery enable
#
vlan 1
#
vlan 5
name QUARANTINE
description QUARANTINE

interface Vlan-interface5
description QUARANTINE
ip address 172.16.0.1 255.255.255.0
dhcp select relay
dhcp relay information circuit-id string VLAN5
dhcp relay information enable
dhcp relay information remote-id sysname
dhcp relay server-address 172.16.10.6

Optional recommended settings:

The switches are automatically managed but MetalSoft allows for local customization of the switch settings on elements beyond the scope of the ports being managed.

  1. Configure global hash settings for LAG (layer3+4)
  2. Enable cut-through switching + dynamic buffer allocation (DBA)
  3. Enable spanning-tree
  4. Enable TTL for IPv4, Option Drop for Ipv6 etc. for security hardening
  5. Configure “administrative distance” for static routes so that they are preferred over learned prefixes. This is default on most servers.

Complete example configuration:

Register (enroll) a switch

Registering switches can be done manually or automatically via ONIE - if the equipment supports it. The following instructions will register a switch manually. Check that the IP ranges specified match the allocated ones for your local infrastructure:

Create a configuration file:

#the hostname of the switch
identifierString: HP5900-H1060
description: ToR switch
#the datacenter label
datacenterName: uk-london
#
provisionerType: vpls
provisionerPosition: tor
driver: hp5900
#connection details. This can be in-band management iP or the out-of-band. In this setup we used the out of band 
managementAddress: 172.16.10.2
managementProtocol: ssh
managementPort: 22
managementUsername: admin
managementPassword: 

#Network address (first IP) of the WAN IPv6 subnet.
#These will be the IPs that will be allocated to hosts
#that need to be reachable from outside an infrastructure
#Both Ipv4 and IPv6 IPs are allocated to servers
#The IPv6 subnet will be automatically created
primaryWANIPv6SubnetPool: fd1f:8bbb:56b3:2000:0000:0000:1000:0000
#Subnet size of the above subnet. If it is fd1f:8bbb:56b3:2000:0000:0000:1000:0000/53 in CIRD notation this number will be 53
primaryWANIPv6SubnetPrefixSize: 53

#Network address (first IP) of the SAN subnet. 
primarySANSubnetPool: 100.64.0.1
#By default, the SAN subnet has a netmask of 21.
primarySANSubnetPrefixSize: 21

# The pool from which to allocate ips to servers during registration
quarantineSubnetStart: 172.16.0.0
quarantineSubnetEnd: 172.16.0.255
#the subnet size (CIDR) of the above pool
quarantineSubnetPrefixSize: 24
#the IP of the VSI in the quarantine network
quarantineSubnetGateway: 172.16.0.1

#This IP range is used as a hack to prevent downtime during dynamic provisioning of 
#ports (since we have multiple secondary IPs on the same interface).
#the traffic is not routed to these IPs.
#The real IPv4 WAN subnet pool needs to be added later using a 'subnet pool'.
primaryWANIPv4SubnetPool: 172.24.4.0
#Subnet size of the above subnet. If it is 172.24.4.0/22 in CIRD notation this number will be 22
primaryWANIPv4SubnetPrefixSize: 22


#Set to true to enable ONIE for this switch.
requiresOSInstall: false
#Set this to the ID of the volume template that holds the ONIE image
volumeTemplateID: 0

Register the new switch by running:

$ metalcloud-cli switch create --raw-config examples/tor_switch.yaml --format yaml

Create a WAN subnet pool

From this subnet pool, infrastructures get allocated entire prefixes that get routed to the infrastructure and from which IPs get allocated on server interfaces designated as WAN interfaces.

Create a file containing:

datacenter: uk-london
prefix: 100.255.226.0
netmask: 255.255.255.0
size: 24
type: ipv4
routable: true
destination: wan

Create the subnet:

$ metalcloud-cli subnet-pool create --config examples/subnet.yaml --format yaml --return-id 

Install the Datacenter Agents on the DC agents machine

Retrieve the configuration URL:

$ metalcloud-cli datacenter get --id uk-london --return-config-url
https://api.poc.metalsoft.io/api/url?rqi=br.ROc8B7Ogy12VrSbVI7koZ9vfpsWs9l3_tjMUd....   

Create directories and export the auto-configuration url:

$ mkdir -p /opt/BSIAgentsVolume /opt/Agent_logs /opt/agents /opt/containerd
$ export DCCONF=`https://api.poc.metalsoft.io/api/url?rqi=br.ROc8B7Ogy12VrSbVI7koZ9vfpsWs9l3_tjM...`

Create a docker compose file in the current directory:

version: '3'
services:
  agents:
    network_mode: host
    container_name: dc-agents
    image: harbor.bigstep.cc/datacenter-agents/datacenter-agents-compiled-v2:latest
    restart: always
    privileged: true
    volumes:
      - /opt/BSIAgentsVolume:/etc/BSIDatacenterAgents
      - /opt/Agent_logs:/var/log/
      - /opt/.ssh:/root/.ssh
    environment:
      - TZ=Etc/UTC
      - URL=${DCCONF}
    hostname: agents-onie
  haproxy:
    network_mode: host
    container_name: dc-haproxy
    image: harbor.bigstep.cc/datacenter-agents/dc-haproxy:latest
    restart: always
    privileged: true
    #this is in case you need to override the default haproxy configuration
    #for example when you need to add SSL certificates
    #volumes:
    #  - /opt/agents/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    environment:
      - TZ=Etc/UTC
    hostname: dc-haproxy

Use the returned URL to install the datacenter agents:

$ docker-compose up -d

Check if the containers are up:

   CONTAINER ID        IMAGE                                                                      COMMAND                  CREATED             STATUS              PORTS               NAMES
b3951a31c4db        harbor.bigstep.cc/datacenter-agents/datacenter-agents-compiled-v2:latest   "docker-entrypoint.s…"   5 hours ago         Up 5 hours                              dc-agents
56ba275db5ea        harbor.bigstep.cc/datacenter-agents/dc-haproxy:latest                      "/docker-entrypoint.…"   6 days ago          Up 57 seconds                           dc-haproxy

Pre-flight checklist

Before proceeding with server registration it is useful to go through the following checklist to verify your configuration:

  1. DC agents machine is able to reach the Quarantine L3 interface

     root@uk-metalsoft-poc-dcagents01  ~# ping 172.16.0.1
      PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
      64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=0.332 ms
    
  2. DC agents machine is able to reach a SAN array in-band interface

    root@uk-metalsoft-poc-dcagents01  ~# ping 100.96.0.2
    PING 100.96.0.2 (100.96.0.2) 56(84) bytes of data.
    64 bytes from 100.96.0.2: icmp_seq=1 ttl=63 time=0.332 ms
    
  3. Router has rule for SAN L3 interfaces (100.64.0.0/21)

    root@bsirouter-hpe01:~# ip r | grep 100.64
    100.64.0.0/21 via 172.16.10.2 dev ens224 proto static 
    
  4. Router has rule is able to reach the OOB network

    root@bsirouter-hpe01:~# ip r | grep 10.255.226
    10.255.226.0/24 via 172.16.10.2 dev ens224 proto static 
    
  5. DC agents machine is able to reach a switch in-band interface

    root@uk-metalsoft-poc-dcagents01  ~# ping 172.16.10.2 
    PING 172.16.10.2 (172.16.10.2) 56(84) bytes of data.
    64 bytes from 172.16.10.2: icmp_seq=1 ttl=254 time=1.01 ms
    
  6. DC agents machine is able to resolve the hostname of the repository using one of the DNS servers configured on the datacenter “DNSServers”:

    root@bsirouter-hpe01:~# nslookup repo.bigstepcloud.com 8.8.8.8
    Server:		8.8.8.8
    Address:	8.8.8.8#53
    
    Non-authoritative answer:
    Name:	repo.bigstepcloud.com
    Address: 84.40.58.43
    
  7. DNS DC agent overrides the DNS record for the repoURL host:

    $ nslookup repo.bigstepcloud.com 172.16.10.6
    Server:		172.16.10.6
    Address:	172.16.10.6#53
    
    Name:	repo.bigstepcloud.com
    Address: 172.16.10.6
    
  8. If using HTTPS for the repository check that HAproxy is configured correctly by configuring a temporary entry in the /etc/hosts file of a server and poinging it to the DC agents machine:

root@bsirouter:~$ cat /etc/hosts
...
172.16.10.6 repo.bigstepcloud.com

Use curl to pull any asset from the repository. This will force a request through the HAproxy and the DC agent proxy

root@bsirouter:~$ curl http://repo.bigstepcloud.com/templates/catalog.json
{
	"OSTemplates":{
		"centos6-8": "CentOS/centos6-8",
		"centos6-9": "CentOS/centos6-9",
		"centos7-4": "CentOS/centos7-4",
		"centos7-5": "CentOS/centos7-5",
                "centos7-6-old": "CentOS/centos7-6-old",
		"centos7-6": "CentOS/centos7-6",
		"centos7-7-old": "CentOS/centos7-7-old",
                "centos7-7": "CentOS/centos7-7",
		"ubuntu16-04": "Ubuntu/ubuntu16-04",
		"WindowsServer2012R2": "Windows/WindowsServer2012R2",
		"WindowsServer2016": "Windows/WindowsServer2016",
		"WindowsServer2019": "Windows/WindowsServer2019",
		"WindowsServer2016Template221118": "Windows/WindowsServer2016Template221118",
		"Cloudware2019Stdv3": "Windows/Cloudware2019Stdv3"
	}
}
Note: Don’t forget to remove the hosts entry afterwards!
  1. Check that the VLAN ranges in the switchProvisioner configured in the datacenter record do not overlap with eachother.
  2. Check that the Controller’s (master DC) repoURL matches that of the datacenter’s:
./metalcloud-cli datacenter get --id master  --show-config --format yaml | grep repoURL
    repoURLRoot: http://repo.bigstepcloud.com
    repoURLRootQuarantineNetwork: http://repo.bigstepcloud.com

These can sometimes causes issues with SAN template replication. This will not be required in the future.

Register (enroll) servers

Follow Adding a New Server for all your servers.

The Servers page should now show several servers registered and available for consumtion: ../_images/servers_page_servers_available.png

Test a server provisioning

$ metalcloud-cli infrastructure create --label test
$ metalcloud-cli instance-array create --boot pxe_iscsi --server-type M8.32v1 --drive-array-template centos7-7 --infra test --label test
$ metalcloud-cli infrastructure deploy --id test --autoconfirm
$ metalcloud-cli infrastructure get --id test
Infrastructure test (198) - datacenter uk-london owner alex.bordei@metalsoft.io
+-------+----------------+----------------+------------------------------------------------------------------------+---------+
| ID    | OBJECT_TYPE    | LABEL          | DETAILS                                                                | STATUS  |
+-------+----------------+----------------+------------------------------------------------------------------------+---------+
| 358   | InstanceArray  | test           | 1 instances (1 RAM, 1 cores, 0 disks pxe_iscsi )                       | active  |
| 46    | DriveArray     | drive-array-46 | 1 drives - 40.0 GB iscsi_hdd CentOS 7.7 [#11] attached to: test [#358] | ordered |
+-------+----------------+----------------+------------------------------------------------------------------------+---------+
Total: 2 elements

Wait until deployment is complete. You now have a fully functional MetalSoft deployment!

Troubleshooting the deployment

If server registration or deployment doesn’t work, follow the Troubleshooting Guide for potential solutions.