Setup of MetalSoft Kubernetes Cluster on K3S

Base setup

To setup MetalSoft Kubernetes Cluster on K3S, we usually use a clean Ubuntu 22.04 installation on which we install k3s:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=servicelb --disable-cloud-controller --write-kubeconfig-mode=600 --disable=traefik" sh -
test -f /etc/rancher/k3s/k3s.yaml && chmod 600 /etc/rancher/k3s/k3s.yaml && export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" |tee -a ${HOME}/.bashrc >> /etc/bash.bashrc

Install Helm:

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add traefik https://helm.traefik.io/traefik
helm repo update

Install MetalLB:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/config/manifests/metallb-native.yaml

MetalSoft Setup

Once k3s is running, navigate to the MetalSoft manifests folder (provided by MetalSoft)

cd manifests

Run the following script

manifests # ./scripts/prerequisites-setup -h

        Available parameters:
         -m = path to manifests folder (default: .)
         -i = MetalLB IP
         -k = absolute path to SSL key file (PEM format)
         -c = absolute path to SSL crt file (PEM format)
         -a = absolute path to CA  crt file (PEM format)

example manifests # ./scripts/prerequisites-setup -i 192.168.1.2 -k ./sslcert/cert.key -c ./sslcert/cert.crt -a ./sslcert/ca.crt

Caution

Please ensure you backup the encryption_keys value to a safe location outside of the Global Controller setup. If lost, will cause loss of data and access to the platform. The encryption_keys value can be retrieved by editing ./manifests/secrets.yaml and copying the encryption_keys line.

At this point, you should be able to access the UI via the hostname you’ve provided. (this hostname should point to the IP for the UI)

Manual steps for MetalSoft Setup if required

Caution

The following steps are only required for troubleshooting if MetalSoft request these steps to be run. These steps are covered by the ./scripts/prerequisites-setup script above, and should not be run without MetalSoft’s assistance.

Once k3s is running, navigate to the MetalSoft manifests folder (provided by MetalSoft)

cd manifests

If storage is set to NFS, you will need to pre-create the folders which NFS will use and access from the Nodes. Usually these folders can be seen with:

grep path: volumes.yaml |awk '{print $2}'

To configure MetalLB with the IP to access the UI:

export ip=<YOUR_IPv4>

then run:

cat <<EOF | kubectl -n metallb-system apply -f -
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool
  namespace: metallb-system
spec:
  addresses:
  - ${ip}-${ip}
EOF

Add SSL certificate to Kubernetes cluster

• Generate your SSL certificate and key. If using a self signed certificate, add the CA certificate to the ConfigMap of the Global Controller.
  To put the CA certificate in the ConfigMap on the Global Controller, edit configmaps.yaml and put the CA certificate in the ca-pemstore ConfigMap resource definition. For example:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ca-pemstore
  namespace: eveng-qa02-metalcloud
data:
  metalsoft_CA.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

• Copy the SSL certificate and key to the Kubernetes cluster and run these

To add the certificate to secrets.yaml

cat crt.pem <crt_chain.pem> |base64 -w0 Copy the output into the tls.crt line of secrets.yaml

To add the key to secrets.yaml

cat key.pem |base64 -w0 Copy the output into the tls.key line of secrets.yaml

Generate encryption keys in the Kubernetes cluster

Caution

Please ensure you backup this encryption_keys value to a safe location outside of the Global Controller setup. If lost, will cause loss of data and access to the platform

cd manifests
./scripts/encryption_keys_gen

Edit secrets.yaml which is in the manifests folder and add the above output to the encryption_keys line

Initiate the MetalSoft setup:

When in manifests folder, create the Namespace:

kubectl apply -f namespace.yaml

Export the name of the created Namespace, so it can be used in the next few steps:

export ns="YOUR_NAMESPACE_FROM_namespace.yaml"

Apply the manifests:

find . -maxdepth 1 -type f \( -name "*.yaml" ! -name traefik-helm-chart-configvalues.yaml \) -size +1|while read z;do
echo "applying manifest: $z ..."
kubectl -n $ns apply -f $z >/dev/null
done

Install traefik:

test -d traefik-helmchart && helm upgrade --install traefik-${ns} ./traefik-helmchart --values traefik-helm-chart-configvalues.yaml -n $ns

Install Debug utilities:

touch /usr/local/bin/diagnostics-collector
curl -fsSLk https://raw.githubusercontent.com/metalsoft-io/scripts/main/env-scripts/metalsoft-updates|bash
curl -fsSLo /usr/local/bin/ms-prerequisite-check https://repo.metalsoft.io/extra/ms-prerequisite-check && chmod +x /usr/local/bin/ms-prerequisite-check

Check the state of the setup

When all of the above is complete, you can check the state of the pods, and services:

kubectl -n $ns get pods

kubectl -n $ns get svc | grep traefik
# this command should show the IP you've assigned for the UI, and not show '<pending>'