Kubernetes (kubespray)

MetalSoft provides an application extension that deploys vanilla upstream Kubernetes on bare-metal servers and operates it afterwards. Tenants deploy the cluster from the Infrastructure Designer; MetalSoft installs Ubuntu 26.04 on every node, and the extension’s Ansible playbooks then build the cluster with kubespray, the upstream Kubernetes SIG installer. Editing the deployment later scales workers or control-plane nodes, upgrades Kubernetes, or re-applies the configuration.

The extension definition and Ansible playbooks live in the Kubernetes (kubespray) extension repository. Its README documents the internals. This page covers the MetalSoft-side setup and day-2 operations.

What gets deployed

Component	Version	Notes
Kubernetes	1.36.4 (default) or 1.35.8	Upgrades one minor version at a time
kubespray	v2.32.0	Runs from the extension’s execution environment; nothing is installed on the Site Controller
Container runtime	containerd	docker.io images are pulled through mirror.gcr.io first
etcd	Stacked on the control-plane nodes	1, 3 or 5 members
CNI	Calico, VXLAN
kube-proxy	iptables (default) or nftables (experimental)
API endpoint	kube-vip, ARP mode	A virtual IP that moves between the control-plane nodes
LoadBalancer Services	MetalLB v0.16.1, L2 mode	IPs from an address range that MetalSoft allocates
Cluster DNS	CoreDNS and NodeLocal DNSCache
Storage	local-path-provisioner, default StorageClass	Node-local disk, no replication
Add-ons	metrics-server (on), Gateway API CRDs (off)
Secrets	Encrypted at rest	Enabled at creation

The cluster has two node groups: k8scp (control plane, 1, 3 or 5 nodes) and k8sworker (1 to 100 workers). Control-plane nodes run etcd and the Kubernetes control plane and do not run regular workloads. All nodes attach to one untagged network, which carries the node IPs, the API virtual IP and the LoadBalancer IPs.

The deployment is fully unattended, and re-runs are safe: the extension keeps the cluster state on the control-plane nodes, checks the current state before each step and resumes a failed run from where it stopped.

Prerequisites

1. A MetalSoft controller running v7.5.0 or newer, with the Ansible Runner capability enabled on the site controller. See Enabling the Ansible Runner Capability for more details.
2. The OS template ubuntu-26-04-lts-waitforssh registered at the target site (see Registering the OS template).
3. A logical network profile at the target site for the cluster network (see Preparing the logical network).
4. Outbound internet access from the cluster network (see Internet access).
5. Servers that meet the server requirements.
6. The extension’s site configuration saved on every target site (see Creating the Kubernetes application extension). Because the extension declares site-level configuration variables, a deployment fails with “not enabled on site” until an operator opens the extension’s site config on each site and saves it. All variables ship with defaults, so this is a one-click save unless overrides are needed.
7. Optional: a DNS workflow extension, such as PowerDNS, so that the api.<cluster>.<zone> record resolves. Without one, the cluster works on IP addresses only, and the API URL uses the virtual IP.

Server requirements

Item	Requirement
CPU	x86_64. Control-plane nodes: at least 2 cores
Memory	Control-plane nodes: at least 8 GB
Disk	One disk for the OS. Pods using the default local-path StorageClass store their data under /opt/local-path-provisioner on the node’s OS disk
Network	At least one NIC on plain switch ports

Control-plane and worker nodes use separate server types, so control-plane nodes can be smaller servers.

Registering the OS template

The extension requires the ubuntu-26-04-lts-waitforssh OS template (Ubuntu/26.04/oob-u26-04-lts-waitforssh in the MetalSoft OS templates repository). It is the stock Ubuntu 26.04 LTS template, except that the installer reboots into the installed OS instead of powering the server off, and the template’s ready method is wait_for_ssh.

Do not use the stock ubuntu-26-04-lts template

MetalSoft runs the extension’s post-deploy Ansible tasks before it powers the instances back on. With the stock template, which powers the servers off after installation, the servers are still off when the playbook starts and the deployment fails with No route to host.

1a. Create the template (with internet access):

metalcloud-cli os-template list-repo
metalcloud-cli os-template create-from-repo Ubuntu/26.04/oob-u26-04-lts-waitforssh

1b. Create the template (air-gapped). On a machine with internet access, clone the repository:

git clone https://github.com/metalsoft-io/os-templates

Copy the os-templates directory, including its .git files, to a machine that runs the CLI and has access to the Global Controller, then run:

metalcloud-cli os-template create-from-directory Ubuntu/26.04/oob-u26-04-lts-waitforssh --dir /path/to/os-templates

The same template is used by the Slurm extension, so it may already be registered.

Preparing the logical network

The extension declares one logical network, k8s-net. A logical network profile with the label referenced by its profileLabel (k8s-net by default) must exist at every site where the cluster will be deployed. Go to Admin > Fabrics > fabric > Logical networks and create or adjust the profile:

• Untagged (native VLAN) on the server ports, as one L2 segment. The API virtual IP and the MetalLB addresses move between nodes with ARP, so all nodes must share the segment.
• A default gateway and outbound internet access. The nodes download Kubernetes components during the deployment.
• Enough addresses: one per node NIC on the network, plus one for the API virtual IP and five for the MetalLB range (the extension requests 4 addresses; MetalSoft reserves a block of 5).
• One VLAN per cluster. If the profile hands every infrastructure the same VLAN, only one cluster can use it at a time, and a second deployment never starts.

The extension creates the following allocations on the network:

Allocation	Use
api-vip	The Kubernetes API virtual IP, held by one control-plane node at a time. MetalSoft also creates the DNS record api.<cluster>.<zone> for it when a DNS extension is enabled.
lb-pool	The address range for LoadBalancer Services, handed to MetalLB.

Clients that use the cluster (kubectl, CI systems, users of LoadBalancer Services) must be able to reach these addresses. If the subnet is private, route it or put a proxy in front of it.

Internet access

The nodes need outbound HTTPS access to:

• github.com, objects.githubusercontent.com, raw.githubusercontent.com
• dl.k8s.io, registry.k8s.io, quay.io, ghcr.io, mirror.gcr.io, registry-1.docker.io
• the Ubuntu package archive

They also need DNS resolution through the site’s resolvers (or the site_dns_servers site configuration).

Creating the Kubernetes application extension

1. Clone the repository containing the extension definition:

git clone https://github.com/metalsoft-io/metalsoft-extensions
cd metalsoft-extensions/kubernetes-kubespray

The definition declares two assets: the Ansible bundle (AnsibleBundle) and the execution environment image (OciImage). Both are already built and hosted, so no repackaging is needed unless you change the playbooks. Adjust infrastructure.logicalNetworks[].profileLabel to match the logical network profile at your sites before you create the extension.

2. Create and activate the extension:

metalcloud-cli extension create kubernetes-kubespray application "Vanilla upstream Kubernetes via kubespray" --definition-source extension.json
metalcloud-cli extension activate <id-of-created-extension>

Activated extensions cannot be edited in place. To change the definition, bump extensionVersion and create the extension again. MetalSoft keeps each version as a separate extension, and an existing cluster stays on the version it was created with.

3. Enable the extension on the site level and save its configuration. After activation, the extension must be activated per site on the site level configuration page. This is also where its configVars, the operator-set, site-level settings, are configured, and it must be done before the first deploy. To save the default configuration from the CLI:

jq '[.configVars[] | {label, value: .defaultValue}]' extension.json \
  | metalcloud-cli extension site-config set <id-of-created-extension> <site> --config-source pipe

You are now ready to deploy the Kubernetes application from the Infrastructure Designer.

Site configuration (configVars)

An operator sets these once per site. They apply to every Kubernetes deployment at that site:

Variable	Type	Default	Description
site_kubespray_overrides	String	""	Additional kubespray settings (group_vars), as YAML or JSON, or base64 of either. Settings the extension manages (API endpoint, etcd, MetalLB, node names and others) are rejected. Settings that cannot change on a running cluster (such as the CNI) apply only when a cluster is created.
site_dns_servers	String	""	Comma-separated upstream DNS servers for CoreDNS, at most 3. Empty uses the nodes’ resolvers.
site_ntp_servers	String	""	Comma-separated NTP servers for the nodes (chrony). Empty uses the site’s NTP servers, else Ubuntu’s defaults.
site_admin_cert_validity_days	Integer	365	Validity of the admin kubeconfig’s client certificate.
site_auto_renew_certificates	Boolean	true	Monthly automatic renewal of the control-plane certificates.
site_backup_retention_count	Integer	5	Number of etcd snapshots and kubeadm backups kept on the control-plane nodes.
site_ansible_forks	Integer	50	Parallelism of the kubespray runs.

The site configuration is recorded on each cluster when it is applied: at creation, and when a tenant raises converge_generation. Other edits keep using the recorded configuration, so that new nodes match the existing ones. A cluster picks up a changed site configuration only on its next converge.

Deployment inputs

Each deployment created from the Infrastructure Designer takes the following inputs:

Input	Type	Default	Description
control_plane_server_type	ServerType	—	Server type for the control-plane nodes. Set at creation only.
worker_server_type	ServerType	—	Server type for the workers. Set at creation only.
os_template	OsTemplate	—	OS template for all nodes. Must be ubuntu-26-04-lts-waitforssh. Set at creation only.
control_plane_count	Integer	1	1, 3 or 5. Changes by exactly 2. An increase can share its edit only with a worker_count increase.
worker_count	Integer	1	1 to 100.
kube_version	Enum	1.36.4	Kubernetes version. Upgrades by one minor version at a time, in an edit of its own; downgrades are refused.
kube_proxy_mode	Enum	iptables	iptables or nftables (experimental). Set at creation only.
pod_network_cidr	String	10.233.64.0/18	Pod network. Must not overlap the node network. Set at creation only.
service_network_cidr	String	10.233.0.0/18	Service network. Must not overlap the node network. Set at creation only.
cluster_dns_domain	String	cluster.local	Cluster DNS domain. Set at creation only.
metrics_server_enabled	Boolean	true	Deploy metrics-server. Can be turned on later, not off.
local_path_provisioner_enabled	Boolean	true	Deploy local-path-provisioner as the default StorageClass. Can be turned on later, not off.
gateway_api_crds_enabled	Boolean	false	Install the Gateway API CRDs. Can be turned on later, not off.
metallb_enabled	Boolean	true	Deploy MetalLB for LoadBalancer Services. Can be turned on later, not off.
drain_timeout_minutes	Integer	10	1 to 30. Time allowed to drain each node during scale-in and upgrades.
drain_policy	Enum	respect-pdb	respect-pdb stops the operation when a PodDisruptionBudget blocks a drain. force-after-timeout evicts the remaining pods after the drain timeout.
converge_generation	Integer	0	Raise it to re-run the full kubespray configuration and apply the current site configuration.
admin_credential_generation	Integer	1	Raise it to issue a new admin kubeconfig and revoke the previous one.

The cluster is named after the extension instance.

Deployment flow

1. MetalSoft allocates the servers and the addresses and installs Ubuntu 26.04 on every node (about 25 minutes). The OS template reboots the nodes, and MetalSoft hands them over once they answer on SSH.
2. The extension’s postDeploy task runs deploy.yaml on the Site Controller:
   1. Preflight checks on the inputs, the site configuration and the allocated addresses.
   2. Prepares the nodes: time synchronization (chrony), firewall (ufw) disabled, cluster identity markers.
   3. Runs kubespray to create the cluster, with the API endpoint on the virtual IP.
   4. Deploys MetalLB with the allocated address range.
   5. Issues the admin kubeconfig, and checks that it has cluster-admin rights.
   6. Records the cluster state on the control-plane nodes, runs health checks and publishes the outputs.

A 1+1 cluster typically takes about 45 minutes from start to a running cluster, including the OS installation. A 3+2 cluster takes about 10 minutes longer.

Outputs

After every successful run, the extension publishes the following outputs on the extension instance:

Output	Description
api_url	Kubernetes API URL: https://api.<cluster>.<zone>:6443 when that name resolves to the virtual IP, else https://<virtual IP>:6443
api_vip	API virtual IP
api_fqdn	API DNS name. Empty when it does not resolve to the virtual IP
admin_kubeconfig	Secret. Kubeconfig of the cluster administrator identity metalsoft-admin-g<N>
admin_kubeconfig_expires	Expiry of the admin kubeconfig’s certificate
admin_credential	Name of the admin identity, and the command to revoke it
ca_cert	Cluster CA certificate (PEM)
kubernetes_version	Running Kubernetes version
kubespray_version	kubespray release and execution environment used
control_plane_nodes, worker_nodes	Node names, comma-separated
lb_pool_ranges	MetalLB address range(s)
cluster_id	Unique cluster ID
cluster_status	Health at the end of the last run: ready, or degraded followed by the nodes that are not Ready
etcd_last_snapshot	Name of the newest etcd snapshot

A run that cannot produce complete outputs fails instead of publishing partial ones.

Using the cluster

1. Copy the admin_kubeconfig output into a file, for example ~/.kube/config.
2. From a machine that can reach the API virtual IP:

kubectl get nodes -o wide
kubectl get pods -A

3. Expose an application with a LoadBalancer Service. MetalLB assigns it an address from lb_pool_ranges:

kubectl create deployment web --image=nginx --replicas=2
kubectl expose deployment web --port=80 --type=LoadBalancer
kubectl get service web

4. Persistent volume claims without a StorageClass use local-path. The volume lives on the disk of the node that first runs the pod, and the pod stays on that node.

Protect the admin kubeconfig

The admin kubeconfig has full cluster-admin rights. Create less privileged identities for daily use, and revoke the admin identity if it leaks (see Revoking the admin kubeconfig).

Scaling and day-2 operations

All day-2 operations are edits of the deployment’s inputs, followed by a deploy. The extension refuses unsafe edits in its preDeploy task, before any server changes (see Refused edits).

Edit	What happens
worker_count up	New servers are installed and joined to the cluster. Existing nodes are not touched.
worker_count down	MetalSoft chooses which servers leave. Before the infrastructure change, the extension cordons all leaving nodes, drains them within drain_timeout_minutes (according to drain_policy) and removes them from the cluster. A leaving worker that is powered off or unreachable is not drained: the extension marks it out of service, so Kubernetes moves its pods to the remaining workers at once. At least one schedulable worker must remain.
control_plane_count +2	Takes an etcd snapshot, joins the new nodes to etcd and the control plane, and refreshes the existing control-plane nodes. When worker_count is raised in the same edit, the new workers join in the same run.
control_plane_count −2	Moves the API virtual IP off the leaving nodes, drains them, removes them from etcd one at a time and refreshes the remaining control-plane nodes. A powered-off leaving node is marked out of service instead of drained, and is removed from etcd first. etcd keeps its quorum throughout.
kube_version up one minor	Takes an etcd snapshot, then upgrades the control-plane nodes one at a time and the workers in up to 5 batches, draining each node first.
Add-on turned on	Deploys the add-on.
converge_generation up	Re-runs the full kubespray configuration at the running version and applies the current site configuration.
admin_credential_generation up	Issues a new admin kubeconfig and revokes the previous one.
Any other input	Health checks, admin kubeconfig renewal when less than 30 days remain, outputs.

Scale-in deploys remove servers, so MetalSoft asks to confirm the data loss before the deploy starts. Pods on leaving workers are rescheduled, but local-path volumes on those workers are lost.

The Kubernetes versions a cluster can reach are those of its extension version. To move beyond them, deploy a new cluster with a newer version of the extension.

Refused edits

The extension refuses the following edits with an E-K8S-NNN message, before any server changes:

• changing an input that is set at creation only;
• stepping kube_version by more than one minor version, or downgrading it;
• combining a control_plane_count or kube_version change with any other change (raising control_plane_count and worker_count together is allowed);
• changing control_plane_count by anything other than 2, or to an even number;
• turning an add-on off;
• lowering converge_generation or admin_credential_generation;
• removing the last schedulable worker.

A refused edit leaves the deploy blocked. Skip the refused tasks (only when the edit adds or removes no servers), set the input back to its previous value and deploy again.

Never skip a failed preDeploy task of a scale-in

When a scale-in’s preDeploy task fails, fix the cause and retry it. If it is skipped, MetalSoft removes the servers without the extension’s preparation. For control-plane nodes, that loses etcd quorum and the cluster.

Revoking the admin kubeconfig

To revoke the current admin kubeconfig immediately, delete its role binding (the command is in the admin_credential output):

kubectl delete clusterrolebinding metalsoft-admin-g<N>

The extension never binds that identity again. To get a new admin kubeconfig, raise admin_credential_generation and deploy; any later edit also issues one.

Backups and recovery

The extension takes an etcd snapshot before every control-plane change and every upgrade, and keeps the newest site_backup_retention_count snapshots in /var/backups/metalsoft-k8s/ on the control-plane nodes. The snapshots stay on the nodes: copy them elsewhere if you need off-cluster backups.

If etcd loses its quorum, stop etcd on a surviving member and restore the newest snapshot with etcdutl snapshot restore as a single member, or use kubespray’s recover-control-plane.yml playbook. See the kubespray documentation.

Execution environment

The extension’s Ansible playbooks run on the Site Controller inside an execution environment (ee-kubernetes-kubespray), a container image holding the playbooks’ dependencies and the pinned kubespray release. The extension ships its own image, declared as an OciImage asset in the extension definition and selected by each Ansible task through the ee option, so no Site Controller configuration is required. The image is based on UBI 10 and requires an x86-64-v3 (AVX2) CPU on the Site Controller. To customize it, rebuild the image from the execution-environment.yml recipe in the extension repository (on an x86_64 machine), and update the OciImage asset in the definition with a new tag.

Limitations

• Fixed MetalLB pool. Each cluster gets one range of 5 addresses for LoadBalancer Services, set at creation.
• No choice of the leaving server. On scale-in MetalSoft chooses which servers leave, so replacing a specific failed node needs operator work.
• No cross-version moves. A cluster stays on the extension version it was created with, and on the Kubernetes versions that version supports.
• Local storage only. local-path volumes are not replicated and are lost with their node. Use a storage system’s CSI driver for replicated storage.
• No firewall management. The extension disables ufw on the nodes; restrict access on the network instead.

Troubleshooting

Symptom	Cause and fix
The deployment fails because the extension is not enabled on the site	Save the extension site config on that site.
The deployment never starts, and the job group reports that the VLAN is not available	Another infrastructure uses the network profile’s VLAN. Use a profile that allocates a separate VLAN per infrastructure.
The playbook fails with No route to host and the servers are powered off	The deployment uses an OS template that powers the servers off after installation. Use ubuntu-26-04-lts-waitforssh.
The deploy fails with an E-K8S-NNN message	The edit was refused. See Refused edits.
A scale-in deploy is refused before any task runs	Confirm the data loss when deploying the scale-in.
A worker scale-in fails while draining	A PodDisruptionBudget blocks the drain under respect-pdb. The message lists the pods. Fix the workload, or set drain_policy to force-after-timeout, then retry the task.
api_url shows the virtual IP instead of the DNS name	api.<cluster>.<zone> does not resolve to the virtual IP from the nodes, usually because no DNS extension is enabled. The cluster works on the IP address.
cluster_status is degraded	Some nodes were not Ready at the end of the run; the status names them. Check them with kubectl describe node, then run any edit to refresh the status.
Image pulls or kubespray downloads fail	Check the nodes’ outbound access to the hosts in Internet access.

Retrying a failed task re-runs the playbook from the recorded cluster state. Each run’s output is available in the extension instance’s job log. The detailed kubespray log stays inside the run’s container on the Site Controller; the job log shows a summary of failed hosts and the last errors.