Slurm HPC cluster

MetalSoft provides an application extension that deploys and manages the lifecycle (install, scale-out, and scale-in) of a Slurm workload manager cluster on bare-metal servers running Ubuntu 26.04 LTS. Tenants deploy the cluster from the Infrastructure Designer, and scale it by editing the extension instance.

The extension definition and Ansible playbooks live in the Slurm extension repository. The extension’s README documents every role and the full deployment flow. This page covers the MetalSoft-side setup.

What gets deployed

Component	Instance array	Count	Role
Head node	slurmhead	1	slurmctld (scheduler), slurmdbd with MySQL 8.4 (accounting), NFS server for /shared, and the login node for tenant users
Compute nodes	slurmcompute	1–128	slurmd on every node, NVIDIA driver on GPU servers, Open MPI 5 with PMIx, and Apptainer

All servers attach to one untagged logical network (slurm-net, one NIC per server). The compute nodes fetch their Slurm configuration from the head node (configless mode), authenticate with MUNGE, and mount the head node’s /shared export for home directories.

Software	Version or source
Operating system	Ubuntu 26.04 LTS, x86_64 only
Slurm (default)	SchedMD 26.05.4, built on the head node from the checksum-pinned release tarball (25.11.8 selectable)
Slurm (optional)	Ubuntu archive slurm-wlm 25.11.2, which has known unpatched CVEs and requires a site-level opt-in
Accounting database	MySQL 8.4, local to the head node
MPI	Open MPI 5 with PMIx 5 (srun --mpi=pmix)
Containers	Apptainer from the official Apptainer PPA (optional)
NVIDIA	Ubuntu archive -server-open driver with Canonical-signed prebuilt kernel modules (no DKMS, compatible with Secure Boot), and Fabric Manager on NVSwitch systems

Prerequisites

1. A MetalSoft controller running release 7.5 or newer, with the Ansible Runner capability enabled on the site controller. See Enabling the Ansible Runner Capability for more details.
2. The OS template ubuntu-26-04-lts-waitforssh registered at the target site (see Registering the OS template).
3. A logical network profile at the target site for the cluster network. The extension attaches every server to it untagged, on a single NIC, as the default route, with instance DNS records enabled. Set infrastructure.logicalNetworks[].profileLabel in extension.json to the label of that profile before you create the extension.
4. Internet access from the servers to the Ubuntu archive, download.schedmd.com, and the Apptainer PPA. HGX B200 and B300 nodes also need developer.download.nvidia.com.
5. Server types for the head node with at least 16 GB of RAM and enough local disk for /shared (home directories). Use the external_nfs_export input for bulk data.
6. The extension’s site configuration saved on every target site (see Creating the Slurm application extension). Because the extension declares site-level configuration variables, a deployment fails with “Extension slurm not enabled on site” until an operator opens the extension’s site config on each site and saves it. All variables ship with defaults, so this is a one-click save unless overrides are needed.

Registering the OS template

The extension requires the ubuntu-26-04-lts-waitforssh OS template (Ubuntu/26.04/oob-u26-04-lts-waitforssh in the MetalSoft OS templates repository). It is the stock Ubuntu 26.04 LTS template with two differences:

• The installer reboots into the installed OS instead of powering the server off.
• The template’s ready method is wait_for_ssh instead of wait_for_power_off.

Do not use the stock ubuntu-26-04-lts template

MetalSoft runs the extension’s post-deploy Ansible tasks before it powers the instances back on. With the stock template, which powers the servers off after installation, the servers are still off when the playbook starts and the deployment fails with No route to host. The wait_for_ssh template is generic, and any application extension that connects to its servers over SSH after deployment can use it.

1a. Create the template (with internet access):

metalcloud-cli os-template list-repo
metalcloud-cli os-template create-from-repo Ubuntu/26.04/oob-u26-04-lts-waitforssh

1b. Create the template (air-gapped). On a machine with internet access, clone the repository:

git clone https://github.com/metalsoft-io/os-templates

Copy the os-templates directory, including its .git files, to a machine that runs the CLI and has access to the Global Controller, then run:

metalcloud-cli os-template create-from-directory Ubuntu/26.04/oob-u26-04-lts-waitforssh --dir /path/to/os-templates

2. If the servers cannot reach repo.metalsoft.io, update the template’s ISO URL asset to point at the Ubuntu 26.04 live server ISO hosted on your repository.

Creating the Slurm application extension

1. Clone the repository containing the extension definition:

git clone https://github.com/metalsoft-io/metalsoft-extensions
cd metalsoft-extensions/slurm

The definition references an Ansible bundle and an execution environment image that are already hosted on repo.metalsoft.io, so no repackaging is needed. Adjust infrastructure.logicalNetworks[].profileLabel to match the logical network profile at your sites before you create the extension.

2. Create and activate the extension:

metalcloud-cli extension create "Slurm HPC Cluster" application "Slurm HPC/AI cluster on bare metal" --definition-source extension.json
metalcloud-cli extension activate <id-of-created-extension>

Activated extensions cannot be edited in place. To change the definition, bump extensionVersion and create the extension again. MetalSoft keeps each version as a separate extension.

3. Enable the extension on the site level and save its configuration. After activation, the extension must be activated per site on the site level configuration page. This is also where its configVars, the operator-set, site-level settings, are configured, and it must be done before the first deploy. To save the defaults from the CLI:

echo '[{"label":"nvidia_driver_branch","value":"580"}]' | metalcloud-cli extension site-config set <id-of-created-extension> <site> --config-source pipe

You are now ready to deploy the Slurm application from the Infrastructure Designer.

Site configuration (configVars)

An operator sets these once per site. They apply to every Slurm deployment at that site:

Variable	Type	Default	Description
nvidia_driver_branch	String	580	NVIDIA driver branch: 580 (long-term support branch) or 595.
nvidia_blackwell_hgx_enabled	Boolean	false	Allows HGX B200 and B300 (NVLink 5) nodes. Adds the NVIDIA CUDA apt repository, pinned so that only nvlsm comes from it, plus the InfiniBand management tools. Experimental. When false, such nodes fail the preflight with a clear message.
allow_ubuntu_slurm_source	Boolean	false	Allows tenants to select slurm_source=ubuntu (the Ubuntu archive’s Slurm 25.11.2, which has known unpatched security issues).

Deployment inputs

Each deployment created from the Infrastructure Designer takes the following inputs. Create-only inputs cannot change after the cluster exists: an edit that changes one fails before MetalSoft touches any server.

Input	Type	Default	Create-only	Description
head_server_type	ServerType	—	Yes	Server type for the head node (at least 16 GB of RAM).
compute_server_type	ServerType	—	Yes	Server type for every compute node (CPU or GPU).
compute_count	Integer	2	No	Number of compute nodes (1–128). Changing it on an existing deployment triggers scale-out or scale-in.
os_template	OsTemplate	—	Yes	OS template for all nodes. Must be Ubuntu 26.04 LTS (ubuntu-26-04-lts-waitforssh).
slurm_source	Enum	upstream	Yes	upstream (SchedMD release built on the head node) or ubuntu (requires allow_ubuntu_slurm_source).
slurm_version	Enum	26.05.4	Yes	SchedMD release: 26.05.4 or 25.11.8. Applies to the upstream source only.
cluster_name	String	slurm	Yes	Slurm ClusterName.
partition_name	String	compute	Yes	Name of the default partition. Cannot be default.
admin_username	String	slurmadmin	Yes	Cluster administrator (sudo and Slurm administrator). Cannot be a system user or the OS template user.
admin_ssh_public_key	String	""	No	Additional SSH public key for the administrator. The extension generates a key pair in any case.
install_nvidia_driver	Boolean	true	Yes	Installs the NVIDIA driver on GPU nodes. When false, Slurm uses only the GPUs that a driver already present in the OS image exposes.
install_apptainer	Boolean	true	Yes	Installs Apptainer on all nodes.
external_nfs_export	String	""	Yes	Optional host:/path NFS export mounted at /data on all nodes.
scale_in_drain_timeout_minutes	Integer	60	No	On scale-in, how long running jobs on the removed nodes get to finish before they are requeued or cancelled (0–720).

Outputs

After every successful deployment or edit, the extension publishes the following outputs on the extension instance:

Output	Description
head_node_ip	IP address of the head (login) node
head_node_hostname	Host name of the head node
ssh_command	ssh <admin_username>@<head_node_ip>
admin_user	Cluster administrator user name
admin_password	Initial password of the administrator (later changes are not reflected)
admin_ssh_private_key	Generated ed25519 private key of the administrator
slurm_cluster_name	Slurm cluster name
slurm_installed_version	Installed Slurm version
default_partition	Default partition name
compute_node_count	Number of compute nodes registered in Slurm
compute_nodes	Slurm names of the compute nodes
gpu_total	Total number of GPUs across the compute nodes
data_mount	External NFS export mounted at /data, or empty

Security note

admin_password and admin_ssh_private_key are declared as password outputs. Anyone who can read the extension instance credentials can log in to the cluster as its administrator. After handover, change the password with passwd and replace the key in ~/.ssh/authorized_keys (keep the managed block markers, and add your own keys outside the block).

Deployment flow

On create, the extension runs one playbook after MetalSoft installs the OS on all servers:

1. Preflight checks, then a reachability and privilege-escalation probe on every node.
2. Node baseline: full OS upgrade, NVIDIA driver on GPU servers, and one reboot.
3. Head node services: MUNGE, NFS export, Slurm build, MySQL, slurmdbd, and the administrator account.
4. Compute nodes: Slurm packages, MUNGE key, /shared mount, and user synchronization.
5. slurm.conf, then slurmctld and slurmd.
6. Validation: srun on every node, a batch job, an MPI job over PMIx, a GPU job when GPUs are present, and sacct.
7. Outputs.

A first deployment typically takes 30–60 minutes after the OS installation. The Slurm build on the head node takes 10–30 minutes, and GPU servers add a reboot with long POST times. Sites with slow internet access take longer because of the full OS upgrade.

Scaling

Edit the deployment’s compute_count.

• Scale-out: new servers get the full node baseline (upgrade, driver, and one reboot), join slurm.conf through scontrol reconfigure without daemon restarts, and are validated inside a short maintenance reservation so that queued jobs cannot start on them before the check passes. Existing nodes are reconciled only, never rebooted or upgraded.
• Scale-in: MetalSoft chooses which servers leave. Before the infrastructure change, the extension drains them, gives running jobs scale_in_drain_timeout_minutes to finish, then requeues batch jobs that allow it and cancels the rest. It removes the nodes from slurm.conf before MetalSoft releases the servers. A node in a tenant reservation blocks the scale-in.
• The head node cannot be scaled, and at least one compute node must remain.

GPU support

GPU class	Status
PCIe GPUs (L40S, H100 and H200 PCIe or NVL, RTX PRO 6000, A100 PCIe, and others)	Supported
HGX A100, H100, and H200 (NVSwitch)	Supported, with Fabric Manager started before slurmd
HGX B200 and B300 (NVLink 5)	Experimental, requires nvidia_blackwell_hgx_enabled
GB200 and GB300 (arm64), pre-Turing GPUs (V100, P100)	Not supported (the preflight fails)

The extension detects GPUs automatically and registers them in Slurm with a type derived from the model name, for example srun --gpus=2 ... or srun --gres=gpu:h100:2 .... MIG and per-job GPU utilization metrics are not available.

Using the cluster

Log in to the head node with the ssh_command output and the administrator credentials, then:

sinfo                                         # nodes and partitions
srun -N2 hostname                             # run a command on two compute nodes
sbatch job.sh && squeue -u $USER              # queue a batch job
srun --gpus=1 nvidia-smi                      # GPU job
sacct                                         # job history

Manage cluster users on the head node with sudo slurm-useradd [--account A] [--ssh-key "..."] <name> and sudo slurm-userdel [--cancel-jobs] [--remove-home] <name>. Compute nodes pick up new users within a minute. Regular users log in to the head node only. Compute nodes refuse their SSH logins, and srun --jobid=<id> --overlap --pty bash opens a shell inside a running job.

Add partitions, limits, and other site settings to /etc/slurm/local.conf on the head node, then run sudo scontrol reconfigure. The extension never overwrites that file.

Execution environment

The extension’s Ansible playbooks run on the Site Controller inside an execution environment (ee-slurm), a container image holding all the dependencies the playbooks need. The extension ships its own image, declared as an OciImage asset in the extension definition and selected by each Ansible task through the ee option, so no Site Controller configuration is required. The image is based on UBI 10 and requires an x86-64-v3 (AVX2) CPU on the Site Controller. To customize it, rebuild the image from the execution-environment.yml recipe in the extension repository, and update the OciImage asset in the definition with a new tag.

Troubleshooting

Symptom	Cause and fix
The deployment fails with “Extension slurm not enabled on site”	Save the extension site config on that site.
The playbook fails with No route to host and the servers are powered off	The deployment uses an OS template that powers the servers off after installation. Use ubuntu-26-04-lts-waitforssh.
The preflight fails on slurm_source=ubuntu	The site has not enabled allow_ubuntu_slurm_source. Use the upstream source or enable the setting.
A node is INVAL or drained with Low RealMemory or a gres count mismatch	The hardware differs from the registered node definition, for example a missing GPU. Fix the node, then run sudo scontrol update nodename=<node> state=resume.
”These inputs cannot change after the cluster was created”	Revert the create-only input in the edit.
The scale-in fails with “node(s) in a reservation”	Delete or shrink the reservation, then retry the edit.
sinfo reports “Unable to contact slurm controller”	slurmctld is down on the head node. Check sudo systemctl status slurmctld and journalctl -u slurmctld.

For any other failure, an edit that changes nothing re-runs the full convergence and validation. Logs are in journalctl -u slurmctld, slurmd, slurmdbd, and munge, in /var/log/slurm/, and the Slurm build log in /var/lib/slurm-ext/build/build-<version>.log on the head node.