Upgrading MetalSoft in air-gapped environments

In an air-gapped environment the Global Controller and the Site Controller cannot pull new images from the MetalSoft registry. MetalSoft supplies offline upgrade packages instead, which you copy to the controllers manually:

• gcimages.tar.gz - the Global Controller images
• manifests.tar.gz - the Global Controller Kubernetes manifests
• scimages.tar.gz - the Site Controller images

The Global Controller steps below are written for the OVA deployment.

Note

For a Kubernetes deployment, load the images into the registry the cluster pulls from, or into the container runtime of every node that runs MetalSoft pods (with containerd: ctr -n k8s.io image import). The manifests are in /opt/metalsoft/manifests/<namespace> instead of /root/manifests.

Upgrading the Global Controller

1. Import the images

Copy gcimages.tar.gz and manifests.tar.gz to the Global Controller, then import the images:

gunzip gcimages.tar.gz
ctr image import gcimages.tar

2. Update the manifests

Back up the current manifests in /root/manifests, then extract the new ones over them.

cp -a /root/manifests /root/manifests.bak-$(date +%Y%m%d)
cd /root/manifests
tar -xvzf /path/to/manifests.tar.gz -C /root/manifests

3. Run the upgrade script

From the manifests folder, run upgrade.sh from the scripts folder. To see the available options:

./scripts/upgrade.sh -h

Example:

./scripts/upgrade.sh -n mynamespace

Before applying the new manifests, the script backs up the MySQL databases (except the system schemas and mysql_audit_raw), one mysqldatabase_<db>_<namespace>.sql.gz file per database, to:

/var/backups/metalsoft/<namespace>/<date>-pre_upgrade_mysqldump/

Use -b <folder> to change the base folder (/var/backups/metalsoft), or -S to skip the backup.

4. Monitor the upgrade

Wait until all pods are Running:

kubectl -n mynamespace get pods

Alternative: OVA helper scripts

Recent OVA versions also ship helper scripts that import the images and retag the manifests. Transfer the provided tar.gz files onto the Global Controller VM, put them into /root/k8s-images, and use the helper script there to import the images:

In recent versions of OVA, you can use:

./prepare-offline license.tar.gz

If that fails, use the following:

gunzip latest.tar.gz
./prepare-offline latest.tar

This will import the latest.tar (which might hold 1 or more images for import).

Alternatively (manual process), import the images using the following command for each tar file:

ctr image import latest.tar

Once the images are imported, update the image tags in the manifests:

cd /root/manifests
metalsoft-update-k8s-tag <newtag>

So, if you are upgrading to v7.3.0, you would run:

metalsoft-update-k8s-tag v7.3.0

This command, available in recent versions of OVA, automatically detects the current version from the manifests and updates the files to the new version.

Then apply the manifests:

kubectl apply -f .

Upgrading the Site Controller

1. Load the images

Copy scimages.tar.gz to the Site Controller and load it:

docker load -i scimages.tar.gz

Older versions of docker cannot load a .tar.gz directly. In that case, use:

gunzip scimages.tar.gz
docker load -i scimages.tar

2. Update the image tags and restart the agents

cd /opt/metalsoft/agents
# edit .env or docker-compose.yaml and update the tags of the images to the new version
# then:
docker compose up -d

3. Verify

# check running docker images:
docker ps
# check the logs of ms-agent:
docker logs -f ms-agent --tail 50

Pulling selected images yourself

If you need to upgrade only specific images, you can pull them from a machine with internet access and transfer them manually. Use the registry credentials provided by MetalSoft.

Global Controller

Pull the image (ctr prompts for the password):

ctr images pull --platform linux/amd64 -u REGISTRY_USER registry.metalsoft.dev/gc/license:v7.0

Save the locally pulled image to a .tar file:

ctr image export --platform linux/amd64 --skip-manifest-json /root/k8s-images/license.tar registry.metalsoft.dev/gc/license:v7.0

Transfer the files onto the Global Controller, preferably into /root/k8s-images, and import each tar file:

ctr image import license.tar

Site Controller

Log into the MetalSoft registry, providing the username and password supplied by MetalSoft:

docker login registry.metalsoft.dev

Pull the image:

docker pull registry.metalsoft.dev/sc/ms-agent:v7.0

Save the locally pulled image to a .tar file:

docker save -o ms-agent.tar registry.metalsoft.dev/sc/ms-agent:v7.0

Transfer the file onto the Site Controller and run the following from the folder you copied the tar to:

docker load -i ms-agent.tar