Structuring Ansible bundles

Since 7.0

This page describes how to structure the Ansible bundle of an extension and the runtime contract: the inventory and variables MetalSoft generates for your playbooks, how to handle scale-out/scale-in, and how to return outputs.

Most extension failures come from guessing this contract instead of following it — group names, variable names and file locations are all fixed by the platform.

Bundle layout

An Ansible bundle is a zip archive with the playbooks at the archive root:

my-bundle.zip
├── deploy.yaml
├── scale.yaml
├── ansible.cfg
└── roles/
    └── my_role/
        ├── defaults/main.yaml
        └── tasks/main.yaml

cd <bundle-dir> && zip -r ../my-extension-v1.0.0.zip . -x '*.DS_Store'
unzip -l ../my-extension-v1.0.0.zip   # deploy.yaml and roles/ must be at the top level

Rules:

The task’s playbook option is a bare filename resolved at the bundle root. Playbooks nested inside a subdirectory are not found.
Never ship a file named job.yml — the runner renames the requested playbook to job.yml, so the name is reserved.
Host the zip at the asset url (≤128 chars), reachable by the global controller.
After changing the bundle, verify the change actually landed in the uploaded zip (unzip -p my-bundle.zip roles/x/tasks/y.yml | grep ...). A stale artifact makes a fix appear to “not take”. An early debug task echoing your bundle version makes it obvious which bundle actually ran.

What MetalSoft generates at runtime

For each Ansible task, the site controller runs ansible-runner in an ephemeral container with the standard ansible-runner layout, under /opt/metalsoft/ansible-jobs/<task_uuid>/ on the site controller:

<task_uuid>/
├── project/                  # your bundle, unzipped
│   └── job.yml               # the requested playbook, renamed
├── inventory/
│   └── inventory.yaml        # generated inventory (see below)
├── env/
│   └── extravars             # generated variables, one merged JSON (see below)
└── artifacts/                # runner output; write context.json here to return data

The extra-vars are passed to the playbook automatically — reference them directly (e.g. {{ extensionInstanceVariables.my_input }}); there is no need to include_vars anything.

The generated inventory

For application extensions, every instance array becomes its own inventory group, named by its label verbatim (prefer underscores in labels for Ansible ergonomics) — an extension declaring several instance arrays gets several groups. On edit deployments, two additional groups per array describe the delta:

all:
  children:
    compute:                        # main group = the first instance array's label, verbatim
      hosts:
        instance-4411:
          ansible_host: 10.0.2.11
          ansible_port: 22
          ansible_user: root
          ansible_password: "..."
        instance-4412:
          ansible_host: 10.0.2.12
          ansible_port: 22
          ansible_user: root
          ansible_password: "..."
    group2:                         # a second instance array (label "group2") = a second group
      hosts:
        instance-4420:
          ansible_host: 10.0.2.20
          ansible_port: 22
          ansible_user: root
          ansible_password: "..."
    compute_scale_out:              # edit deploys: hosts being ADDED to the "compute" array
      hosts:
        instance-4413:
          ansible_host: 10.0.2.13
    compute_scale_in:               # edit deploys: hosts being REMOVED from the "compute" array
      hosts:
        instance-4410:
          ansible_host: 10.0.2.10

Each instance array gets its own <label>_scale_out / <label>_scale_in groups, but empty scale groups are omitted from the generated inventory — in the example above, group2 has no pending changes, so group2_scale_out / group2_scale_in do not appear. Plays targeting a scale group must therefore tolerate the group being absent (Ansible treats a play whose hosts: group does not exist as a no-op with a warning).

Host names are the instances’ permanent subdomains.
ansible_host is the instance’s IPv4 address on the logical network that has providesDefaultRoute: true.
ansible_port comes from the OS template’s SSH port; ansible_user/ansible_password are the OS template credentials / per-instance initial password.
A play targeting a group name that does not exactly match an instance-array label silently runs on zero hosts.

The generated variables (extra-vars)

One merged JSON, containing at least:

extensionInstanceVariables:       # the extension inputs, keyed by input LABEL
  compute_nodes: 3                # isPassword values arrive decrypted
  pull_secret: "..."
compute:                          # one key per instance array (label, with - replaced by _):
  my_custom_var: value            # that array's customVariables
extensionInstanceRecordSet:       # deployment record set — networks, IPs, DNS (see below)
  ...
configVars:                       # operator-set site configuration (see extension.json reference)
  DNSResolvers: "10.0.0.53,10.0.0.54"
context:                          # data returned by earlier tasks of THIS deployment (see below)
  ...

The merged extra-vars is not limited to these keys — when developing, inspect the actual env/extravars on the site controller rather than assuming. Config values may arrive as comma-separated strings; normalize defensively:

dns_servers: >-
  {{ (configVars.DNSResolvers | default('')).split(',') | map('trim')
     | reject('equalto', '') | list or ['1.1.1.1', '8.8.8.8'] }}

extensionInstanceRecordSet

The record set describes what was actually provisioned: instance groups, logical networks, subnets, IP allocations (with the tags you declared and the resulting FQDNs), IP ranges and per-instance details. Top-level fields:

deploymentId, extensionInstanceId, extensionInstanceName, repositoryBaseUrl,
siteLabel, dnsResolverIp, dnsResolvers[], ntpServers[], baseDomain,
clusterIpv4Ips[], clusterIpv4IpRanges[], serverInstances[], serverInstanceGroups[]

serverInstanceGroups[]: {serverInstanceGroupId, serverInstanceGroupLabel, tags[], logicalNetworks[]} where each logical network carries {id, name, label, vlan_id, subnets: [{subnet, netmask, gateway, mtu, ipType, ipAllocations[], ipRanges[]}]}.
ipAllocations[] entries: {ip, cidr, gateway, netmask, maskBits, networkAddress, fqdn, tags} — tags are exactly what you declared in extension.json, and fqdn is filled from your declared dnsRecords.
serverInstances[]: {serverInstanceId, serverInstanceGroupId, host, logicalNetworks: [{name, label, vlan_id, uplinks: [MACs]}], ipv4Ips[], serverInterfaces[], tags}.

Canonical lookup pattern — find an allocated VIP by the role tag you declared (a good place for these is a globals role’s defaults/main.yaml):

my_group: >-
  {{ extensionInstanceRecordSet.serverInstanceGroups | default([])
     | selectattr('serverInstanceGroupLabel', 'equalto', 'compute') | first | default({}) }}
my_network: >-
  {{ my_group.logicalNetworks | default([]) | selectattr('name', 'equalto', 'cluster-network')
     | first | default({}) }}
api_vip: >-
  {{ my_network | json_query('subnets[].ipAllocations[]') | default([], true)
     | selectattr('tags.role', 'equalto', 'api-vip') | first | default({}) }}
# api_vip.ip and api_vip.fqdn are now usable

Guidance:

Build FQDNs from extensionInstanceRecordSet.baseDomain — it is the site default zone that {{default_zone_name}} resolves to in your declared dnsRecords. Never mirror it as an extension input: the platform ignores such an input while your playbook would obey it, silently desyncing your URLs from the DNS records that actually exist. The allocation fqdn fields are the ground truth to cross-check against.
dnsResolvers[] / dnsResolverIp may be public defaults (e.g. 1.1.1.1) rather than the site’s authoritative internal resolver. Prefer an explicit site configVars value and treat the record-set resolvers as a fallback.

Scale-out and scale-in

Scaling an application extension is an edit: the user changes an instance-count input, the platform adjusts the resources, and the onEdit stages run. The generated inventory describes the delta via the _scale_out / _scale_in groups.

Key facts:

Scale-in hosts are excluded from the main group. The main group is already the surviving set — do not compute groups['compute'] | difference(groups['compute_scale_in']); it is a no-op.
Stage timing on edit:
- at preDeploy, scale-in hosts are still provisioned and reachable — drain/remove them from the cluster here; scale-out hosts may already appear in the inventory but are not yet reachable.
- at postDeploy, scale-out hosts are provisioned and reachable — join/configure them here; scale-in hosts are gone.
The established pattern is to register the same scale playbook on both stages and target the _scale_in / _scale_out groups directly — each play is a no-op when its group is empty:

"onEdit": [
    {
        "stage": "preDeploy",
        "tasks": [
            {
                "label": "scaleIn",
                "taskType": "ExtensionTaskAnsible",
                "options": { "asset": "my-bundle", "playbook": "scale.yaml" }
            }
        ]
    },
    {
        "stage": "postDeploy",
        "tasks": [
            {
                "label": "scaleOut",
                "taskType": "ExtensionTaskAnsible",
                "options": { "asset": "my-bundle", "playbook": "scale.yaml" }
            }
        ]
    }
]

- name: Drain and remove nodes leaving the cluster
  hosts: compute_scale_in
  gather_facts: false
  tasks:
    - name: Drain node
      ...

- name: Join new nodes to the cluster
  hosts: compute_scale_out
  gather_facts: false
  tasks:
    - name: Join node
      ...

A complete working example is the Red Hat OpenShift extension (scale.yaml + onEdit in its extension.json).

Outputs and cross-deployment persistence

Fresh runner per task

Every lifecycle task runs in a fresh runner volume. Files a previous task wrote (install dirs, tool state, downloaded artifacts) are gone by the next task — even within the same deployment. Data is handed off explicitly, via context.json.

Returning data: context.json

To return data from a playbook, write a JSON file to artifacts/<ident>/context.json inside the runner directory — the run identifier is available as the ANSIBLE_RUNNER_IDENT environment variable:

- name: Return outputs to the platform
  ansible.builtin.copy:
    dest: "/runner/artifacts/{{ lookup('env', 'ANSIBLE_RUNNER_IDENT') }}/context.json"
    content: "{{ {'cluster_kubeconfig': kubeconfig_content, 'console_url': console_url} | to_nice_json }}"
    mode: "0644"

The artifacts copy is consumed and then cleaned up by the platform right after the task completes — keep a debug copy elsewhere if you need to inspect it later.

What happens to the keys:

Within the same deployment: later tasks of the same deployment receive earlier tasks’ keys via the context extra-var. On the next deployment (onEdit, onDelete), context starts empty again.
A key matching a declared input label is stored durably with the extension instance (encrypted if the input has isPassword: true) and re-injected into extensionInstanceVariables.<label> on every later deployment.
A key matching a declared output label (since 7.4) is stored as the instance’s output value, visible to the user, and also re-injected into extensionInstanceVariables on later deployments.
Keys matching neither are dropped after the deployment’s job chain ends.

Outputs are replaced on every successful run

The stored outputs are replaced, not merged, on every successful task run — including runs that write no context.json at all (which reset the outputs to empty). Practical consequences:

Every playbook that a lifecycle stage can run (including a scale play that “has nothing to output”) must end by re-emitting the full outputs payload, guarded to skip when the run already wrote a context.json or when the values are empty.
A write that omits previously set output keys wipes those keys.

Recommended pattern for playbook-generated values

For values generated at deploy time that later deployments need (kubeconfig, admin URLs, credentials): declare the label as an output (a label cannot be declared as both an input and an output — inputs and outputs share one label namespace) and emit the key from context.json. The stored output value is re-injected into extensionInstanceVariables.<label> on later deployments. Before the first successful write the value is absent or empty — guard reads with | default(x, true) fallbacks and assert required keys with a clear “deploy must run first” message.

Special context.json keys

mount-and-boot (with iso.path) and eject-mounted-iso trigger platform jobs that mount/boot or eject an ISO. They target all server instances of the extension instance. Emit them from an onEdit scale task and will only aply to the new nodes. Example

- name: Set context JSON payload
  ansible.builtin.set_fact:
    _context_payload:
      mount-and-boot:
        execution: parallel
        iso:
          path: "{{ iso_final_path }}"
        options:
          sendSuccessMessageAfterBoot: true
        application: openshift

- name: Write context JSON artifact
  ansible.builtin.copy:
    content: "{{ _context_content | to_nice_json }}"
    dest: "{{ context_json_file_path }}"
    mode: "0644"

Example eject-mounted-iso

- name: Set context JSON payload
  ansible.builtin.set_fact:
    _context_payload:
      eject-mounted-iso: {}

Secrets

BMC and OS credentials are also available out-of-band from the runner via a local HTTP endpoint:

- name: Fetch the instance password
  ansible.builtin.uri:
    url: "http://localhost/ansible/secret?name=password&folder={{ playbook_dir | dirname | basename }}"
    method: GET
    return_content: yes
  register: password_response
  no_log: true

See Ansible tasks — Accessing secrets for the list of available secrets per context.

Execution timeouts

Size each task’s executionTimeout to the playbook’s worst-case duration (fresh-deploy timings, not re-run timings — the first boot-to-API wait of a product can legitimately take 20–30+ minutes). If the timeout is omitted, the platform default (1 hour) applies: the task is reported failed/timed-out while the playbook keeps running, its context.json is never collected (outputs lost, follow-up jobs never fire) and the execution folder is cleaned up.

Testing the bundle offline

Render templates and validate your variable handling against fixtures before any live deploy:

tests/fixtures/sample-variables.json — realistic extra-vars: extensionInstanceVariables plus an extensionInstanceRecordSet with your groups/networks/allocations and role tags.
tests/inventory.yaml — a static inventory with your real group names and host vars (groups/hostvars are magic variables and cannot be faked via play vars).
tests/render-spec.yaml — a hosts: localhost play: include_vars your role defaults → include_tasks your fact-building → render templates → assert the expected results.

cd tests && ansible-playbook -i inventory.yaml render-spec.yaml
ansible-playbook --syntax-check deploy.yaml scale.yaml

The vmware-cloud-foundation9 extension ships a complete tests/ harness to copy.

Troubleshooting

Play matches no hosts — the hosts: group name does not match the instance-array label verbatim.
Playbook not found — the playbook is nested inside a directory in the zip instead of the archive root, or the playbook option doesn’t match the filename.
A variable is undefined — inspect the actual env/extravars and inventory/inventory.yaml under /opt/metalsoft/ansible-jobs/<task_uuid>/ on the site controller instead of guessing names.
A fix “doesn’t take” — the uploaded zip is stale, or a task retry did not re-download the bundle; verify the content of the extracted project/ directory.