Enabling metric monitoring

Available since: version 7.5

Administrators that wish to monitor the health and performance of the system can leverage the various metrics that MetalSoft optionally exposes.

MetalSoft services expose Prometheus metrics on a /metrics endpoint, but that endpoint does not start unless metrics are explicitly enabled. A single environment variable, ENABLE_METRICS, controls whether the endpoint runs, and is opt-in on every service.

The ENABLE_METRICS switch

Accepted values are identical on the Go and NestJS stacks, since a single deployment manifest typically sets the variable for both: 1, t, or true, in any case, enable the endpoint. Any other value — including a typo, an empty string, or an unset variable — disables it. Defaulting to on would expose a listener nobody asked for, so anything that is not a recognized truthy value reads as off.

Note

On the NestJS stack, enabling metrics also activates the prom-client default collectors: an event-loop-delay histogram and a garbage-collection observer that run continuously once active. These have a real, ongoing cost, unlike Go’s collectors, which are pull-only and cost nothing while unscraped. Leave ENABLE_METRICS unset on any NestJS deployment that nothing scrapes.

Enabling metrics on platform services

Use this procedure for any of the 10 NestJS services or 7 Go services.

1. Update the ConfigMap

Edit configmaps.yaml and add ENABLE_METRICS to the ConfigMap that supplies environment variables to the target service:

  ENABLE_METRICS: "1"

2. Apply the ConfigMap

kubectl apply -f configmaps.yaml -n demo-metalsoft

3. Restart the affected deployments

The gate is evaluated once, at process startup, so the change takes effect only after a restart.

cd manifests/
grep -l "ENABLE_METRICS" *-deployment.yaml 2>/dev/null | sed 's/-deployment\.yaml//g' | while read z
do kubectl -n demo-metalsoft rollout restart deployment $z
done

Enabling metrics on the site controller agent

The site controller agent has its own metrics toggle in the install one-liner, alongside the existing capability toggles for file transfer, VNC, syslog, and other agent capabilities. See Deploying the Site Controller for the full install procedure.

To enable metrics on an agent:

1. On the Global Controller, click Sites, click the site name, and click Site controller configuration.
2. Select the Metrics capability along with any other capabilities the installation requires.
3. Click Generate command and run the resulting one-liner on the site controller.

Selecting the option appends ENABLE_METRICS=1 to the generated command. Leaving it unchecked means the agent exposes nothing, which matches how every other agent capability already works.

Metrics on customer premises are a deliberate, per-install choice rather than a default: an agent already deployed keeps running exactly as it does today, and re-running an install without selecting the option turns metrics off.

Metric reference

The following tables are read off real scrapes of both stacks, not from source: 36 metric families on NestJS, 43 on Go.

Application metrics

These metrics are written by MetalSoft and are identical in both stacks: same names, types, labels, and histogram bucket boundaries. Build dashboards and alerts on these.

Metric	Type	Labels	Description
build_info	gauge	version	Always 1; the running version is the label, so it can be joined onto any other series to attribute a change to a deploy.
http_request_duration_seconds	histogram	method, route, status	Request latency, plus request and error rate through _count. route is the matched pattern, never the raw URI.
kafka_handler_duration_seconds	histogram	pattern, outcome	How long a message handler ran and whether it threw. A rising outcome="error" indicates a failing consumer, not a broker problem.
kafka_message_lag_seconds	histogram	topic	How old a message was when the consumer reached it — the primary consumer-health signal. Alert on this.
kafka_consumer_offset_lag	gauge	topic, partition	Messages still behind the partition high-water mark — the size of the backlog, where the histogram above reports how stale it is.

Bucket boundaries, in seconds:

Metric	Buckets (seconds)
http_request_duration_seconds	0.005, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10
kafka_handler_duration_seconds	0.01, 0.05, 0.1, 0.5, 1, 5, 15, 60
kafka_message_lag_seconds	0.1, 0.5, 1, 5, 15, 60, 300, 900 (out to 15 minutes)

Latency is always a histogram, never a summed counter: a count plus a total yields a mean and nothing else, and the tail is where incidents live. Every series also carries an app=<serviceName> label.

Process metrics

Seven metrics are common to both stacks and can go on one shared panel. Five exist on only one side, so a panel using those needs a per-stack variant.

Metric	Stack	Description
process_cpu_seconds_total	Both	Total CPU seconds burned. Rate it to get cores in use.
process_resident_memory_bytes	Both	RSS, the real physical memory held — the number an OOM kill is decided on.
process_virtual_memory_bytes	Both	Address space reserved. Routinely huge and not usually a problem on its own.
process_open_fds	Both	Open file descriptors. Climbing steadily indicates a socket or file leak.
process_max_fds	Both	The descriptor ceiling. Alert on the ratio of open to max, never the raw count.
process_start_time_seconds	Both	Unix time the process started. Gives uptime and detects restarts.
process_cpu_user_seconds_total	NestJS	CPU spent in application code.
process_cpu_system_seconds_total	NestJS	CPU spent in kernel calls. High relative to user time points at I/O churn.
process_heap_bytes	NestJS	Process heap as the OS sees it, distinct from the V8 heap below.
process_virtual_memory_max_bytes	Go	Address-space limit from RLIMIT_AS.
process_network_receive_bytes_total	Go	Bytes read from the network by this process.
process_network_transmit_bytes_total	Go	Bytes written to the network by this process.

NestJS runtime metrics

These come from the prom-client default collectors. Event-loop lag is the one to watch: Node is single-threaded, so rising lag means requests are queueing behind synchronous work no matter how healthy CPU and memory look.

Metric	Description
nodejs_eventloop_lag_seconds	Current delay between a timer being due and running — the headline saturation signal for a Node service.
nodejs_eventloop_lag_p50_seconds	Median lag over the collection interval.
nodejs_eventloop_lag_p90_seconds	90th percentile lag.
nodejs_eventloop_lag_p99_seconds	99th percentile lag — exposes intermittent blocking the mean hides.
nodejs_eventloop_lag_min_seconds	Best case over the interval.
nodejs_eventloop_lag_max_seconds	Worst case over the interval.
nodejs_eventloop_lag_mean_seconds	Average over the interval.
nodejs_eventloop_lag_stddev_seconds	Spread. A large value means blocking is bursty rather than constant.
nodejs_heap_size_total_bytes	V8 heap reserved.
nodejs_heap_size_used_bytes	V8 heap in use. Growing monotonically across GCs is the classic leak shape.
nodejs_heap_space_size_total_bytes	Per V8 space (new, old, code, large object), labeled space.
nodejs_heap_space_size_used_bytes	The same, in use. Pinpoints which generation is growing.
nodejs_heap_space_size_available_bytes	The same, still available.
nodejs_external_memory_bytes	Memory held by C++ objects bound to JS, Buffers above all — invisible in the heap numbers.
nodejs_gc_duration_seconds	Histogram of GC pause durations. Long pauses appear as latency the handler never spent.
nodejs_active_handles	Open libuv handles by type — sockets, servers, timers (label: type).
nodejs_active_handles_total	The same, totaled. Rising with steady traffic is a leak.
nodejs_active_requests	In-flight async requests, such as filesystem and DNS calls.
nodejs_active_requests_total	The same, totaled.
nodejs_active_resources	Everything keeping the event loop alive, per async_hooks (label: type).
nodejs_active_resources_total	The same, totaled. Useful when a process will not exit.
nodejs_version_info	Always 1; Node version in the labels (version, major, minor, patch).

Go runtime metrics

These come from the client_golang collectors. Goroutine count and GC pause are the two worth alerting on. Most go_memstats_* metrics only earn their place during an investigation.

Metric	Description
go_goroutines	Live goroutines. Unbounded growth is the standard Go leak, usually a blocked channel or an unreleased request.
go_threads	OS threads created. Climbing means goroutines are blocking in syscalls.
go_gc_duration_seconds	GC pause durations as a summary with pre-computed quantiles, so these cannot be aggregated across instances.
go_memstats_last_gc_time_seconds	When GC last ran. A stale value under load means GC is not keeping up.
go_memstats_next_gc_bytes	Heap size that will trigger the next collection.
go_memstats_heap_alloc_bytes	Heap bytes allocated and still reachable — the everyday “how much memory” number.
go_memstats_alloc_bytes	Same value, legacy name.
go_memstats_alloc_bytes_total	Cumulative bytes ever allocated. Rate it to get allocation pressure, the real driver of GC cost.
go_memstats_mallocs_total	Cumulative objects allocated.
go_memstats_frees_total	Cumulative objects freed. The gap between these two rates is net object growth.
go_memstats_heap_objects	Live object count.
go_memstats_heap_inuse_bytes	Heap bytes in active spans.
go_memstats_heap_idle_bytes	Heap bytes held but unused. Staying high while RSS stays high means memory is retained, not leaked.
go_memstats_heap_released_bytes	Heap bytes handed back to the OS.
go_memstats_heap_sys_bytes	Heap bytes obtained from the OS.
go_memstats_sys_bytes	Total bytes obtained from the OS — the closest runtime analogue to RSS.
go_memstats_stack_inuse_bytes	Goroutine stack memory in use. Tracks goroutine count closely.
go_memstats_stack_sys_bytes	Stack memory obtained from the OS.
go_memstats_mspan_inuse_bytes	Allocator span bookkeeping in use.
go_memstats_mspan_sys_bytes	Allocator span bookkeeping from the OS.
go_memstats_mcache_inuse_bytes	Per-P allocator cache in use.
go_memstats_mcache_sys_bytes	Per-P allocator cache from the OS.
go_memstats_gc_sys_bytes	Memory spent on GC metadata.
go_memstats_buck_hash_sys_bytes	Memory spent on profiling hash tables.
go_memstats_other_sys_bytes	Everything else the runtime took from the OS.
go_gc_gogc_percent	Current GOGC. Confirms tuning actually reached the process.
go_gc_gomemlimit_bytes	Current GOMEMLIMIT soft ceiling. Should reflect the container memory limit.
go_sched_gomaxprocs_threads	GOMAXPROCS. Worth checking against the container CPU limit, since the default sees host cores and not the cgroup.
go_info	Always 1; Go version in the label.

Labels that mean different things per stack

route is the matched route pattern. Express hands it over for free. httprouter v1.3.0 does not, and it cannot be recovered after matching, because reversing matched params into a pattern is ambiguous: path /a/a matches both /a/:x and /:x/a equally plausibly. The Go stack therefore captures the route at registration time, via metrics.NewRouter(). Anything served outside that wrapper is labeled route="unmatched".

pattern is the Nest handler method name on NestJS, and the Kafka topic on Go, because Go dispatches handlers by topic and has no method name to report. Both answer “which message stream is slow,” but a panel grouping on pattern shows method names for one stack and topics for the other.

Alerting on consumer lag

Both stacks record lag only while messages are being handled, so a fully stalled consumer emits no samples at all and looks identical to an idle one. Alert on the sample rate falling to zero, not on the lag value rising — a rising value means messages are still flowing, which is the less severe case.