Server Integrity Management Guide (Drift Detection & Health Monitoring)

Available since 7.5

Server Drift Detection and Health Monitoring are critical features designed to ensure that server configurations and operational status remain consistent with defined baselines and best practices. This guide outlines the necessary prerequisites, configuration steps, and operational procedures for managing your server integrity.

Prerequisites Checklist

Before enabling either Server Drift Detection or advanced monitoring features, the following components must be in place and configured:

1. Git Integration:
   • Server functionality requires Git to be active on the environment. MetalSoft is installed with a local Git repository by default.
2. Firmware Catalog & Baseline (Required for Drift Detection):
   • A valid Firmware Catalog must exist, containing all supported firmware versions for the servers being monitored.
   • A specific Baseline (snapshot) must be defined against which drift can be measured.

Setup Guides: For detailed instructions on setting up prerequisites, please consult the official documentation:

• Manage Server Firmware
• Managing Firmware Catalogs
• Managing Firmware Baselines
• Manufacturer Specific Firmware Catalogs

Server Drift Detection and Firmware Backup

Server Drift Detection monitors changes to server configurations compared to known baselines (snapshots).

A. Enabling Drift Detection (Configuration)

Drift detection must be enabled at two levels: the Site level and the individual Server level.

Note - Server Drift detection and Servr Firmware Backup are enabled by default

1. Site-Level Configuration:

• Navigate to Sites $\rightarrow$ [Site Name] $\rightarrow$ Configuration.
• Ensure both of these options are checked/enabled:
  • Enable Server Drift Detection
  • Enable Server Firmware Backup

Note - You can enable firmware backup without drift detection, but you must enable both drift detection and firmware backup if you require drift detection

2. Server-Level Configuration:

• Navigate to Servers $\rightarrow$ [Server Name] $\rightarrow$ Overview.
• Under the server settings, ensure both toggles are enabled:
  • Drift detection enabled
  • Firmware backup enabled

Critical Warning: Server Drift Detection is dependent on multiple services. If any of the three components (Site-level Drift Detection, Site-level Backup, and Server-level enablement) are disabled, the feature will not function. Note - You can enable firmware backup without drift detection, but you must enable both drift detection and firmware backup if you require drift detection

B. Monitoring Drift Status

To view the current status and history of server changes:

1. Navigate to Servers $\rightarrow$ [Server Name] $\rightarrow$ Drift Detection.
2. Snapshots: The “Snapshots” section displays the current set of known good configurations (the baseline).
3. Drift History: Any detected inconsistencies will appear here. When drift is found, you have options to address it:
   • View Configuration Drift: Review what changed.
   • Acknowledge: Accept the detected change as expected (if appropriate).
   • Sync Target Snapshot with Latest Snapshot: Revert the server’s configuration to match the defined baseline.

C. Advanced Firmware Integrity Monitoring (Enhanced Checks)

MetalSoft offers advanced, specialized checks to ensure firmware integrity beyond standard snapshot comparisons:

1. Git-Based External Change Detection:

   • This mechanism actively checks if firmware has been modified outside of the MetalSoft platform.
   • It utilizes the local Git repository to compare the server’s current firmware state against the last established baseline snapshot, providing granular detection similar to Switch Drift Detection functionality.
   • Alerting: Any detected external drift triggers a per-server alert email sent to the list defined in the global Monitoring tab within the configuration settings.

2. Minimum Version Compliance Check:

   • This dedicated check verifies that installed firmware meets minimum version requirements defined for each component within the baseline, independent of Git snapshots.
   • It compares the current installed firmware (read live from our database) against the minimumVersion specified in the baseline definition.
   • Alerting: The system alerts once per baseline check run to a dedicated subscribed email list (distinct from the external change list). This notification details all servers that fail to comply with the defined minimum version requirement for that specific component.

---

Server Health Monitoring and Allocation Policy

Server Health Monitoring actively checks a server’s operational status, ensuring immediate visibility into potential hardware or software issues.

A. Enabling Health Monitoring

1. Site Level: Navigate to Sites $\rightarrow$ [Site Name] $\rightarrow$ Configuration.
   • Check the box for Enable Server Health Monitoring.
2. Server Level: Navigate to Servers $\rightarrow$ [Server Name].
   • Ensure the toggle for Health monitoring enabled is activated.

Upon activation, the system:

• Checks the server health status.
• Displays the current operational status on the Overview page.
• Creates a structured event record and outputs any significant changes via Kafka, which can be monitored using the Kafka Significant Events Guide.

B. Server Allocation Policy Management

To control how server health status impacts resource assignment, visit Servers $\rightarrow$ Server Allocation Policy. Here, you can define policies that prevent the allocation of resources based on specific conditions:

• Prevent server allocation if it presents critical health issues.
• Prevent server allocation if it presents any general health issue.
• Prevent server allocation if its firmware version drifts from the assigned baseline (Refer to Server Drift Detection for details).

Note if you select “Prevent server allocation if it presents any general health issue”, then you cannot select “Prevent server allocation if it presents critical health issues” as that will be covered already. This will be changed to a dropdown.

Scheduling Jobs

Both drift detection and health monitoring run periodically. The schedule frequency can be managed by editing the corresponding jobs in the Job Queue:

1. Navigate to Job Queue $\rightarrow$ Scheduled Jobs.
2. Edit the relevant job entries:
   • system-defined-server-health-monitoring-site-[Site-ID] (for server health monitoring)
   • system-defined-server-backup-drift-detection-site-[Site-ID] (for backup drift detection)
   • system-defined-server-firmware-baseline-drift-check-site-[Site-ID] (for firmware baseline drift detection)