Merge branch 'main' into fix/fq-retain-taints-during-validation

Author: lalitadithya

distros/kubernetes/nvsentinel/charts/gpu-health-monitor/templates/configmap.yaml

@@ -37,6 +37,7 @@ data:
     [dcgmfieldsmonitoring]
     gputemplimitmonitoringenabled = {{ .Values.dcgmFieldsMonitoring.gpuTempLimitMonitoringEnabled }}
     gputemplimitstoreonly = {{ .Values.dcgmFieldsMonitoring.gpuTempLimitStoreOnly }}
+    gputemplimitminconsecutivepolls = {{ .Values.dcgmFieldsMonitoring.gpuTempLimitMinConsecutivePolls }}
     gpupowerbrakemonitoringenabled = {{ .Values.dcgmFieldsMonitoring.gpuPowerBrakeMonitoringEnabled }}
     gpupowerbrakestoreonly = {{ .Values.dcgmFieldsMonitoring.gpuPowerBrakeStoreOnly }}
     gpupowerbrakeminconsecutivepolls = {{ .Values.dcgmFieldsMonitoring.gpuPowerBrakeMinConsecutivePolls }}

distros/kubernetes/nvsentinel/charts/gpu-health-monitor/values.yaml

@@ -90,6 +90,11 @@ dcgmFieldsMonitoring:
   # Dry-run for GpuThermalMarginWatch. When true, this check's
   # events are emitted with processingStrategy=STORE_ONLY
   gpuTempLimitStoreOnly: true
+  # Consecutive polls with the margin below the slowdown T.Limit before the GPU is
+  # failed. The margin is sampled once per poll and swings by tens of degrees within
+  # seconds under load, so one sample past the threshold is usually a transient.
+  # 1 fails on first observation.
+  gpuTempLimitMinConsecutivePolls: 3
   # Enable GpuPowerBrakeWatch: fails a GPU whose clocks-event-reasons mask has the
   # HW power brake bit (0x80) set, i.e. the power delivery path is forcing clocks
   # down. DCGM's POWER health watch does not report this; its dominant code

distros/kubernetes/nvsentinel/charts/lifecycle-manager/templates/clusterrole.yaml

@@ -55,6 +55,13 @@ rules:
   verbs:
   - update
   - patch
+- apiGroups:
+  - resource.k8s.io
+  resources:
+  - resourceslices
+  verbs:
+  - list
+  - watch
 - apiGroups:
   - coordination.k8s.io
   resources:

distros/kubernetes/nvsentinel/charts/lifecycle-manager/values.yaml

@@ -63,11 +63,16 @@ config:
     # since a toleration for this taint is always added in test provider templates which
     # consume the Tolerations field. Setting cordon.remove to true will remove the node
     # cordon (and this taint) if present.
+  # Expressions can read the node being validated (node) and the node's ResourceSlices (resourceSlices).
   readinessCriteria:
+    # The device plugin advertises nvidia.com/gpu in node.status.allocatable. In GPU Operator GPUCluster (DRA)
+    # mode, there is no device plugin and the gpu.nvidia.com DRA driver publishes the GPUs in ResourceSlices.
     - name: gpu-allocatable
       expression: >-
-        has(node.status.allocatable) && "nvidia.com/gpu" in node.status.allocatable &&
-        quantity(node.status.allocatable["nvidia.com/gpu"]) > 0
+        (has(node.status.allocatable) && "nvidia.com/gpu" in node.status.allocatable &&
+        quantity(node.status.allocatable["nvidia.com/gpu"]) > 0) ||
+        resourceSlices.exists(s, s.spec.driver == "gpu.nvidia.com" && has(s.spec.devices) &&
+        size(s.spec.devices) > 0)
     - name: not-under-quarantine
       expression: >-
         !(has(node.metadata.annotations) &&

docs/METRICS.md

@@ -13,6 +13,7 @@ This document outlines all Prometheus metrics exposed by NVSentinel components.
 - [Health Monitors](#health-monitors)
   - [Health Event Publisher](#health-event-publisher)
   - [GPU Health Monitor](#gpu-health-monitor)
+  - [NIC Health Monitor](#nic-health-monitor)
   - [Syslog Health Monitor](#syslog-health-monitor)
   - [CSP Health Monitor](#csp-health-monitor)
 - [Change Stream Metrics](#change-stream-metrics)
@@ -301,6 +302,14 @@ These metrics track GPU health events detected via DCGM (Data Center GPU Manager
 
 ---
 
+### NIC Health Monitor
+
+| Metric Name | Type | Labels | Description |
+|------------|------|--------|-------------|
+| `nic_health_monitor_poll_cycle_last_completed_timestamp_seconds` | Gauge | `node`, `category` | Unix timestamp initialized at monitor startup and advanced after each completed poll cycle (`state` or `counter`). Use `time() - metric` to measure how long the category has gone without completing a poll, including a stall in the first poll. |
+
+---
+
 ### Syslog Health Monitor
 
 The syslog health monitor tracks GPU-related errors detected from system logs.

docs/configuration/gpu-health-monitor.md

@@ -253,6 +253,7 @@ gpu-health-monitor:
   dcgmFieldsMonitoring:
     gpuTempLimitMonitoringEnabled: true
     gpuTempLimitStoreOnly: true
+    gpuTempLimitMinConsecutivePolls: 3
 ```
 
 ### gpuTempLimitMonitoringEnabled
@@ -263,6 +264,10 @@ Enables the watch. On by default.
 
 Dry run. When true, this check's events are emitted with `processingStrategy=STORE_ONLY`, so they are persisted and exported as metrics but excluded from the remediation pipeline: no node condition and no cordon. Defaults to true, so the watch is observable before it can act. Set it to `false` once you have confirmed the thresholds suit your hardware and cooling.
 
+### gpuTempLimitMinConsecutivePolls
+
+Consecutive polls with the margin below the slowdown threshold before the GPU is failed. Defaults to 3. The margin is read as one sample per poll, and under load it can move by tens of degrees within seconds, so a single sample past the threshold is usually a transient that the GPU's own hardware slowdown has already handled. A sustained excursion is the actionable case. A sample at or above the threshold resets the counter, and `1` fails on first observation. A GPU with no usable sample is skipped and keeps its counter, so a gap in DCGM data neither raises nor clears a finding. While the counter is below the threshold the GPU is not reported either way, so a restart cannot publish a healthy event for a GPU that is still past the threshold.
+
 To interpret a firing check, see the [GPU Thermal Margin runbook](../runbooks/gpu-thermal-margin.md).
 
 ## NVLink Suppression on Unbridged PCIe Cards
@@ -304,7 +309,7 @@ Dry run. When true, this check's events are emitted with `processingStrategy=STO
 
 ### gpuPowerBrakeMinConsecutivePolls
 
-Consecutive polls with the bit set before the GPU is failed. A brake asserted for a single poll can be a load transient; a sustained assertion is the actionable case. A clear resets the counter, so a flapping brake never accumulates to a failure. `1` fails on first observation. A GPU with no usable sample is skipped and keeps its counter, so a gap in DCGM data neither raises nor clears a finding. This includes DCGM's int64 "no data" sentinels, whose low byte has the brake bit set and which would otherwise read as an assertion.
+Consecutive polls with the bit set before the GPU is failed. A brake asserted for a single poll can be a load transient; a sustained assertion is the actionable case. A clear resets the counter, so a flapping brake never accumulates to a failure. `1` fails on first observation. A GPU with no usable sample is skipped and keeps its counter, so a gap in DCGM data neither raises nor clears a finding. This includes DCGM's int64 "no data" sentinels, whose low byte has the brake bit set and which would otherwise read as an assertion. While the counter is below the threshold the GPU is not reported either way, so a restart cannot publish a healthy event for a GPU whose brake is still asserted.
 
 ## DCGM Startup Gate
 

docs/configuration/validation.md

@@ -27,8 +27,10 @@ lifecycle-manager:
     readinessCriteria:
       - name: gpu-allocatable
         expression: >-
-          has(node.status.allocatable) && "nvidia.com/gpu" in node.status.allocatable &&
-          quantity(node.status.allocatable["nvidia.com/gpu"]) > 0
+          (has(node.status.allocatable) && "nvidia.com/gpu" in node.status.allocatable &&
+          quantity(node.status.allocatable["nvidia.com/gpu"]) > 0) ||
+          resourceSlices.exists(s, s.spec.driver == "gpu.nvidia.com" && has(s.spec.devices) &&
+          size(s.spec.devices) > 0)
       - name: not-under-quarantine
         expression: >-
           !(has(node.metadata.annotations) &&
@@ -58,13 +60,20 @@ lifecycle-manager:
         command:
           - sh
           - -c
-          - dcgmi diag --host "nvidia-dcgm.gpu-operator.svc:5555" --run 2 --json
+          - |
+            # One DCGM Service exists per cluster: nvidia-dcgm-dra in GPU Operator GPUCluster (DRA) mode,
+            # nvidia-dcgm otherwise. Use the first name that resolves.
+            for h in nvidia-dcgm-dra.gpu-operator.svc nvidia-dcgm.gpu-operator.svc; do
+              getent hosts "$h" >/dev/null 2>&1 && exec dcgmi diag --host "$h:5555" --run 2 --json
+            done
+            echo "no DCGM hostengine Service resolved" >&2
+            exit 1
         supportsBatchingNodes: false
         minimumNodesPerBatch: 1
         batchFailurePolicy: fail
 ```
 
-This configuration allows a client to create a ValidationRequest that runs dcgm-diag-test as a Kubernetes Job via the k8s-job-provider. Before the test group starts, the targeted node must report allocatable GPU capacity and not be under quarantine, as enforced by readinessCriteria. If the ValidationRequest does not specify spec.tests, defaultTests provides which tests to run in the request.
+This configuration allows a client to create a ValidationRequest that runs dcgm-diag-test as a Kubernetes Job via the k8s-job-provider. Before the test group starts, the targeted node must report allocatable GPU capacity and not be under quarantine, as enforced by readinessCriteria. The gpu-allocatable criterion accepts either source of GPU capacity: the device plugin, which sets `nvidia.com/gpu` in the node allocatable resources, or the `gpu.nvidia.com` DRA driver, which publishes the GPUs in ResourceSlices in GPU Operator GPUCluster mode. If the ValidationRequest does not specify spec.tests, defaultTests provides which tests to run in the request.
 
 ## Enabling New Node Validation
 
@@ -281,7 +290,7 @@ status:
 | Key | Type | Purpose |
 |---|---|---|
 | defaultTests | []string | The default set of tests run against ValidationRequests which do not include any tests |
-| readinessCriteria | []CriteriaSpec | A set of CEL expressions which must all evaluate to true before a validation test can be started on a given node. Each entry is evaluated against an environment containing the node being validated. If an operator is externally applying a node cordon or taint and would like to block validation until these are applied, they can add these properties to readinessCriteria. If not met, this blocks a node from starting validation and fails validation if the criteria were initially met and then reverted |
+| readinessCriteria | []CriteriaSpec | A set of CEL expressions which must all evaluate to true before a validation test can be started on a given node. Each entry is evaluated against an environment containing the node being validated (`node`) and the ResourceSlices whose `spec.nodeName` is that node (`resourceSlices`). When a criterion reads `resourceSlices`, the lifecycle-manager watches ResourceSlices so a slice published after the last node update still unblocks a pending request. It derives the drivers to watch from the expressions: slice events are filtered to the `spec.driver` values compared with string literals, as `s.spec.driver == "gpu.nvidia.com"` or `s.spec.driver in [...]`. A criterion that reads `spec.driver` any other way, or reads slices without testing the driver, turns the filter off and every driver's slice events are processed (logged at startup). Filtering only affects which events wake the controller; the `resourceSlices` variable always holds all of the node's slices. If an operator is externally applying a node cordon or taint and would like to block validation until these are applied, they can add these properties to readinessCriteria. If not met, this blocks a node from starting validation and fails validation if the criteria were initially met and then reverted |
 | maxConcurrentGroups | int | The maximum number of test groups that may run concurrently. Groups are additionally constrained by node overlap. Two groups that share a node never run at the same time regardless of this setting |
 | templateMountPath | string | The directory from which templateFile paths are resolved |
 | providers | map[string]ProviderConfig | Test provider settings that apply to all tests using this provider, keyed by the name tests[].provider references (see below) |
@@ -321,7 +330,7 @@ status:
 | Key | Type | Purpose |
 |---|---|---|
 | condition | string | The name of the node condition the controller uses to track whether a node has already been validated. For a node to be targeted, this condition must be absent or false and every criteria expression must evaluate to true. Once a ValidationRequest is created, the controller sets this condition to True on the node so that subsequent evaluations no longer match |
-| criteria | []CriteriaSpec | A set of CEL expressions evaluated against each node to determine whether it requires new node validation. All expressions must evaluate to true, along with the condition check above. The CEL environment exposes the node being validated |
+| criteria | []CriteriaSpec | A set of CEL expressions evaluated against each node to determine whether it requires new node validation. All expressions must evaluate to true, along with the condition check above. The CEL environment exposes the node being validated (`node`) and its node-local ResourceSlices (`resourceSlices`), the same as readinessCriteria |
 | newNodeTests | []string | The list of tests to run for new nodes. These take precedence over defaultTests when a ValidationRequest is created for a new node |
 | batchPeriodSeconds | int64 | The window during which the controller collects eligible new nodes before creating ValidationRequests for them as a batch. Only applies to new node validation |
 

docs/runbooks/gpu-thermal-margin.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-`GpuThermalMarginWatch` monitors each GPU's live thermal margin (DCGM field 153, `DCGM_FI_DEV_GPU_TEMP_LIMIT`) against the per-SKU hardware-slowdown T.Limit offset published by the metadata-collector (NVML field 194, `FI_DEV_TEMPERATURE_SLOWDOWN_TLIMIT`). When a GPU's margin drops below its slowdown threshold, the GPU is at or past the temperature at which the hardware engages thermal slowdown, the fatal event for GpuThermalMarginWatch occurs. The feature is described in [ADR-042: GPU Thermal Margin](../designs/042-gpu-temp-limit-field-monitoring.md).
+`GpuThermalMarginWatch` monitors each GPU's live thermal margin (DCGM field 153, `DCGM_FI_DEV_GPU_TEMP_LIMIT`) against the per-SKU hardware-slowdown T.Limit offset published by the metadata-collector (NVML field 194, `FI_DEV_TEMPERATURE_SLOWDOWN_TLIMIT`). When a GPU's margin stays below its slowdown threshold for `gpuTempLimitMinConsecutivePolls` consecutive polls (3 by default), the GPU has been at or past the temperature at which the hardware engages thermal slowdown, and the fatal event for GpuThermalMarginWatch occurs. The feature is described in [ADR-042: GPU Thermal Margin](../designs/042-gpu-temp-limit-field-monitoring.md).
 
 **Key points:**
 
@@ -124,5 +124,6 @@ The condition clears automatically once the live margin returns to at or above t
 The check is configured through the `gpu-health-monitor` Helm chart, which renders the `[dcgmfieldsmonitoring]` section of `config.ini`:
 
 - Enable the check: Helm value `dcgmFieldsMonitoring.gpuTempLimitMonitoringEnabled` renders `gputemplimitmonitoringenabled`.
+- Consecutive polls required to fail: Helm value `dcgmFieldsMonitoring.gpuTempLimitMinConsecutivePolls` renders `gputemplimitminconsecutivepolls`.
 
 The per-GPU threshold (`slowdown_tlimit_c`) is not a Helm value. It is collected at runtime by the metadata-collector and written to `/var/lib/nvsentinel/gpu_metadata.json`.

health-monitors/gpu-health-monitor/gpu_health_monitor/cli.py

@@ -207,17 +207,22 @@ def cli(
 
     thermal_margin_enabled = False
     thermal_margin_store_only = False
+    thermal_margin_min_consecutive_polls = 1
     power_brake_enabled = False
     power_brake_store_only = False
     power_brake_min_consecutive_polls = 1
     if config.has_section("dcgmfieldsmonitoring"):
         fields_monitoring_config = config["dcgmfieldsmonitoring"]
         thermal_margin_enabled = fields_monitoring_config.getboolean("gputemplimitmonitoringenabled", fallback=False)
         thermal_margin_store_only = fields_monitoring_config.getboolean("gputemplimitstoreonly", fallback=False)
+        thermal_margin_min_consecutive_polls = fields_monitoring_config.getint(
+            "gputemplimitminconsecutivepolls", fallback=1
+        )
         log.info(
-            "GpuThermalMarginWatch field monitor: enabled=%s store_only=%s",
+            "GpuThermalMarginWatch field monitor: enabled=%s store_only=%s min_consecutive_polls=%s",
             thermal_margin_enabled,
             thermal_margin_store_only,
+            thermal_margin_min_consecutive_polls,
         )
 
         power_brake_enabled = fields_monitoring_config.getboolean("gpupowerbrakemonitoringenabled", fallback=False)
@@ -361,6 +366,7 @@ def process_exit_signal(signum, frame):
         poll_interval_seconds=poll_interval,
         dcgm_k8s_service_enabled=dcgm_k8s_service_enabled,
         thermal_margin_enabled=thermal_margin_enabled,
+        thermal_margin_min_consecutive_polls=thermal_margin_min_consecutive_polls,
         dcgm_mode=dcgm_mode,
         suppressed_error_codes=suppressed_error_codes,
         suppress_unbridged_pcie_nvlink_down=suppress_nvlink_down_unbridged_pcie,