Feature Summary
Allow fault-quarantine to keep its quarantine taints on a node while post-remediation validation runs, the same way it keeps the cordon today.
Problem/Use Case
When a quarantine session recovers and fault-quarantine creates a ValidationRequest, it keeps the cordon but always removes its rule-set taints (triggerValidationOnUnquarantine in fault-quarantine/pkg/reconciler/reconciler.go). With taint-only rule sets (cordon.shouldCordon = false), the node therefore accepts workloads while validation tests are still running.
Taint-only quarantine is useful when other components manage the cordon. For example, k8s-driver-manager cordons and uncordons the node every time the driver pod starts (NVIDIA/k8s-driver-manager#212). After a remediation reboot, fault-quarantine treats that uncordon as a manual uncordon and the session is cancelled before validation runs. A taint that fault-quarantine owns is not affected by external cordon changes, but today it does not survive the hand-off to validation.
lifecycle-manager already supports this: schedulingGate.taints entries are tolerated by test pods and lifted when validation passes, and the docs say they are expected to be applied externally before the ValidationRequest is created.
Proposed Solution
An opt-in fault-quarantine.validation.retainTaints setting, default false. When it is true and a ValidationRequest is created, fault-quarantine leaves the session's quarantine taints on the node and still removes its bookkeeping annotations. lifecycle-manager lifts the taints listed in schedulingGate.taints (with remove: true) when validation passes, and leaves them when it fails. The default behaviour does not change.
Component
Fault Management (fault-quarantine), Deployment/Config
Sent from Claude Code
Feature Summary
Allow fault-quarantine to keep its quarantine taints on a node while post-remediation validation runs, the same way it keeps the cordon today.
Problem/Use Case
When a quarantine session recovers and fault-quarantine creates a ValidationRequest, it keeps the cordon but always removes its rule-set taints (
triggerValidationOnUnquarantineinfault-quarantine/pkg/reconciler/reconciler.go). With taint-only rule sets (cordon.shouldCordon = false), the node therefore accepts workloads while validation tests are still running.Taint-only quarantine is useful when other components manage the cordon. For example, k8s-driver-manager cordons and uncordons the node every time the driver pod starts (NVIDIA/k8s-driver-manager#212). After a remediation reboot, fault-quarantine treats that uncordon as a manual uncordon and the session is cancelled before validation runs. A taint that fault-quarantine owns is not affected by external cordon changes, but today it does not survive the hand-off to validation.
lifecycle-manager already supports this:
schedulingGate.taintsentries are tolerated by test pods and lifted when validation passes, and the docs say they are expected to be applied externally before the ValidationRequest is created.Proposed Solution
An opt-in
fault-quarantine.validation.retainTaintssetting, defaultfalse. When it istrueand a ValidationRequest is created, fault-quarantine leaves the session's quarantine taints on the node and still removes its bookkeeping annotations. lifecycle-manager lifts the taints listed inschedulingGate.taints(withremove: true) when validation passes, and leaves them when it fails. The default behaviour does not change.Component
Fault Management (fault-quarantine), Deployment/Config
Sent from Claude Code