[onedr0p]

Fixes #319.

An orphaned device hold — a leaked filesystem freeze's dead superblock pinning open_cnt on the DRBD device with every mountpoint gone (the #311 aftermath) — makes drbdsetup down fail with (-12) Device is held open by someone forever. Teardown parked on that error indefinitely (648 attempts / 12h in the field report), paging noise for deletion-targeted volumes with zero workload impact.

The issue proposed escalating to an event + terminal condition, and optionally attempting Driver.Restart as recovery. Neither survives contact with the mechanics:

• Restart cannot work here: its downResource runs the same drbdsetup down that is refused while open_cnt > 0. (The fsfreeze: leaked bd_fsfreeze_count after agent restart permanently unmounts a volume (catch-22: Thaw needs a mountpoint, mount is blocked by the freeze) #311 workaround succeeded because that state had open:no.)
• A terminal give-up is wrong too: the finalizer must survive (Teardown wedges: drbdsetup down SIGKILLed in a loop, only a reboot clears it (recurring) #195), and the slow retry is what notices a reboot.

The fix: complete the teardown instead of reporting it better

The only thing an orphaned hold pins is the DRBD minor itself. The backing device, the space, and the CR are all reclaimable — they were only blocked because teardown was serialized behind down succeeding first. When the hold is provably orphaned, teardown now routes around the pinned minor:

1. drbd.Driver.ForceDetach (new): disconnect peers, then drbdsetup detach <minor> --force. Force-detach is legal on a held-open device — dropping a failing disk under live I/O is its purpose — and it releases the backing. Refuses the wedge signature; no-ops when already Diskless or not in the kernel, so retries converge.
2. Pools.SweepDelete destroys the freed backing; the finalizer releases through teardown's normal tail. A Normal TeardownReclaimed event names what happened.
3. The rendered config and minor assignment deliberately stay: the kernel minor is pinned until a reboot, and releasing the number would hand it to a new volume. After the eventual reboot, the existing SweepOrphans startup pass sees an unowned .res with no kernel resource and reaps both — the residue is self-cleaning through machinery that already exists.

The orphan proof (agent.orphanHold)

All of, before the reclaim may fire:

• deletion-targeted volume (a removed replica's volume lives on and keeps the parked retry), known minor;
• the busy streak already crossed busyFailLimit (~5 min) — an in-flight NodeUnstage never lasts that long;
• the held-open failure lists openers (the kernel's opener report relayed by drbdsetup) and every opener pid has exited;
• the device is mounted in no mount namespace on the node. The agent runs hostPID, so the scan walks every process's mountinfo (deduped by ns/mnt) — this is what distinguishes a dead superblock from a force-deleted pod whose container still runs with the volume mounted in its own namespace while the host table is clean (Teardown wedges: drbdsetup down SIGKILLed in a loop, only a reboot clears it (recurring) #195: never destroy a backing under a consumer). Raw-block publishes are matched by the devtmpfs bind root, filesystem mounts by the device's major:minor.

Any doubt — no opener list, an unreadable /proc, a surviving mount, a live pid — reads as "live consumer" and the parked retry stands, exactly as before.

The wedge stays parked, quietly

A kernel-wedged teardown (LINBIT/drbd#137) has no equivalent: DRBD still pins the backing mid-detach, and releasing the finalizer would leak it forever. That path keeps park-and-page, but now latches — reportWedged re-emitted its Warning Event, Error log, and a status write every 5-minute cycle (123 of them in the field report); it now emits once on the transition and stays at V(1) while parked, with the paging gauge still re-raised every cycle. reportBusy's parked-cycle Error log (the 648-line stream) demotes the same way; its Event and status Message were already latched.

Testing

• Unit: ForceDetach (sequence, wedge refusal, Diskless/absent no-ops), the orphan-proof helpers (opener parsing, cross-namespace mount scan incl. the container-namespace hazard, unreadable-/proc fail-closed), reconciler paths (reclaim after the full escalation, live-opener and mounted-device stay parked, wedge latch).
• Integration (envtest): green.
• golangci-lint: 0 issues.
• E2E (new spec, test/e2e/orphanhold_test.go): reproduces agent: reconcile backoff should not retry unwinnable errors indefinitely (busy openers, kernel wedge) #319 on the real QEMU cluster — freeze the staged filesystem through the staging mountpoint, lazily unmount it out from under the fix(agent): recover volumes wedged by a leaked filesystem freeze #312 thaw-at-unstage guard, delete the pod, verify the kernel still counts an opener with no mounts left (open:yes), then delete the PVC and assert the MiroirVolume completes deletion through the reclaim, the backing LV is destroyed, the zombie minor remains, and the TeardownReclaimed event fired. Runs on the conformance leg (RUN_SPECS=1); the spec timeout accounts for riding the full busy escalation (test.sh go-test timeout raised to 35m).

[greptile-apps]

Greptile Summary

This PR adds a guarded reclaim path for DRBD volume deletions blocked by orphaned device holds. The main changes are:

• New orphan-hold checks for DRBD opener PIDs and cross-namespace mount visibility.
• A ForceDetach path that releases the backing device while keeping the pinned minor reserved until reboot.
• Reconciler logic that reclaims deletion-targeted volumes only after the busy escalation threshold is crossed.
• Latched busy and wedged reporting to avoid repeated identical events and status churn.
• Unit and QEMU e2e coverage for the reclaim path and live-consumer safeguards.

Confidence Score: 5/5

Safe to merge with low risk.

The reclaim path is narrowly gated by deletion state, known minor, busy escalation, opener parsing, PID liveness, and a fail-closed mount scan. The previously reported mountinfo failure and namespace dedupe cases are addressed in the current diff, with focused unit coverage and an e2e reproduction.

No files require special attention.

T-Rex Logs

What T-Rex did

• • Ran DRBD force-detach and wedge tests; the targeted tests passed.
• • Executed unit tests for the orphan-hold agent helper and reconciler reclaim/parking; the tests passed.
• • Built the e2e package with the e2e build tag and verified compilation succeeded.
• • Identified tooling blockers that prevent full end-to-end validation, including missing local tools for QEMU/Kubernetes e2e, envtest, lint, and cluster validation.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
internal/agent/orphanhold.go	Adds fail-closed orphaned-hold detection helpers for parsing DRBD opener PIDs and scanning all visible mount namespaces.
internal/agent/reconciler.go	Routes deletion teardown around proven orphaned DRBD holds via force detach and quiets repeated busy/wedged escalation reporting.
internal/drbd/drbd.go	Refactors peer disconnect logic and adds idempotent ForceDetach with wedge refusal and Diskless/absent no-ops.
test/e2e/orphanhold_test.go	Adds an ordered QEMU e2e reproduction that verifies reclaim of a frozen orphaned DRBD hold and expected residue.

_{Reviews (3): Last reviewed commit: "fix(agent): keep a shared mount namespac..." | Re-trigger Greptile}