Repository navigation
Open
0 / 10 of 1 issue completedLabels
Description
Priority
Wave 1 — local lifecycle correctness. Live progress presentation is a separate #2831/#2859 workstream.
Problem
Local startup and shutdown do not share one bounded ownership boundary. Readiness can wait indefinitely, startup failure can bypass cleanup, shutdown uses per-process waits without one total deadline, and cancellation may be converted into an HTTP 500.
Essential scope
- Transactional
RunHelperacquisition: record each resource immediately and unwind everything acquired before startup failure. - Finite overall deadlines for head readiness, component readiness, configured endpoint readiness, cancellation cleanup, and shutdown.
- Idempotent shutdown.
- Start commands in process groups or another explicit descendant-ownership mechanism.
- Signal, wait, escalate, reap, and continue cleaning other components within one total shutdown budget.
- Re-raise
CancelledError,KeyboardInterrupt, andSystemExitafter bounded cleanup rather than converting them to ordinary HTTP failures. - Report unresolved remote cleanup when feat(core): define resources-session identity and close protocol #2609/feat(resources): add session TTL and lifecycle adoption #3037 data exists without claiming local process exit released it.
Moved or deferred scope
- Oldest active attempt, progress heartbeats, recent model/tool activity, and stalled-but-live presentation belong to feat: live visibility into in-progress rollouts #2859 under [epic] Rollout observability & reliability fast-follows #2831.
- Automatic restart and circuit breaking are out of scope.
- Remote session identity and close belong to feat(core): define resources-session identity and close protocol #2609/feat(resources): add session TTL and lifecycle adoption #3037.
Dependencies
No hard dependency for local correctness. #144 depends on the durable process ownership and shutdown API established here. #2859 later consumes stable progress and identity APIs without blocking this issue.
Acceptance criteria
- Fault injection after each startup stage releases every earlier acquisition.
- Every readiness stage has one finite overall deadline and identifies the failed component.
- Shutdown is idempotent and terminates owned descendants within one configured deadline.
- Cancellation propagates after bounded cleanup and is not converted to HTTP 500.
- Cleanup continues when one owned component does not exit.
- Remote cleanup remains a separately reported state when available.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDev Todo
Activity
[-]feat(core): own startup, shutdown, cancellation, and stall detection[/-][+]feat(core): own startup, shutdown, and cancellation[/+]fix(cli): bound Gym server startup wait (NVIDIA-NeMo#3735)
fix(cli): bound Gym server startup wait (#3735)