Skip to content

Fix interrupted Autopilot suites that never start - #3509

Draft
Serhan-Asad wants to merge 3 commits into
mainfrom
pdd/core-3473-mwl6f633-82136e
Draft

Serhan-Asad wants to merge 3 commits into
mainfrom
pdd/core-3473-mwl6f633-82136e

Conversation

@Serhan-Asad

@Serhan-Asad Serhan-Asad commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #3473

A suite owner can receive SIGTERM while waiting for admission, before any test command starts. The queue now records that attempt as interrupted with the original signal and no test exit code. Core shows waiting separately from execution and preserves the latest interruption across the existing runner retries. This does not establish who sent the incident’s SIGTERM.

Bind all 128 criteria in the 34 affected Runtime scenarios to their exact Story paths and text. No Story, fixture, operation, or expected value changes. The two Stories that timed out in the batch were retried individually at 085e6cf: workflow-step-results-2922 (20261117-213840-du3v) and workflow-github-symlinked-repo (20261117-213955-o46j). Lint passed for both. Both saved reports returned no verdict (login-queue-timeout, zero failed checks, usage unknown). The other 32 Stories still have real-binding-unavailable with zero provider calls. The Runtime capacity refusal remains unresolved.

No Story: native OS signals and process receipts exercise the existing validation loop; no new Runtime interface.

The focused native tests passed 148/148 and affected replays passed 672/672. Both interruption regressions failed at the current merge base on the waiting status or ticket receipt and passed at this head. No contract change in the launcher: it forwards the existing queue environment to Core status (prompts/cli/autopilot-launcher.prompt.md:241).

Choices Autopilot made:

  • Use a finished ticket with an interrupted outcome and a null test exit (preserves terminal queue accounting without fabricating a test result) (changes public behavior: an interrupted waiter exposes the original signal and a not-run receipt).
  • Project a dead unstarted waiter as interrupted without rewriting storage from status (a process that has already died cannot write a terminal receipt) (changes public behavior: a stale waiter no longer appears live).
  • Keep the interrupted attempt separate from its replacement (a later test result belongs to the retry) (changes public behavior: status and evidence retain the interruption without merging it into the retry verdict).
  • Migrate criterion IDs and their references only (preserves every authored expectation and satisfies the interpreter’s content-binding rule).
  • Run the existing Core interruption regression as a replay (makes the base comparison exercise the behavior fix rather than only the migrated scenario metadata).

Runtime: not run (user_stories/adopt-approved-corpus.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-author.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-invalid-mode.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-invalid-type.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-mode.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-setup-auto.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-setup-observe.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-setup-trust.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-setup.md: real-binding-unavailable)
Runtime: not run (user_stories/autopilot-trust.md: real-binding-unavailable)
Runtime: not run (user_stories/help-teaches-pdd.md: real-binding-unavailable)
Runtime: not run (user_stories/interpreter-build.md: real-binding-unavailable)
Runtime: not run (user_stories/name-the-interpreter.md: real-binding-unavailable)
Runtime: not run (user_stories/native-criterion-evidence.md: real-binding-unavailable)
Runtime: not run (user_stories/native-validation-profile.md: real-binding-unavailable)
Runtime: not run (user_stories/new-command-names.md: real-binding-unavailable)
Runtime: not run (user_stories/product-issue-binding.md: real-binding-unavailable)
Runtime: not run (user_stories/product-map.md: real-binding-unavailable)
Runtime: not run (user_stories/product-next.md: real-binding-unavailable)
Runtime: not run (user_stories/product-repair-link.md: real-binding-unavailable)
Runtime: not run (user_stories/product-saved-checks.md: real-binding-unavailable)
Runtime: not run (user_stories/product-search-cache.md: real-binding-unavailable)
Runtime: not run (user_stories/product-start.md: real-binding-unavailable)
Runtime: not run (user_stories/product-state-updates.md: real-binding-unavailable)
Runtime: not run (user_stories/product-status.md: real-binding-unavailable)
Runtime: not run (user_stories/product-unavailable-actor.md: real-binding-unavailable)
Runtime: not run (user_stories/product-unresolved-link.md: real-binding-unavailable)
Runtime: not run (user_stories/product-worktree-links.md: real-binding-unavailable)
Runtime: not run (user_stories/service-build-version.md: real-binding-unavailable)
Runtime: not run (user_stories/use-a-signed-in-workflow-agent.md: real-binding-unavailable)
Runtime: not run (user_stories/workflow-name-guidance.md: real-binding-unavailable)
Runtime: not run (user_stories/workflow-prove.md: real-binding-unavailable)
Runtime: not run (user_stories/workflow-step-results-2922.md: login-queue-timeout after individual retries (2 Stories, no failed checks, usage unknown); real-binding-unavailable (32 )
Runtime: not run (user_stories/workflow-github-symlinked-repo.md: login-queue-timeout after individual retries (2 Stories, no failed checks, usage unknown); real-binding-unavailable (32 )
Path: a — prompts/cli/autopilot-core.prompt.md, src/autopilot-core.mjs — prompts/cli/autopilot-core.prompt.md — prompt changed — Test: test/autopilot-suite-interruption.vectors.test.mjs, test/runtime/adopt-approved-corpus.json, test/runtime/autopilot-author.json, test/runtime/autopilot-invalid-mode.json, test/runtime/autopilot-invalid-type.json
Path: a — prompts/cli/test-suite-queue.prompt.md, src/test-suite-queue.mjs — prompts/cli/test-suite-queue.prompt.md — prompt changed — Test: test/autopilot-suite-interruption.vectors.test.mjs, test/runtime/adopt-approved-corpus.json, test/runtime/autopilot-author.json, test/runtime/autopilot-invalid-mode.json, test/runtime/autopilot-invalid-type.json
Path: b — src/autopilot-launcher.mjs — prompts/cli/autopilot-launcher.prompt.md — no contract change — Test: test/autopilot-suite-interruption.vectors.test.mjs, test/runtime/adopt-approved-corpus.json, test/runtime/autopilot-author.json, test/runtime/autopilot-invalid-mode.json, test/runtime/autopilot-invalid-type.json
Path: d — test/autopilot-suite-interruption.vectors.test.mjs, test/runtime/adopt-approved-corpus.json, test/runtime/autopilot-author.json, test/runtime/autopilot-invalid-mode.json, test/runtime/autopilot-invalid-type.json, test/runtime/autopilot-mode.json, test/runtime/autopilot-setup-auto.json, test/runtime/autopilot-setup-observe.json, test/runtime/autopilot-setup-trust.json, test/runtime/autopilot-setup.json, test/runtime/autopilot-trust.json, test/runtime/help-teaches-pdd.json, test/runtime/interpreter-build.json, test/runtime/name-the-interpreter.json, test/runtime/native-criterion-evidence.json, test/runtime/native-validation-profile.json, test/runtime/new-command-names.json, test/runtime/product-issue-binding.json, test/runtime/product-map.json, test/runtime/product-next.json, test/runtime/product-repair-link.json, test/runtime/product-saved-checks.json, test/runtime/product-search-cache.json, test/runtime/product-start.json, test/runtime/product-state-updates.json, test/runtime/product-status.json, test/runtime/product-unavailable-actor.json, test/runtime/product-unresolved-link.json, test/runtime/product-worktree-links.json, test/runtime/service-build-version.json, test/runtime/use-a-signed-in-workflow-agent.json, test/runtime/workflow-name-guidance.json, test/runtime/workflow-prove.json, test/runtime/workflow-step-results-2922.json, test/test-suite-queue.test.mjs, tests/runtime/workflow-github-symlinked-repo.json — unowned — no contract change — Test: test/autopilot-suite-interruption.vectors.test.mjs, test/runtime/adopt-approved-corpus.json, test/runtime/autopilot-author.json, test/runtime/autopilot-invalid-mode.json, test/runtime/autopilot-invalid-type.json
Evidence: runtime=not-run; vectors=pass (672/672); formal=not-applicable; tests=pass (148/148)

@Serhan-Asad

Copy link
Copy Markdown
Collaborator Author

Review: FAIL for head 085e6cf (a fresh Autopilot reviewer that did not write the change; same-family, openai).

The waiting-owner SIGTERM regression and focused suite passed (16/16), and the engine recorded 672/672 passing replays with the new regression failing on the base control. However, Core can retry a successful suite based on a different attempt's ticket, and it misses the interrupted ticket entirely when the Core worktree path traverses a symlink. The ownership Paths match; all 128 criterion hashes bind to exact Story paths and text, with no expectation changes. Runtime remains unverified: the saved retries for two Stories show login-queue-timeout with no failed checks, and the other 32 report real-binding-unavailable. Read: prompts/cli/autopilot-core.prompt.md, prompts/cli/test-suite-queue.prompt.md, src/autopilot-core.mjs, src/autopilot-launcher.mjs, src/test-suite-queue.mjs, test/autopilot-suite-interruption.vectors.test.mjs, test/runtime/adopt-approved-corpus.json, test/runtime/autopilot-author.json, test/runtime/autopilot-invalid-mode.json, test/runtime/autopilot-invalid-type.json, test/runtime/autopilot-mode.json, test/runtime/autopilot-setup-auto.json, test/runtime/autopilot-setup-observe.json, test/runtime/autopilot-setup-trust.json, test/runtime/autopilot-setup.json, test/runtime/autopilot-trust.json, test/runtime/help-teaches-pdd.json, test/runtime/interpreter-build.json, test/runtime/name-the-interpreter.json, test/runtime/native-criterion-evidence.json, test/runtime/native-validation-profile.json, test/runtime/new-command-names.json, test/runtime/product-issue-binding.json, test/runtime/product-map.json, test/runtime/product-next.json, test/runtime/product-repair-link.json, test/runtime/product-saved-checks.json, test/runtime/product-search-cache.json, test/runtime/product-start.json, test/runtime/product-state-updates.json, test/runtime/product-status.json, test/runtime/product-unavailable-actor.json, test/runtime/product-unresolved-link.json, test/runtime/product-worktree-links.json, test/runtime/service-build-version.json, test/runtime/use-a-signed-in-

Must fix:

  • P1 src/autopilot-core.mjs:2805 Correlate queue tickets to the specific Core invocation, not just the checkout and a lower time bound. A separate invocation in the same checkout can enqueue later and interrupt before admission. In an isolated probe, Core labeled its own completed exit-zero suite as not run, saved the other ticket and signal as evidence, and spent a retry. Bind the ticket to its owning attempt and test that Core status and evidence keep the successful result separate from the unre
  • P1 src/autopilot-core.mjs:2804 Recognize the current ticket when the Core state/worktree path traverses a symlink. The suite process records the physical cwd but Core looks up its lexical path; in an isolated probe the interrupted ticket and matching SIGTERM wrapper receipt existed, yet status said the suite had started and the retry evidence lost the interruption. Use a lookup that survives both paths referring to the same checkout and add a regression for the exposed status and retained signal
  • P2 PR-body:50 Give each of the two timed-out Stories its own exact Runtime reason, login-queue-timeout. The saved reports for workflow-step-results-2922 and workflow-github-symlinked-repo show no verdict and no failed checks, but both Runtime lines also include the unrelated real-binding-unavailable count for the other 32 Stories. Keep that aggregate detail in the narrative rather than either per-Story line.

@Serhan-Asad

Copy link
Copy Markdown
Collaborator Author

Autopilot ended the run for issue #3473: the full test suite runner failed for head 085e6cf (killed by SIGTERM, 3 tries). Log: 3473-suite-mwl8qc9r-8184c0.log on this machine.

This PR stays open as a draft. It is not merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[alpha] A terminated waiting suite retains running queue evidence without starting any tests

1 participant