Skip to content

fix(test): re-run a CI pass wedged by a lost child exit - #1806

Merged
tmchow merged 1 commit into
mainfrom
tmchow/test-stall-zombie-rerun
Sep 29, 2026
Merged

tmchow merged 1 commit into
mainfrom
tmchow/test-stall-zombie-rerun

Conversation

@tmchow

@tmchow tmchow commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

When a bun test worker loses a child process's exit, CI now notices within about a minute, stops the tests, and runs the suite once more in a fresh process. Before, the job sat for 20 minutes and then failed. That happened on 5 of the 6 red test runs on 2026-09-29, including release PR #1781.

What the stall is

The process listing from #1787's watchdog showed the same thing in every stalled run: one bun test --test-worker still alive, its only child a [python3] <defunct> or [git] <defunct> that had exited about 40 seconds into the pass. bun never took that exit, so the worker stayed blocked in the spawnSync call while every other file finished. The hung file differed per run (doc-claims-validator, ce-babysit-pr-snapshot, and ce-work-unit-workspace-fallback three times). The workspace harness's own 20s spawnSync timeout did not fire either, so adding timeouts to test spawns would not help. Only a process outside the worker can recover.

Design decisions

  • Narrow trigger. Only a zombie whose parent is a bun test process counts, and only once it has lasted 60 seconds (CE_TEST_LOST_EXIT_SECONDS overrides). bun takes a child's exit within milliseconds, so a slow test does not trip it. A zombie under any other process is ignored.
  • Whole-pass re-run. A killed pass writes no junit report, so the runner cannot tell which files finished. Re-running everything costs about 4 minutes.
  • Once. A second lost exit fails the job. A stall with no zombie still fails at the 20-minute limit with no re-run, as before.

Validation

  • tests/run-tests-script.test.ts creates a real zombie under a process named like a bun test worker. The runner recovers when only the first attempt loses the exit and fails when both do. A parser test checks that a zombie under a non-bun parent is ignored.
  • bun run test passes locally.

Security Disclosure

No security-relevant changes.

Agent Disclosure

  • Model: Claude Code · claude-opus-5-5[1m]

…ailing at 20 minutes

The stall watchdog's process listing showed every recent stalled `test`
job as a bun test worker holding a `<defunct>` python3 or git child: bun
never took the child's exit, and no per-test or spawnSync timeout frees
the worker. The runner now polls for that zombie, and after 60 seconds
kills the pass and runs it once more in a fresh process. A stall without
the zombie still fails at the 20-minute limit.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-29T18:52:18.352765Z 93a5299 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tmchow
tmchow merged commit cdef1f1 into main Sep 29, 2026
7 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant