Skip to content

fix(acp): retry a failed session directory removal after the agent exits - #1626

Merged
okwasniewski merged 4 commits into
e2e-inc:mainfrom
Varshith-Kali:fix/1623-acp-teardown-eperm
Nov 8, 2026
Merged

okwasniewski merged 4 commits into
e2e-inc:mainfrom
Varshith-Kali:fix/1623-acp-teardown-eperm

Conversation

@Varshith-Kali

@Varshith-Kali Varshith-Kali commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1623.

On Windows, teardown() sent SIGTERM to the agent and then removed its session directory at once. Windows refuses to remove a directory while a process still has it as its working directory, so rmSync threw EPERM synchronously and took the worker down with WORKER_CRASH. Because the test never passed, no recording was saved, so every rerun burned another model call.

The directory is now removed at once, and retried after the agent exits when the first removal fails. On POSIX the first removal usually succeeds while the agent is still stopping, so a worker exiting right after the attempt leaves nothing behind; on Windows a process the agent started in the directory still holds it, so the removal fails there and waits for the exit. A failed removal never throws: a leftover temp directory is harmless.

Verified

  • 2 regression tests in packages/acp/tests/executor.test.ts: the removal is retried after the agent exits when the first try fails (forced by a file in a read-only directory), and on POSIX the directory is gone at once while the agent is still alive. The new immediate-path test fails on the remove-only-after-exit code (red verified) and passes with the fix. Both are skipped on Windows, where Node terminates the child immediately for SIGTERM and the delay handler cannot run, so they check the ordering on POSIX only. The fallback test also skips as root, where a read-only directory does not block removal.
  • packages/acp executor tests 33/33 pass; full acp suite 36/39. The 3 cli.test.ts failures are pre-existing and unrelated: identical on clean origin/main, caused by Chromium refusing to launch as root without --no-sandbox in this sandbox; they should pass in CI.
  • typecheck clean, oxlint clean.
  • The Windows EPERM itself cannot reproduce on Linux (POSIX allows removing a live process's cwd), and no CI workflow runs on Windows, so the EPERM path itself is unverified.

View guided diff

On Windows, teardown sent SIGTERM and removed the agent's working directory at once. Windows cannot remove a directory a live process holds as its working directory, so rmSync threw EPERM synchronously and took the worker down with WORKER_CRASH, losing the recording and forcing another model call on the next run.

Wait for the child to exit before removing its directory, remove it at once when the child already exited, retry transient locks (maxRetries 10, retryDelay 200), and never let a failed removal throw: a leftover temp directory is harmless.

Also notes in the app.command docs that on Windows executable must name a real executable file, since npm is npm.cmd.

Fixes e2e-inc#1623

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@github-actions github-actions Bot added the size:S Up to 29 changed lines, not counting tests, fixtures, translated docs, or the lockfile label Nov 7, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-opus-5-5] automated review on behalf of @okwasniewski

Thanks. The fix is right: removing the directory after the agent exits is the root cause from #1623, and "removes the session directory only after the agent exits" fails on the old code. A few small asks.

Must fix

  1. session.ts:166: a sync rmSync with maxRetries: 10, retryDelay: 200 blocks the event loop while it retries. With a removal forced to fail on Node 24, the worker froze for 9 s with 0 timer ticks. On Windows, a grandchild the agent started in the session cwd can still hold the directory after the direct child exits, because child.kill only terminates the direct child. That freeze then lands on every teardown, including mid-test ones.

  2. session.ts:158-176: the existing exited promise already settles on exit and on spawn error, and is already settled when the child is gone. The removed flag, the already-exited branch and both listeners can collapse to one async line, which also fixes item 1:

    void exited.catch(() => rm(cwd, { recursive: true, force: true, maxRetries: 10, retryDelay: 200 })).catch(() => undefined);

    with rm from node:fs/promises. Tried locally: typecheck clean, executor tests 33/33. As it stands, the close listener, the exit listener, the try/catch, the removed guard and the retry options all survive mutation.

  3. Comments at session.ts:158-160 and executor.test.ts:500-503 say the child holds the dir as its working directory. spawn at session.ts:116 sets no cwd; what holds it is whatever the agent starts there. Reword, or drop them along with item 2.

  4. Please drop the docs/reference/config.mdx:509 note from this PR.

    • It's a core runner doc in a fix(acp) PR.
    • docs/quickstart.mdx:43 says to use WSL on Windows.
    • It contradicts the 'pnpm' and 'npm' examples right next to it.

    Native-Windows app.command is a separate decision.

  5. PR body: no CI workflow runs on Windows, so the EPERM path itself is unverified. Please say that instead of "covered by CI on Windows".

Nits

  • Changeset: drop "with retries for transient locks, and a failed removal never throws". Users don't need the internals.
  • executor.test.ts:492: the comment implies Windows coverage. It checks ordering on POSIX only, because on Windows SIGTERM isn't catchable.
  • Optional: try removal at once and defer to exited only on failure. That keeps POSIX from leaking the dir when the worker exits before a slow agent does.

This was an automated review.

Replace the sync rmSync retry loop with the async rm on the session exit
promise: a removal forced to fail no longer blocks the event loop, and
the removed flag, the already-exited branch, and both listeners collapse
to one line. Skip the signal-based ordering test on Windows, where Node
terminates the child immediately for SIGTERM, and drop the config docs
note about app.command executables from this PR.

Executor tests 33/33 pass; full acp suite 36/39 with the same 3
pre-existing cli.test.ts Chromium-as-root failures as the base head.
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Review round addressed as 4dcf561.

Must fix:

  1. The sync rmSync retry loop is gone. Removal is now one async line on the session exit promise, so a forced failure can no longer freeze the worker for 9s, and a grandchild holding the directory past the direct child exit is handled the same way.
  2. The removed flag, the already-exited branch, and both listeners collapsed to that one line, exactly as suggested.
  3. The comments no longer say the child holds the directory as its working directory. spawn sets no cwd; what holds it is a process the agent started in the session directory.
  4. The docs/reference/config.mdx note about app.command executables is dropped from this PR.
  5. The PR body no longer claims CI covers Windows. It now says no CI workflow runs on Windows, so the EPERM path itself is unverified.

Nits:

  • Changeset no longer mentions retries or never-throwing internals.
  • The ordering test is skipped on Windows and its comment now says it checks POSIX only.
  • The already-exited test comment now says the removal runs from the already-settled exit promise.

Not taken: the optional try-removal-at-once. On POSIX rmdir succeeds on a directory a live process holds as its cwd, so trying at once would remove the directory while the agent is alive and break the ordering test. The leak it targets (worker exits before a slow agent) leaves only a harmless temp directory.

Verified: executor tests 33/33 pass, full acp suite 36/39 with the same 3 pre-existing cli.test.ts Chromium-as-root failures as the base head; typecheck and oxlint clean.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread packages/acp/src/session.ts Resolved
@okwasniewski
okwasniewski dismissed their stale review November 8, 2026 04:14

Superseded by new review

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-opus-5-5] automated review on behalf of @okwasniewski

Re-review of 4dcf561. All CR 2 asks are done: the async rm runs on exited, the cwd comments are fixed, the config.mdx note is gone and the body is accurate. The teardown is correct on spawn failure, on a hung agent (the 3s SIGKILL path) and on an agent that already exited. CI is green.

Must fix

  1. packages/acp/tests/executor.test.ts:496: the ordering test never waits on ACP_EXIT_DELAY_MS. The assertions after step.end() are synchronous, so the test only catches a sync rmSync. An immediate void rm(cwd, ...) still passes, and so does dropping the env var or the fixture's delay handler in scripted-agent.ts:60-65. After the existsSync check, add:
    await new Promise((resolve) => setTimeout(resolve, 500));
    expect(() => process.kill(pid, 0)).not.toThrow();
    expect(existsSync(cwd)).toBe(true);
    This fails against an immediate async rm and passes on your code. If you don't take it, drop the env var and the fixture handler, since they're dead code.

Should fix (maintainer call, not blocking)
2. session.ts:165: on POSIX, removal now waits for the agent to exit. After the last test, the worker exits through exitAfterFlush (run/worker/entry.ts:75). A real agent rarely exits that fast after SIGTERM, so each worker can leak one e2e-acp-* temp dir per run. On main, POSIX removed it right away. One option is to try rm at once and fall back to the exited path on failure, keeping the ordering test for that fallback. @okwasniewski: accept the leak or take the fallback?

Nits

  • session.ts:159-164: the comment can be two lines. "retry loop never blocks" is an internal detail.
  • executor.test.ts:503-507 describes Windows behavior in a test that runs only on POSIX.

Try removing the session directory at once, and fall back to removing it once the agent exits when that fails. On POSIX the removal usually succeeds even while the agent is still stopping, so a worker exiting right after the attempt leaves nothing behind; on Windows a process the agent started in the directory still holds it, so the removal fails there and waits for the exit. The ordering test now forces the immediate removal to fail, with a file in a read-only directory, so it keeps covering the fallback path, and waits 500ms after the step ends to catch an immediate async removal. The teardown comment is two lines now, and the test comment no longer describes Windows behavior in a POSIX-only test.

Executor tests 33/33 pass as a non-root user (the CI environment); as root the read-only directory does not block removal, so the ordering test cannot demonstrate the fallback there. Full acp suite green with E2E_CHROMIUM_SANDBOX=0 for the real-Chromium cli tests. Typecheck clean, oxlint clean.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Done in 3b6ec1c.

  1. The ordering test now waits 500ms after the step ends and asserts the agent is still alive and the directory still there, on top of the immediate check.
  2. The teardown tries rm at once and falls back to the exited path when that fails. The ordering test forces the immediate removal to fail with a file in a read-only directory, so it keeps covering the fallback path; on POSIX a live agent does not hold the directory, so without that the test would pass against the immediate path and never reach the fallback.

Nits: the session.ts comment is two lines now, and the test comment no longer describes Windows behavior in a POSIX-only test.

Two notes: if you would rather accept the leak, the fallback is one line to revert. And the ordering test leans on file permissions, so as root the read-only directory does not block removal and the test cannot demonstrate the fallback there; CI runs non-root, where it passes. Executor tests 33/33 green as a non-root user.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

View guided diff | Re-trigger cubic

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review.

Re-review of 3b6ec1c. Thanks for adding the at-once-with-fallback path. The code is correct: on Windows a failed first removal is caught and retried after the agent exits (no #1623 regression), and there are no unhandled rejections. What's left is wording plus one test.

Must fix:

  1. The wording no longer matches the behavior. On POSIX the immediate rm succeeds while the agent is still alive (checked with ACP_EXIT_DELAY_MS=1500: 200ms after end the dir is gone and the pid is alive). So "removed only after the agent exits" is false in the changeset, the PR title, and the test name at executor.test.ts:495. Suggested changeset: "Fix WORKER_CRASH (EPERM) on Windows when an ACP session ends: a failed session directory removal no longer crashes the worker and is retried after the agent exits." Rename the test to something like "retries the session directory removal after the agent exits when the first try fails". Retitle the PR to match.
  2. The immediate path is untested. Reverting to remove-only-after-exit passes all 33 tests. Add a POSIX test with ACP_EXIT_DELAY_MS set that checks the dir is gone while the pid is still alive. The test at :524 passes under either version, so it can be repurposed for this (that also lets you drop the 50ms sleep at :535).
  3. The fallback test fails as root, because the read-only dir doesn't block root. Add || process.getuid?.() === 0 to its skipIf.

Nits:

  • :511-512: the existence checks right after step.end() can't fail (the async rm hasn't run yet). Keep only the post-500ms checks.
  • :507/:517: register a chmodSync(cwd, 0o755) cleanup before step.end, so a failing test doesn't leave an undeletable dir in tmp.
  • session.ts:159: one clause on why it tries at once (the exitAfterFlush leak).

Mutation results: 6/8 killed. The survivors are remove-only-after-exit (must fix 2) and the fallback without retries (Windows-only, expected).

@okwasniewski
okwasniewski dismissed their stale review November 8, 2026 10:16

Superseded by re-review of 3b6ec1c

The re-review found the wording no longer matched the behavior: on POSIX
the immediate removal succeeds while the agent is still alive, so only
after the agent exits was false in the changeset, the PR title, and the
ordering test name. The changeset now says a failed removal is retried
after the agent exits, the ordering test is renamed to say the same, and
a new POSIX test checks the directory is gone while the agent is still
alive (fails on the remove-only-after-exit code, passes with the fix).
The fallback test now skips as root, where a read-only directory does
not block removal, and restores the directory permissions on the way
out so a failing run leaves no undeletable temp dir.

Executor tests 32 passed, 1 skipped (the fallback test, as root);
typecheck clean, oxlint clean.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali Varshith-Kali changed the title fix(acp): remove the session directory only after the agent exits fix(acp): retry a failed session directory removal after the agent exits Nov 8, 2026
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

All three must-fix items from the re-review are done in 58e41bb:

  • Wording fixed: the changeset now says a failed removal is retried after the agent exits, the ordering test is renamed to match, and the PR title is updated.
  • New POSIX test checks the directory is gone while the agent is still alive (ACP_EXIT_DELAY_MS=1500, checked 500ms after the step ends). It fails on the remove-only-after-exit code and passes with the fix.
  • The fallback test now skips as root, where a read-only directory does not block removal.

Nits done too: the post-step.end() checks are dropped, the fallback test restores the directory permissions on the way out, and the teardown comment says why it tries at once (the worker can exit before the agent does).

Executor tests 32 passed, 1 skipped (the fallback test, as root here); typecheck and oxlint clean.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

View guided diff | Re-trigger cubic

@okwasniewski
okwasniewski dismissed their stale review November 8, 2026 16:17

Automated review: CR 4 asks done in 58e41bb.

@okwasniewski

Copy link
Copy Markdown
Member

Automated review.

Re-review of 58e41bb: all CR 4 asks are done and verified. Labeling Ready for Human Review.

  • Changeset, PR title and test name now match the behavior: try at once, retry after the agent exits if that fails.
  • The new POSIX test (ACP_EXIT_DELAY_MS=1500, dir gone + pid alive after 500ms) catches remove-only-after-exit. The fallback test skips as root (checked with a faked uid 0).
  • Nits done: post-wait checks only, chmod cleanup registered before step.end, reason comment at session.ts:159.
  • acp suite 39/39, the dir tests pass 5 runs in a row, CI green. Mutation testing: remove-only-after-exit, no fallback and bare void rm are all caught. Dropping the final .catch survives because no test fails the fallback removal. That would need an injected rm failure, not blocking.

Optional: the comment at executor.test.ts:492 explains the Windows skip but not the root skip.

For maintainers: there's no Windows CI, so the #1623 path is only covered by the chmod simulation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:S Up to 29 changed lines, not counting tests, fixtures, translated docs, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

@e2e-dev/acp: WORKER_CRASH (EPERM) on Windows when the session tears down

2 participants