fix: Resolve turbo watch hang with mixed interruptible persistent tasks#12449
Merged
anthonyshew merged 1 commit intomainfrom Mar 26, 2026
Merged
fix: Resolve turbo watch hang with mixed interruptible persistent tasks#12449anthonyshew merged 1 commit intomainfrom
anthonyshew merged 1 commit intomainfrom
Conversation
In watch mode, persistent tasks are now fire-and-forget in the visitor: the engine callback is sent immediately and the executor runs as a detached background task. This eliminates the two-run split and oneshot gate that caused the hang when interruptible persistent tasks blocked the gate from ever firing. Fixes #12433
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions Bot
added a commit
that referenced
this pull request
Mar 26, 2026
## Release v2.8.21-canary.12 Versioned docs: https://v2-8-21-canary-12.turborepo.dev ### Changes - release(turborepo): 2.8.21-canary.11 (#12450) (`b14aa0b`) - fix: Resolve turbo watch hang with mixed interruptible persistent tasks (#12449) (`326532d`) Co-authored-by: Turbobot <turbobot@vercel.com>
github-actions Bot
added a commit
that referenced
this pull request
Mar 28, 2026
## Release v2.8.21 Versioned docs: https://v2-8-21.turborepo.dev ### Changes - release(turborepo): 2.8.20 (#12396) (`45230ec`) - fix: Disable husky hooks in `update-examples` workflow (#12397) (`56b79ff`) - docs: Add link to Docker guide in prune --docker flag section (#12401) (`e7f0db7`) - feat: Add `global` configuration key behind `futureFlags.globalConfiguration` (#12399) (`5f190cf`) - chore: Update CODEOWNERS to remove /docs owner (#12402) (`3233d3a`) - fix: Strip JSX components from heading anchors and TOC entries (#12404) (`3abe553`) - fix: Move docs app icons into app/ directory (#12403) (`ddf3918`) - feat: Add experimental structured logging with `--json` and `--log-file` flags (#12405) (`7ca0601`) - release(turborepo): 2.8.21-canary.1 (#12407) (`adebb95`) - docs: Downgrade Next.js (#12408) (`281e89b`) - chore: Deprecate the `turbo scan` command (#12406) (`4a12c26`) - release(turborepo): 2.8.21-canary.2 (#12409) (`b9ef212`) - fix(eslint-plugin-turbo): Guard against missing tasks/pipeline in forEachTaskDef (#12411) (`6c107c2`) - release(turborepo): 2.8.21-canary.3 (#12413) (`a2e6635`) - chore: Upgrade Next.js (#12415) (`b9e6174`) - Revert "fix: Flush stale mouse tracking events from stdin during TUI cleanup" (#12416) (`646b06e`) - fix: Add NixOS environment variables to default passthroughs (#12417) (`4f12c69`) - release(turborepo): 2.8.21-canary.4 (#12419) (`19cb539`) - fix: Resolve security vulnerabilities in `tar` and `rustls-webpki` (#12418) (`f09b138`) - release(turborepo): 2.8.21-canary.5 (#12420) (`8aca047`) - docs: Promote `turbo query` from experimental to stable (#12421) (`0692aba`) - docs: Clarify `turbo-ignore`'s future (#12422) (`c5a8235`) - release(turborepo): 2.8.21-canary.6 (#12423) (`3ebf536`) - feat: Rework turbo ls to use query internals and add turbo query ls shorthand (#12424) (`84fd6e3`) - docs: Clarify environment variables across packages dependency behavior (#12390) (`e44b0d8`) - docs: Expand subpath imports example (#12412) (`a7fec57`) - fix(examples): Update of `with-svelte` example (#11952) (`41d1b2e`) - release(turborepo): 2.8.21-canary.7 (#12425) (`7155a67`) - fix: Preserve source dependencies when adding workspace deps in `turbo-gen` (#11935) (`01c56cc`) - docs: Add Git history requirements to `turbo query affected` docs (#12426) (`edc16d5`) - fix: Prevent horizontal overflow from long inline code on narrow viewports (#12428) (`a5d641b`) - release(turborepo): 2.8.21-canary.8 (#12429) (`46814d0`) - feat: Send git SHA and dirty hash to remote cache (#12427) (`192034a`) - fix: Upgrade tokio to 1.47.1+ to fix pidfd_reaper panic (#12431) (`8c25d47`) - release(turborepo): 2.8.21-canary.9 (#12432) (`2e2f8c3`) - fix: Use script-shell=bash for cross-platform with-shell-commands example (#12436) (`d5c2192`) - docs: Add AI guide to sidebar navigation (#12438) (`021d288`) - docs: Move `experimentalObservability` into `futureFlags` section (#12439) (`85812cc`) - fix: Skip Unix domain sockets and other special files during file hashing (#12445) (`eb8f75e`) - fix: Preserve dedupePeers and unknown pnpm lockfile settings (#12443) (`1529b92`) - release(turborepo): 2.8.21-canary.10 (#12446) (`014111c`) - fix: Align dry run cache status with normal run by checking caching guards (#12448) (`48aa171`) - release(turborepo): 2.8.21-canary.11 (#12450) (`b14aa0b`) - fix: Resolve turbo watch hang with mixed interruptible persistent tasks (#12449) (`326532d`) - release(turborepo): 2.8.21-canary.12 (#12451) (`379d47b`) - fix: Avoid `setsid()` in PTY spawn to prevent macOS Gatekeeper CPU spikes (#12452) (`dcc9f6a`) - release(turborepo): 2.8.21-canary.13 (#12453) (`19f46e6`) - feat: Add `packagesFromLockfile()` NAPI binding to `@turbo/repository` (#12454) (`c58ee79`) - release(library): 0.0.1-canary.21 (#12455) (`3637185`) - release(turborepo): 2.8.21-canary.14 (#12456) (`3f87769`) - refactor: Move cache hit SHA context to verbose logging (#12435) (`23c15b4`) - release(turborepo): 2.8.21-canary.15 (#12457) (`6353482`) - docs: Add missing --force flag documentation (#12440) (`e3b89b0`) - fix: Prevent panic in turbo watch with persistent tasks (#12459) (`337b2e8`) - release(turborepo): 2.8.21-canary.16 (#12461) (`e79a56b`) - fix: Support `turbo watch` in single-package workspaces (#12460) (`ae78ce1`) - release(turborepo): 2.8.21-canary.17 (#12463) (`0bafae2`) - fix: Missing deps after npm lockfile parsing (#12464) (`fe5a86e`) - release(turborepo): 2.8.21-canary.18 (#12465) (`c014134`) - docs: Add AI agent detection and automatic markdown rewrites (#12462) (`50bd872`) - fix: Resolve generator name conflicts across workspaces (#12467) (`d5d37a8`) - release(turborepo): 2.8.21-canary.19 (#12468) (`7552e93`) - fix: Remove root package.json from `--affected` global triggers (#12469) (`91ebb97`) - release(turborepo): 2.8.21-canary.20 (#12470) (`c5a4690`) - fix: Show run summary after TUI exits (#12471) (`ffa47d1`) --------- Co-authored-by: Turbobot <turbobot@vercel.com>
anthonyshew
added a commit
that referenced
this pull request
Mar 31, 2026
…file changes The fire-and-forget change (PR #12449) moved the TaskTracker into a detached tokio::spawn for persistent tasks. Since persistent processes run forever, the tracker's sender clone was never dropped, causing ExecutionTracker::finish() to block indefinitely. This prevented Run::run() from returning, which starved the watch loop of the ability to process any subsequent file-change events. Fix: use a no-op TaskTracker (with a throwaway channel) for fire-and- forget persistent tasks so the real execution tracker can complete. Also adds debug! traces at key decision points in the watch loop to make future watch-mode issues easier to diagnose. Closes #12505
anthonyshew
added a commit
that referenced
this pull request
Mar 31, 2026
…n file changes (#12509) ## Summary Fixes #12505 - **Cause**: The fire-and-forget change (PR #12449) moved a `TaskTracker` into a detached `tokio::spawn` for persistent tasks. Since persistent processes run forever, the tracker's `mpsc::Sender` clone is never dropped, which blocks `ExecutionTracker::finish()` indefinitely. This prevents `Run::run()` from returning, starving the watch loop of the ability to process any subsequent file-change events. - **Fix**: Use a no-op `TaskTracker` (throwaway channel) for fire-and-forget persistent tasks so the real execution tracker can complete. - **Observability**: Adds `debug!` traces at key decision points in the watch loop (event receipt, run processing, task stopping, run completion) to make future watch-mode issues diagnosable with `-vv`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #12433
turbo watchhangs on startup when the task graph contains bothinterruptible: trueand non-interruptible persistent tasks. Only the first interruptible persistent task starts; everything else is blocked forever.Root cause: The watch coordinator split tasks into an "interruptible run" (non-persistent + persistent-interruptible) and a "non-interruptible run" (persistent-non-interruptible), gated behind a oneshot channel. Because the interruptible run contained persistent tasks that never complete, the gate never fired.
Fix: In watch mode, persistent tasks are now fire-and-forget in the visitor — the engine callback is sent immediately (so dependency ordering continues) and the executor runs as a detached background task. Child processes remain tracked by the
ProcessManagerfor laterstop_tasks()calls. This eliminates the two-run split and oneshot gate entirely.Changes
visitor/mod.rs: For persistent tasks in watch mode, send the callback immediately and spawn the executor as a detached background taskwatch.rs: Removepersistent_tasks_handle, the two-run split, and the oneshot gate. Addbackground_stoppersto track PMs from completed runs. Updatestop_impacted_tasksto filter out non-interruptible persistent tasksrun/mod.rs: Removecreate_run_for_interruptible_tasks,create_run_for_non_interruptible_tasks,has_non_interruptible_tasksengine/lib.rs: Removecreate_engine_for_interruptible_tasks,create_engine_for_non_interruptible_taskswatch_mixed_persistent_testfixture +watch_mixed_persistent_tasks_all_starttestTesting
To reproduce the original bug locally, create a monorepo with two packages where one has
persistent: true, interruptible: trueand the other haspersistent: true(non-interruptible), then runturbo watch dev. Before this fix, only one app starts. After, both start.All 12 watch integration tests pass, including the new regression test.