perf: Use stack-allocated OidHash in FileHashes and skip expanded hashes on normal runs#12065
Merged
anthonyshew merged 1 commit intoFeb 28, 2026
Conversation
…hes on normal runs Two memory optimizations for large monorepos: 1. FileHashes now stores OidHash (40 bytes, stack, Copy) instead of heap-allocated String (~80-96 bytes each). OidHash moved from turborepo-scm to turborepo-hash where it belongs architecturally. The OidHash->String conversion only happens at the JSON serialization boundary for --summarize/--dry output. 2. calculate_file_hashes takes a needs_expanded_hashes flag. When false (normal turbo run), the per-file hash maps are computed for the collapsed task hash but not retained in the TaskHashTracker. This skips O(tasks * files) of memory that previously lived for the entire run duration but was never read.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Coverage Report
|
anthonyshew
deleted the
anthonyshew/oidhash-filehashes-conditional-expanded
branch
github-actions Bot
added a commit
that referenced
this pull request
Mar 2, 2026
## Release v2.8.13-canary.9 Versioned docs: https://v2-8-13-canary-9.turborepo.dev ### Changes - fix: Treat `npm: alias` dependencies as external, not workspace references (#12061) (`b179cb8`) - test: Port 18 more prysk tests to Rust (other/ + lockfile-aware-caching/) (#12062) (`7887af2`) - release(turborepo): 2.8.13-canary.8 (#12063) (`2a5522a`) - fix: Preserve file: protocol entries in pruned yarn v1 lockfile (#12064) (`ae5c1a1`) - perf: Use stack-allocated OidHash in FileHashes and skip expanded hashes on normal runs (#12065) (`677b248`) - test: Port all 8 find-turbo prysk tests to Rust (#12066) (`f827fca`) - fix: Support pnpm per-workspace lockfiles in turbo prune (#12067) (`23d047d`) - test: Port final 2 prysk tests to Rust (100% complete) (#12068) (`6d7e057`) - fix: Resolve Berry prune failure when resolutions contain patch overrides (#12069) (`6fe3c5e`) - test: Add lockfile fixture for yarn berry resolution pruning (issue #2791) (#12071) (`6cc1654`) - chore: Remove prysk test framework entirely (#12070) (`ed2d05a`) - refactor: Clean up test infrastructure and eliminate duplication (#12072) (`338911d`) - fix: Retain injected workspace package entries during pnpm lockfile pruning (#12073) (`acbe869`) - ci: Exclude turborepo-lsp and turborepo-schema-gen from test builds (#12075) (`4ce12e2`) - refactor: Clean up test infrastructure + improve test quality (#12074) (`4571f2b`) - ci: Remove redundant cargo build from coverage job (#12077) (`3c9bbe2`) - perf: Speed up lockfile test suite (#12078) (`20024df`) - ci: Remove integration test serialization (#12079) (`24d7c02`) - fix: Preserve `file:` and `link:` protocol entries in pruned bun lockfile (#12076) (`2635d9a`) - fix: Stop running unnecessary npm install in engines tests (#12081) (`24e4905`) - test: Add lockfile fixture for pnpm v9 injected workspace deps (issue #8243) (#12082) (`4d4929b`) - fix: Filter orphaned Yarn packageExtensions entries during lockfile pruning (#12084) (`68eb223`) - fix: Align experimentalObservability on object maps rather than arrays (#12089) (`9b9d1e4`) - examples: Upgrade with-react-native-web example to use latest versions (#12085) (`980ca43`) - fix: duplicate /signup? in Vercel URL (#12088) (`e865b51`) - ci: Deduplicate Rust test compilation with nextest archive (#12083) (`962cf39`) - fix: Prevent yarn integration tests from hanging on corepack prompts (#12090) (`29b0da7`) - fix: Prevent turbo dev from hanging when daemon file watching fails (#12091) (`b0d2f62`) - ci: Skip pnpm install for Rust test jobs (#12092) (`ebd137f`) - perf: Optimize npm lockfile parser (#12093) (`e4b4a66`) - chore: Trim unused dependency features for faster compilation (#12094) (`03b79e0`) - fix: Prevent lockfile-aware yarn test from hanging on corepack downloads (#12095) (`bf516e4`) - fix: Exclude turborepo-repository from JS smoke test in release workflow (#12097) (`fecc400`) --------- Co-authored-by: Turbobot <turbobot@vercel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two memory optimizations targeting large monorepos where per-file hash storage is a significant memory consumer.
FileHashesnow storesOidHashinstead ofString— eliminates one heap allocation (~80-96 bytes) per file across the entire repo.OidHashis a 40-byte stack-allocatedCopytype. TheOidHash→Stringconversion now only happens at theexpanded_inputstrait boundary, which only runs for--summarize/--dryJSON output.Expanded hashes are conditional —
calculate_file_hashestakes aneeds_expanded_hashesflag. On normalturbo run(no--dry,--summarize, or observability), the per-fileArc<FileHashes>maps are computed for the collapsed task hash but not retained inTaskHashTracker. This skipsO(tasks × files)of path+hash data that previously lived for the entire run.What changed
crates/turborepo-hash/src/oid_hash.rs(new) —OidHashmoved fromturborepo-scmtoturborepo-hash. It's a pure hash-value type, architecturally belonging withFileHashesandTurboHash.crates/turborepo-hash/src/lib.rs—FileHashesinner type changed fromStringtoOidHash. Capnp serialization uses&**value(deref to&str).crates/turborepo-scm/src/lib.rs— RemovedOidHashdefinition, re-exports fromturborepo_hash.crates/turborepo-task-hash/src/lib.rs— Removed threeString::from(v)conversions when constructingFileHashes. Addedneeds_expanded_hashesparameter tocalculate_file_hashes. AddedOidHash→Stringconversion inexpanded_inputstrait impl (the single JSON boundary point).crates/turborepo-run-cache/src/lib.rs— RemovedString::from(v)conversion.crates/turborepo-lib/src/run/mod.rs— Passesneeds_expanded_hashesflag computed fromdry_run,summarize, andobservability_handle.Testing
test_expanded_inputs_none_when_not_collected— verifies that whenTaskHashTrackeris constructed with an empty expanded hashes map (simulatingneeds_expanded_hashes=false),get_expanded_inputsreturnsNonerather than panicking, even though the collapsed task hash exists.FileHashestest values to use valid 40-char hex strings forOidHash.