Repository navigation
Conversation
chrisburr
marked this pull request as ready for review
July 16, 2026 22:37
baszalmstra
reviewed
Jul 17, 2026
baszalmstra
left a comment
Contributor
There was a problem hiding this comment.
I think this looks good but can you please make the comments way less verbose?
Comment on lines
+568
to
+572
| let tmp = pointer_path.with_file_name(format!("{name}.{}.tmp", std::process::id())); | ||
| if tokio::fs::write(&tmp, &bytes).await.is_err() { | ||
| return; | ||
| } | ||
| let _ = tokio::fs::rename(&tmp, pointer_path).await; |
Contributor
There was a problem hiding this comment.
We can use tempfile here and use persist.
added 5 commits
July 23, 2026 14:28
The ephemeral backend-environment prefix under `~/.cache/rattler/cache/backends-v0/` was keyed on `EphemeralEnvSpec::cache_key()`, a hash of the *input* spec that includes the resolved `exclude-newer` cutoff. A relative `exclude-newer` (e.g. `"14d"`, or a per-package `"0d"`) resolves to `now - duration` at full precision on every solve, so the key changed every invocation and a fresh ~56 MB backend prefix was installed and never reclaimed — `backends-v0` grew unbounded (one prefix per host platform per solve whenever the workspace metadata cache is cold: fresh checkout, `pixi clean`, CI job). Derive the prefix path from a fingerprint of the *resolved* binary records (`pixi_utils::EnvironmentFingerprint`, name + sha256) instead, so the cache identity is a function of the solve's output. Two solves that resolve to the same packages under different (relative) cutoffs now share one prefix — behaving like a normal solve cached in the past — while genuinely different resolutions still get distinct prefixes. `exclude_newer` influences only the solve, never the cache key; `include_unknown_timestamp` handling is untouched. This mirrors the content-addressing already used for the build-artifact cache (`compute_artifact_cache_key`). The pre-solve fast path is removed: the prefix path now depends on the resolved records, so the on-disk marker is consulted after the solve rather than before it. Backend instantiation therefore always runs a (repodata-warm) conda solve; the marker still lets a later process skip the install.
Content-addressing the backend env prefix on the resolved records moved the on-disk marker check after the solve, so every backend instantiation paid a repodata fetch + binary solve (~250 ms repodata-warm) — including paths with no install at all, like SourceBuildKey's backend-identifier resolution before an artifact-cache lookup — and reusing a prefix required the repodata gateway even offline. Restore the fast path with a second cache level: a tiny spec-keyed pointer file (<hint>-<spec hash sans cutoff>.pointer.json) in the backends dir records the EnvironmentFingerprint the spec last resolved to. When the stored cutoff equals the spec's and the fingerprint- addressed prefix carries its marker, the env is reused with no gateway query or solve. The pointer is keyed on the spec minus exclude-newer, with the cutoff stored inside and compared on read: a moving (relative) cutoff overwrites one pointer per spec instead of accreting a file per resolved cutoff, and never follows a pointer written under a different cutoff. The pointer is purely advisory — records are always read from the pointed-to prefix's own marker, so a stale or corrupt pointer costs one solve, never correctness.
Review feedback: replace the hand-rolled write-temp + rename in write_pointer with pixi_utils::atomic_write, which uses tempfile and persist. The unique temp name from tempfile also covers the concurrent writer case the pid-suffixed name handled before.
Review feedback: trim the comments added by this PR down to the essential rationale.
cargo doc -Dwarnings rejects public docs linking to the private prefix_cache_key method.
chrisburr
force-pushed
the
fix/backend-env-cache-content-address
branch
from
July 23, 2026 12:28
dd4157e to
3ae56d8
Compare
6 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR contains a fix and an optional enhancement:
The first commit is a solution by itself by using the resolved records to build the cache key. The downside of this is that it removes the fast-path from the code which can currently avoid running the solver. This is similar to refactor: lockfile-v7 #5607.
The second commit restores the fast path for non-relative values of
exclude-newerby adding a spec keyed pointer file to track the cut off used without including it in the hash. Unfortunately this doesn't allow the fast path to work with a relativeexclude-newerbut that seems to be a non-trivial thing to fix as the information is already lost by this point.Feel free to close this PR if you want to go a different direction.
Fixes #6617
How Has This Been Tested?
Tests inside the PR plus building pixi manually and ensuring it behaves as I expect.
AI Disclosure
Tools: Claude
I lost track of the prompt as I took things in a different direction than Claude's original suggestions.
Checklist: