Skip to content

Cache prefix info from staging environments for linking checks#2210

Open
wolfv wants to merge 4 commits intomainfrom
claude/fix-staging-cache-relinking-yEkyw
Open

Cache prefix info from staging environments for linking checks#2210
wolfv wants to merge 4 commits intomainfrom
claude/fix-staging-cache-relinking-yEkyw

Conversation

@wolfv
Copy link
Copy Markdown
Member

@wolfv wolfv commented Feb 24, 2026

Summary

This PR enables linking checks to correctly attribute shared libraries to packages from inherited staging caches by caching and merging prefix information (path-to-package and package nature mappings) from the staging cache's host environment.

Problem

When an output inherits from a staging cache, the cache's host dependencies are installed into the prefix but their conda-meta records are not present. This causes linking checks to fail when trying to attribute shared libraries to packages that were built in the staging cache, since the package metadata is unavailable.

Solution

The changes implement a caching mechanism for prefix information:

  • New CachedPrefixInfo struct: A serializable representation of PrefixInfo that stores path-to-package and package nature mappings as strings, suitable for storage in staging cache metadata
  • Capture at cache build time: When building a staging cache, the prefix info is captured from the host environment (while conda-meta records are present) and stored in the cache metadata
  • Merge during post-processing: When performing linking checks on an output that inherits from a staging cache, the cached prefix info is merged into the current prefix info, allowing the checks to attribute libraries to their providing packages
  • Serialization support: Added Clone, Serialize, and Deserialize derives to PackageNature to support serialization

Key Changes

  • Added CachedPrefixInfo struct with serialization support for storing prefix mappings
  • Implemented PrefixInfo::merge_cached() to merge cached mappings into the current prefix info
  • Updated StagingCacheMetadata to include cached_prefix_info field
  • Modified staging cache build/restore functions to return CachedPrefixInfo alongside dependencies and sources
  • Updated BuildOutput to store cached_prefix_info from inherited staging caches
  • Modified linking checks to merge cached prefix info before performing attribution
  • Updated all call sites to handle the new return type

Implementation Details

  • The CachedPrefixInfo uses String keys instead of PackageName to avoid deserialization failures if package name formats change
  • Cached entries are only added if they don't already exist (via entry().or_insert())
  • The cached prefix info is marked with #[serde(default)] for backward compatibility with existing cache metadata

https://claude.ai/code/session_014mGW5MYfcfu8NFWxUJbU1J

claude and others added 3 commits February 25, 2026 11:31
@claude @wolfv
fix: install staging cache host deps for correct linking checks
12e46b5 
When a package output inherits from a staging cache, the cache's host
dependencies (e.g. zlib, libiconv) were not installed in the host
prefix. This caused the linking checks to fail to attribute shared
libraries to their providing packages, showing them as "not found"
(red/unmarked) instead of properly linked.

The root cause: `install_environments` only installed the output's own
host dependencies, but the staging cache's host packages (which
provided the libraries the build artifacts link against) were absent.
Without their conda-meta records and library files in the prefix,
`PrefixInfo::from_prefix()` couldn't map DSOs to packages.

The fix merges the staging cache's host resolved packages into the
output's host environment before installation, ensuring all relevant
packages are present for post-processing (linking checks, relinking).

Closes #2186

https://claude.ai/code/session_014mGW5MYfcfu8NFWxUJbU1J
@claude @wolfv
fix: cache prefix info in staging metadata instead of reinstalling ho…
697048c 
…st packages

Instead of reinstalling the staging cache's host packages during
install_environments (which is slow), cache the PrefixInfo data
(path-to-package and package nature mappings) in the staging cache
metadata. When linking checks run, merge this cached info into the
current prefix's PrefixInfo so shared libraries from the staging
cache's host packages can be correctly attributed.

This avoids the performance cost of re-downloading and installing
packages while still providing the linking check with the information
it needs.

https://claude.ai/code/session_014mGW5MYfcfu8NFWxUJbU1J
@wolfv
more fixes
00f8fe0
@wolfv wolfv force-pushed the claude/fix-staging-cache-relinking-yEkyw branch from c5bd2c8 to 00f8fe0 Compare February 25, 2026 12:08
@wolfv
Copy link
Copy Markdown
Member Author

wolfv commented Feb 25, 2026

This doesn't work well yet.

@carterbox carterbox mentioned this pull request Mar 16, 2026
Open
Hofer-Julian added a commit to Hofer-Julian/rattler-build that referenced this pull request Apr 2, 2026
@Hofer-Julian
fix: resolve overlinking false positives for staging outputs
4ea4b07 
When a package output inherits from a staging cache, the staging
cache's host dependencies aren't installed in the prefix during
overlinking checks. This means libraries like libz.so.1 can't be
attributed to zlib, even though zlib is a run dependency via inherited
run_exports.

Fix: at staging build time, capture a LibraryNameMap (filename to
package mapping) from PrefixInfo while conda-meta still exists. Store
it in the staging cache metadata, thread it to BuildOutput, and use it
as a fallback in perform_linking_checks when resolve_libraries can't
find the library on disk.

Fixes prefix-dev#2186

Supersedes prefix-dev#2210
@Hofer-Julian Hofer-Julian mentioned this pull request Apr 2, 2026
Open
Hofer-Julian added a commit to Hofer-Julian/rattler-build that referenced this pull request Apr 7, 2026
@Hofer-Julian
fix: resolve overlinking false positives for staging outputs
84721eb 
When a package output inherits from a staging cache, the staging
cache's host dependencies aren't installed in the prefix during
overlinking checks. This means libraries like libz.so.1 can't be
attributed to zlib, even though zlib is a run dependency via inherited
run_exports.

Fix: at staging build time, capture a LibraryNameMap (filename to
package mapping) from PrefixInfo while conda-meta still exists. Store
it in the staging cache metadata, thread it to BuildOutput, and use it
as a fallback in perform_linking_checks when resolve_libraries can't
find the library on disk.

Fixes prefix-dev#2186

Supersedes prefix-dev#2210
Hofer-Julian added a commit to Hofer-Julian/rattler-build that referenced this pull request Apr 8, 2026
@Hofer-Julian
fix: resolve overlinking false positives for staging outputs
da570ec 
When a package output inherits from a staging cache, the staging
cache's host dependencies aren't installed in the prefix during
overlinking checks. This means libraries like libz.so.1 can't be
attributed to zlib, even though zlib is a run dependency via inherited
run_exports.

Fix: at staging build time, capture a LibraryNameMap (filename to
package mapping) from PrefixInfo while conda-meta still exists. Store
it in the staging cache metadata, thread it to BuildOutput, and use it
as a fallback in perform_linking_checks when resolve_libraries can't
find the library on disk.

Fixes prefix-dev#2186

Supersedes prefix-dev#2210
Hofer-Julian added a commit to Hofer-Julian/rattler-build that referenced this pull request Apr 8, 2026
@Hofer-Julian
fix: resolve overlinking false positives for staging outputs
1461b12 
When a package output inherits from a staging cache, the staging
cache's host dependencies aren't installed in the prefix during
overlinking checks. This means libraries like libz.so.1 can't be
attributed to zlib, even though zlib is a run dependency via inherited
run_exports.

Fix: at staging build time, capture a LibraryNameMap (filename to
package mapping) from PrefixInfo while conda-meta still exists. Store
it in the staging cache metadata, thread it to BuildOutput, and use it
as a fallback in perform_linking_checks when resolve_libraries can't
find the library on disk.

Fixes prefix-dev#2186

Supersedes prefix-dev#2210


test: add e2e test for staging overlinking check
Hofer-Julian added a commit to Hofer-Julian/rattler-build that referenced this pull request Apr 8, 2026
@Hofer-Julian
fix: resolve overlinking false positives for staging outputs
654a1a4 
When a package output inherits from a staging cache, the staging
cache's host dependencies aren't installed in the prefix during
overlinking checks. This means libraries like libz.so.1 can't be
attributed to zlib, even though zlib is a run dependency via inherited
run_exports.

Fix: at staging build time, capture a LibraryNameMap (filename to
package mapping) from PrefixInfo while conda-meta still exists. Store
it in the staging cache metadata, thread it to BuildOutput, and use it
as a fallback in perform_linking_checks when resolve_libraries can't
find the library on disk.

Fixes prefix-dev#2186

Supersedes prefix-dev#2210


test: add e2e test for staging overlinking check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wolfv @claude