Skip to content

Handle missing HNSW ids in filtered local query#7043

Open
Raditya0902 wants to merge 1 commit into
chroma-core:mainfrom
Raditya0902:fix-query-where-missing-id
Open

Handle missing HNSW ids in filtered local query#7043
Raditya0902 wants to merge 1 commit into
chroma-core:mainfrom
Raditya0902:fix-query-where-missing-id

Conversation

@Raditya0902
Copy link
Copy Markdown

Description of changes

  • Improvements & Bug fixes

    • Fixes a local query edge case where query(where=...) could fail with Error finding id ... when a metadata-filtered candidate exists in metadata/WAL state but is missing from the HNSW reader/index mapping after cache reload.
    • Skips IdNotFound candidates only in the metadata-filtered local KNN path.
    • Preserves unfiltered and id-only query behavior.
    • Returns an empty KNN result when all filtered candidates are missing from HNSW.
    • Adds a regression test for the cache-reload/missing-candidate case.
  • New functionality

    • None.

Fixes #7032.

Test plan

  • Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Tested with:

cargo test -p chroma-frontend test_query_where_skips_hnsw_missing_candidate_after_cache_reload
cargo test -p chroma-frontend local

@github-actions
Copy link
Copy Markdown

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@propel-code-bot
Copy link
Copy Markdown
Contributor

Fix local filtered KNN queries when metadata candidates are missing in HNSW mapping

This PR fixes a local query edge case in rust/frontend/src/executor/local.rs where query(where=...) could fail with an IdNotFound-style error if metadata-filtered IDs exist in metadata/WAL but are missing from the HNSW reader mapping (for example after cache reload/reset). The knn path now conditionally tolerates missing HNSW IDs only for metadata-filtered execution, while preserving prior behavior for unfiltered and ID-only queries.

The change introduces a skip_missing_hnsw_ids control flag tied to filter mode, updates offset-ID resolution to skip only LocalHnswSegmentReaderError::IdNotFound in that scoped path, and returns an empty KNN result when all filtered candidates are skipped. A targeted regression test reproduces the cache-reload scenario and validates non-error empty-result behavior.

This summary was automatically generated by @propel-code-bot

Copy link
Copy Markdown
Contributor

@propel-code-bot propel-code-bot Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review found no issues; the edge-case handling and regression coverage appear sound.

Status: No Issues Found | Risk: Low

Review Details

📁 1 files reviewed | 💬 0 comments

Instruction Files
├── AGENTS.md
└── CLAUDE.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query(where=...) raises 'Error finding id' after batched adds until WAL flushes — should return empty + warn instead

1 participant