fix(session-search): strip FTS5 operators from truncation query terms#18692
Open
liuhao1024 wants to merge 1 commit into
Open
fix(session-search): strip FTS5 operators from truncation query terms#18692liuhao1024 wants to merge 1 commit into
liuhao1024 wants to merge 1 commit into
Conversation
When _truncate_around_matches receives an FTS5 query with boolean
operators (AND, OR, NOT), those operators were split into individual
terms and searched for in the conversation text. Since common English
words like 'and' appear everywhere, this produced noisy match positions
and mis-centered truncation windows.
Add _strip_fts5_operators() helper that removes:
- Boolean operators (AND, OR, NOT) — case-insensitive
- NEAR(...) clauses
- Column filters (e.g. role:user)
- FTS5 special characters (+, {}, (), ^, ~, *)
- Quoted-phrase delimiters (content preserved)
The cleaned query is used for all three matching strategies (phrase,
proximity co-occurrence, individual terms).
Fixes NousResearch#4238, NousResearch#4239
alt-glitch
added
type/bug
Collaborator
Cyrene963
pushed a commit
to Cyrene963/hermes-agent
that referenced
this pull request
May 3, 2026
Community PRs applied: - NousResearch#18596: Enable secret redaction by default (SECURITY) - NousResearch#18650: Sanitize malformed tool messages + auto-recover on API 400 - NousResearch#18607: Emergency compression before max_iterations exhaustion - NousResearch#18603: Compression fallback to main model on 413 rate limit - NousResearch#18638: Pass threshold_percent on model switch - NousResearch#18663: Strip extra_content from tool_calls for strict APIs - NousResearch#18618: Forward explicit_api_key to OpenRouter - NousResearch#18632: Show cache tokens in /insights breakdown - NousResearch#18614: Add idempotency guard for patch duplicate loops - NousResearch#18600: Raise ValueError when HERMES_HOME unset in profile mode - NousResearch#18616: Allow ZWJ emoji in context files - NousResearch#18582: Reload .env on /restart - NousResearch#18547: Stabilize system prompt prefix for KV cache reuse - NousResearch#18692: Strip FTS5 operators from session search truncation terms Fix: Add order_by_last_active=True to list_sessions_rich call (pre-existing commit 142b4bf code sync)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_truncate_around_matches()treats FTS5 boolean operators (AND,OR,NOT) as content terms when splitting the query. Since common English words like "and" appear in virtually every conversation, this produces false match positions and mis-centered truncation windows.Similarly, FTS5 syntax like
NEAR(...), column filters (role:user), and special characters (+,*,^) pollute the search terms.Fix
Add
_strip_fts5_operators()helper that extracts plain content terms from an FTS5 query by removing:AND,OR,NOT) — case-insensitive, word-boundary awareNEAR(...)clausesrole:user)+,{},(),^,~,*)The cleaned query is used for all three matching strategies:
If stripping removes everything (query was purely operators), falls back to the raw query.
Tests
_strip_fts5_operators()covering all operator types_truncate_around_matches()with FTS5 queriesFixes #4238, Fixes #4239