fix(prompt): allow ZWJ emoji in context files#18616
Open
liuhao1024 wants to merge 1 commit into
Open
Conversation
U+200D (Zero Width Joiner) is the standard Unicode mechanism for gendered and compound emoji sequences. The invisible-char scanner blocked entire context files (SOUL.md, AGENTS.md, etc.) when any ZWJ was present, silently censoring legitimate emoji. Remove U+200D from _CONTEXT_INVISIBLE_CHARS while keeping all other truly suspicious invisible characters (zero-width space, directional overrides, BOM). Add regression tests for both the allowed ZWJ path and the continued blocking of the remaining invisible characters. Closes NousResearch#18581
alt-glitch
added
type/bug
Cyrene963
pushed a commit
to Cyrene963/hermes-agent
that referenced
this pull request
May 3, 2026
Community PRs applied: - NousResearch#18596: Enable secret redaction by default (SECURITY) - NousResearch#18650: Sanitize malformed tool messages + auto-recover on API 400 - NousResearch#18607: Emergency compression before max_iterations exhaustion - NousResearch#18603: Compression fallback to main model on 413 rate limit - NousResearch#18638: Pass threshold_percent on model switch - NousResearch#18663: Strip extra_content from tool_calls for strict APIs - NousResearch#18618: Forward explicit_api_key to OpenRouter - NousResearch#18632: Show cache tokens in /insights breakdown - NousResearch#18614: Add idempotency guard for patch duplicate loops - NousResearch#18600: Raise ValueError when HERMES_HOME unset in profile mode - NousResearch#18616: Allow ZWJ emoji in context files - NousResearch#18582: Reload .env on /restart - NousResearch#18547: Stabilize system prompt prefix for KV cache reuse - NousResearch#18692: Strip FTS5 operators from session search truncation terms Fix: Add order_by_last_active=True to list_sessions_rich call (pre-existing commit 142b4bf code sync)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_CONTEXT_INVISIBLE_CHARSso that standard Unicode emoji sequences no longer silently block entire context filesProblem
_CONTEXT_INVISIBLE_CHARSinagent/prompt_builder.pyincluded\u200d(ZWJ), which is the standard Unicode mechanism for gendered and compound emoji (e.g.U+1F938 U+200D U+2640 U+FE0Ffor woman cartwheeling). When any context file (SOUL.md, AGENTS.md, HERMES.md, etc.) contained a ZWJ emoji,_scan_context_content()replaced the entire file with a[BLOCKED: ...]message, silently losing all context.Fix
Remove
\u200dfrom the set. The remaining characters are all legitimately suspicious in context files:\u200b\u200c\u2060\ufeff\u202a–\u202eZWJ (
\u200d) is different: it is the Unicode Consortium's standard mechanism for joining emoji into sequences. Blocking it censors legitimate user content.Tests
test_zwj_emoji_allowed_in_context: ZWJ emoji in SOUL.md passes through unchangedtest_other_invisible_chars_still_blocked: All other invisible characters remain blockedCloses #18581