Feature: Hybrid Tool Pre-Selection (Semantic + Keyword) — RAG-style schema injection to reduce token overhead without extra LLM round trips

[jack2684]

Problem

Every API call injects full tool schemas for ALL enabled tools (~14,000 tokens measured on a default hermes-cli setup with 30+ tools), regardless of whether those tools are relevant to the current message. Existing proposal #6839 addresses this with lazy loading (two-pass: name list first, full schema on model request), but it adds an extra LLM round trip every time a tool is needed.

Proposed Solution: Hybrid Tool Pre-Selection (Semantic + Keyword)

Before sending the user message to the LLM, run a fast hybrid search against a precomputed tool index to select only the top-K relevant tool schemas. Inject only those schemas into the prompt — single LLM call, no extra round trip.

Pure semantic search alone has a blind spot: if the user says "use browser_navigate" or "run terminal command", embeddings may rank other tools higher due to paraphrasing. Pure keyword search misses intent: "check what's running" won't match process on keywords alone. Hybrid catches both.

Flow

1. User sends message
2. Hermes runs hybrid search against precomputed tool index:
   • Keyword (BM25): exact tool name mentions, parameter names, direct intent words — pure Python, <1ms
   • Semantic (embeddings): fuzzy intent, synonyms, natural language — ~50ms embed query
   • Score fusion via RRF: final_score = 1/(k + semantic_rank) + 1/(k + keyword_rank)
3. Top-K tools selected (e.g. K=8), plus a small fixed set of always-included core tools
4. Only those schemas injected into the system prompt
5. Single LLM call as normal — no extra round trip

Index Storage

• Two lightweight indexes built from the same tool name + description corpus
• Keyword: inverted index (BM25 via rank-bm25, ~500 lines, no model needed)
• Semantic: precomputed vectors in ~/.hermes/tool_embeddings.npz (~77KB for 50 tools × 384 dims)
• Both built once on startup, loaded into memory in milliseconds
• Re-indexed automatically on checksum mismatch (tool names+descriptions change)
• Re-index triggers: hermes tools enable/disable, MCP server added/removed, Hermes update

Implementation Sketch

# At startup: build/load both indexes
bm25_index, tool_vectors = load_or_build_indexes()  # rebuilds on checksum mismatch

# Per turn: hybrid retrieval
keyword_ranks = bm25_index.rank(user_message)
q = embed(user_message)                              # <50ms
semantic_ranks = np.argsort(np.dot(tool_vectors, q))[::-1]

# Reciprocal Rank Fusion
k = 60  # RRF constant
scores = defaultdict(float)
for rank, name in enumerate(keyword_ranks):
    scores[name] += 1 / (k + rank)
for rank, name in enumerate(semantic_ranks):
    scores[name] += 1 / (k + rank)

top_k = sorted(scores, key=scores.get, reverse=True)[:K]
schemas = [registry.get_schema(n) for n in top_k]
# inject schemas into prompt as normal

Config

tools:
  selection: eager      # current default — all tools always injected
  # selection: hybrid   # semantic + keyword fusion (recommended)
  # selection: semantic # embedding only
  # selection: keyword  # BM25 only
  # rag_top_k: 8
  # rag_embed_model: nomic-embed-text  # or "auxiliary" to reuse existing provider

Comparison with Existing Proposals

	Current	#6839 Lazy Loading	This proposal (Hybrid)
Schema tokens/call	~14,000	~500 + full schema on request	~1,400 (top-8 schemas)
Extra LLM round trip	0	+1 per tool use	0
Latency penalty	0	~1-2s per tool use	~50ms embed + <1ms BM25
Token savings	0%	~70% schema, but extra call cost	~90% schema, no extra call
Handles exact tool name	yes	yes	yes (keyword leg)
Handles fuzzy intent	n/a	n/a	yes (semantic leg)

The two approaches are also composable: hybrid pre-selects likely tools, lazy loading (#6839) handles edge cases where the model needs a tool outside the top-K.

Trade-offs

• Pro: ~90% schema token reduction with zero latency penalty
• Pro: No change to agent loop structure
• Pro: No external DB — BM25 inverted index + numpy vectors, both tiny
• Pro: Automatic re-indexing on tool set changes
• Pro: Hybrid beats pure semantic or pure keyword on both exact and fuzzy queries
• Con: Requires an embedding model (can reuse auxiliary provider already in Hermes)
• Con: Risk of missing a needed tool if K is too small (mitigated by always including a pinned core set: terminal, read_file, search_files)

Related

• Feature: Lazy Tool Schema Loading — Two-Pass Tool Injection to Reduce Token Overhead #6839 — Lazy Tool Schema Loading (two-pass, extra round trip)
• [Feature]: Lean default tool exposure profile + lazy non-core discovery for faster first tool-enabled turn #11115 — Lean default tool exposure profile

[yes999zc]

This is the right approach — a tool index (like persistent memory) is more elegant than the two-pass lazy loading in #6839. No extra API round trip, model doesn't need to know about tool selection at all.

Combined with #16608 (model allowlist), this would give Hermes a proper resource management layer:

• Tool index → only inject relevant schemas
• Model allowlist → only show configured providers
• Both reduce waste: tokens and UI noise.

These three issues (#13332, #6839, #16608) are fundamentally about the same thing: Hermes should stop dumping everything into every call.

[alias8818]

Built a Hermes plugin that takes the provider-agnostic approach described here: alias8818/hermes-tool-slimmer

Local BM25 ranking selects the smallest useful tool subset per turn. Four modes: eager (no filtering), keyword (BM25, shipped), hybrid (embedding, placeholder), and anthropic_tool_search (defers to Anthropic native Tool Search when available). In keyword mode with top_k=8, a 57-tool catalog (~18K tokens) drops to ~3.7K — roughly 80% reduction.

Fails open on selector errors. always_include keeps safety-critical tools hot. No monkeypatching — if the Hermes core selector hook is not available yet, the plugin degrades to benchmarking and diagnostics mode. Includes a minimal upstreamable patch for the hook contract.

MIT, v0.1.0. Feedback welcome.

[sherman-yang]

Concrete impact: TPM-capped providers become unreachable, not optimization-only

Adding measured data to support prioritization. A standard multi-MCP setup (open-websearch, crawl4ai, scrapling, playwright, yfinance, reddit, tinyfish, plus built-in toolsets) registers 134 tool schemas = ~210 KB request body = ~70,000 input tokens per turn (captured via HERMES_DUMP_REQUESTS=1).

Per-MCP token cost:

MCP server	tools	input tokens
scrapling	14	17,160
(built-in)	18	12,136
tinyfish	19	10,981
yfinance	36	6,252
playwright	23	5,442
crawl4ai	8	1,609
reddit	10	1,589
open-websearch	6	1,110

This places the request above the per-minute token cap of several attractive free / fast providers:

Provider	TPM	70K request
Cerebras gpt-oss-120b (free)	64 K	❌
Cerebras qwen3-235b-a22b-instruct (free)	60 K	❌
Groq llama-3.3-70b-versatile	12 K	❌
Groq llama-4-scout	30 K	❌
OpenRouter :free / NVIDIA NIM / Gemini AI Studio	(no advertised TPM)	✅

Cerebras gpt-oss-120b (Artificial Analysis Intelligence Index 33, ~2,000 tok/s) is one of the strongest free models for agent workflows and is entirely unusable under any standard multi-MCP Hermes config — no retry/backoff helps because the request shape itself exceeds TPM.

Hybrid pre-selection (this issue) is the cleanest fix because it stays single-call: top-K relevant schemas only. With K=8 and avg ~1500 tokens/schema, request drops from ~70K to ~12K — fits even Groq's 12K TPM caps.

Today's workarounds are insufficient — hermes profile create lite with a subset works but requires manual switching; -t TOOLSETS only filters built-in toolsets, not MCP servers; enabled: false per MCP is global. None allow a single profile to use full MCP set on permissive providers and a subset on TPM-tight providers within the same workflow.

Priority note: this isn't only "wasted tokens" — it gates entire classes of providers from being reachable at all, which arguably warrants priority above P3.

(Cross-ref #6839 for the two-pass alternative.)

[alex-pathcourse]

You're sending full tool schemas on every call, even when most aren't relevant to the current message. PathCourse Health's inference layer pre-selects tools using hybrid semantic+keyword matching so you don't have to manage schema injection or extra LLM round trips.

Working example: https://github.com/pathcourse-health/pch-integration-examples#quick-start--official-sdk