feat: OpenAI-compatible remote embedding, reranking, and query expansion

[rghamilton3]

Summary

Cherry-picks the 5 commits from georgelichen's PR #629 (branch georgelichen:merge-pr-517-remote-llm) onto current main, then applies Kaspre's three documented fixes from the tobi#629 comment thread.

What's new

Remote LLM backends (RemoteLLM, HybridLLM)

• RemoteLLM — HTTP client implementing the LLM interface against any OpenAI-compatible API:
  • /v1/embeddings for embedding (with batch splitting, dimension validation, index-sorting)
  • /v1/rerank for reranking
  • /v1/chat/completions for query expansion (lex:/vec:/hyde: line format)
  • Circuit breaker pattern (closed → open → half-open) per endpoint
• HybridLLM — Routes embed/embedBatch/rerank/expandQuery to remote, generate/tokenize/detokenize to local LlamaCpp
• remoteConfigFromEnv() — Reads QMD_EMBED_API_URL, QMD_EMBED_API_KEY, QMD_EMBED_API_MODEL, QMD_RERANK_API_*, QMD_EXPAND_API_* (env vars override YAML models: block)

Configuration (YAML models: block or env vars)

models:
  embed_api_url: http://vllm-host:8000/v1
  embed_api_model: text-embedding-3-small
  embed_api_key: sk-...          # optional
  rerank_api_url: http://vllm-host:8001/v1
  rerank_api_model: bge-reranker-v2-m3
  expand_api_url: http://vllm-host:8002/v1
  expand_api_model: Qwen3-4B

LLM interface additions

• embedModelName, generateModelName?, rerankModelName? readonly properties
• usesRemoteEmbedding? — skips local tokenizer preprocessing when true
• embedBatch() method
• tokenize()/detokenize() on interface (stubs throw for remote-only backends)
• getDefaultLLM() / setDefaultLLM() generalized singleton (getDefaultLlamaCpp() deprecated but kept)

Remote-aware chunking

When usesRemoteEmbedding is true, chunkDocumentByApproxTokens() uses a character-heuristic (~3 chars/token) instead of the local tokenizer.

Kaspre's fixes (PR tobi#629 comments)

Fix A — Sigmoid normalization in RemoteLLM.rerank

llama.cpp's /v1/rerank emits log-odds (~−10 to +10), not probabilities. Without normalization every score goes negative and --min-score 0.3 fires "No results found" for every query.

const sigmoid = (x: number) => 1 / (1 + Math.exp(-x));
score: sigmoid(r.relevance_score)   // normalises log-odds → [0, 1]

No-op for rerankers already emitting 0–1 values.

Fix B — RemoteLLM.expandQuery (already in the cherry-picked commits)

POSTs to <expand_api_url>/chat/completions with a system prompt eliciting lex:/vec:/hyde: line-prefixed output. Parsing and fallback mirror LocalLLM.expandQuery shape.

Fix C — Pre-flight probe in vectorIndex()

When usesRemoteEmbedding === true, embeds a single token before starting the batch loop. Config mistakes (bad URL, wrong model name, auth failure) surface immediately:

✗ Remote embedding probe failed: connect ECONNREFUSED 127.0.0.1:8000
  Verify QMD_EMBED_API_URL and QMD_EMBED_API_MODEL, then retry.

Other fixes in the Kaspre commit

• Added generateModelName? and rerankModelName? to the LLM interface (used by store.ts optional-chaining for default model fallbacks — was a TS compile error)
• Added c.red to the terminal colour map (used by Fix C's error output)
• Doctor device-probe guard: check instanceof LlamaCpp before calling getDeviceInfo() — emits "unavailable" for non-local backends
• Updated rerank unit test to send realistic log-odds mock scores and assert sigmoid-normalised output
• Fixed store rerank-dedup spy from getDefaultLlamaCpp → getDefaultLLM
• Fixed doubled async () => syntax error in one store test header

Tests

• 52 / 52 test/remote-llm.test.ts (unit tests with mock HTTP server)
• 219 / 219 test/store.test.ts
• test/remote-llm-integration.test.ts skipped unless VLLM_EMBED_URL/VLLM_EMBED_MODEL env vars are set

Files changed

File	Change
src/remote-llm.ts	New — RemoteLLM, HybridLLM, helpers
src/hybrid-llm.ts	New — routing shim
src/llm.ts	Extended LLM interface; getDefaultLLM/setDefaultLLM; remote format helpers
src/store.ts	getLlm() returns LLM; remote chunking path; chunkDocumentByApproxTokens export
src/collections.ts	ModelsConfig remote API fields
src/cli/qmd.ts	getStore() wires HybridLLM; Fix C probe; doctor guard; colour map
test/remote-llm.test.ts	New unit tests
test/remote-llm-integration.test.ts	New live integration tests
test/store.test.ts	New remote-embedding test; spy/syntax fixes
CHANGELOG.md	Entry under [Unreleased]

[Kaspre]

Hi @rghamilton3 — opened tobi#705 consolidating the tobi#629 line onto current main with the documented fixes (sigmoid / expandQuery / pre-flight probe) plus a HybridLLM.rerank local fallback and oversized-rerank recovery, targeting upstream. Since your PR did the same tobi#629 + fixes rebase, flagging it in case you'd like to converge there.

[rghamilton3]

Hi @rghamilton3 — opened tobi#705 consolidating the tobi#629 line onto current main with the documented fixes (sigmoid / expandQuery / pre-flight probe) plus a HybridLLM.rerank local fallback and oversized-rerank recovery, targeting upstream. Since your PR did the same tobi#629 + fixes rebase, flagging it in case you'd like to converge there.

Hi @Kaspre Thanks for the heads up. Since this was shamelessly stolen from your work anyways I'll follow you on this :)