feat: Add OpenAI embedding, query expansion, and re-ranking support

[jonesj38]

Summary

Optional OpenAI integration for embeddings and query expansion. Dramatically faster for users who prefer API-based inference over local models.

Performance

Operation	Local (llama-cpp)	OpenAI
Query expansion	30-40s	200ms
Full re-embed (30k chunks)	~2 hours	~10 min
Tokenizer load	30s	0s
Search latency	30-60s	3-5s
Reranking (30 docs)	10-15s	1-2s

Features

• OpenAI Embeddings — text-embedding-3-small (1536 dims), native batch API, ~$0.02/1M tokens
• OpenAI Query Expansion — gpt-4o-mini for lex/vec/hyde variants
• OpenAI Reranking — API-based reranking replaces local qwen3-reranker, eliminating model download and GGUF inference overhead
• Tiktoken chunking — eliminates model load time for tokenization
• Robust retry logic — exponential backoff with jitter for rate limits
Usage

export OPENAI_API_KEY="sk-..." export QMD_OPENAI=1 qmd embed -f # Re-embed with OpenAI qmd search "query"

Design

• Opt-in — local models remain the default
• Graceful fallback — errors don't crash, just skip
• Replace local reranking with OpenAI — no GGUF model download or local inference needed
• No breaking changes — existing workflows unchanged
Files Changed

• src/openai-llm.ts — new OpenAI LLM implementation
• src/llm.ts — embedding config, provider switching
• src/store.ts — tiktoken chunking integration
• src/qmd.ts — QMD_OPENAI env var support
Dependencies

• openai — API client
• tiktoken — fast BPE tokenization

[lyrl]

Great. I was looking for this. But the rerank in there doesn't support api calls?

[darkhanakh]

This is great! :) Good job man 🔥

[vincentkoc]

Love this!

[alexleach]

Can one change config.baseUrl easily? I would like to connect to my own hosted OpenAI-compatible server. It is actually local, but as qmd is running in a container, I need to host the models in Docker Model Runtime to gain GPU acceleration. That is OpenAI-compatible, and like other hosted implementations, it just needs a way to configure the baseUrl...

[jonesj38]

Thanks for the patience on this. I've refreshed it:

Update (2026-03-28)
Rebased feat/openai-embeddings onto current main
Resolved conflicts and cleaned commit history
Force-pushed updated branch (--force-with-lease)
Verified local build passes (bun run build)
Current PR status is now mergeable.

I also got feedback from @alexleach running OpenAI-compatible remote endpoints in minimal Docker environments.

adds configurable OPENAI_BASE_URL, and
avoids initializing/building node-llama-cpp when OpenAI mode is selected.

Waiting for him to re-submit PRs

Cheers

[jonesj38]

PR rebased to main

[paralizeer]

+1 for merging this. Just closed our PR #490 (remote Ollama embeddings) in favor of this one — it's the more complete solution.

We've been running QMD on an ARM64 VPS (Oracle Cloud Ampere, no Vulkan/GPU) and remote Ollama embeddings via HTTP is the only viable path there. This PR solves that cleanly while also adding query expansion and reranking through OpenAI-compatible endpoints.

Tested the remote embedding approach on our infra and it works great — the baseUrl override that @alexleach asked about would also cover self-hosted OpenAI-compatible servers (Ollama, vLLM, etc.).

@tobi this would unblock a lot of headless/ARM/Docker deployments. Ready to help test if needed.

[viniciushsantana]

Tested this fork with Ollama. The YAML config wiring needs two additions, plus generateEmbeddings needs to be patched to actually use the OpenAI provider. Here's what I did:

1- YAML config map forexpansion_model or base_url

collections.ts EmbeddingProviderConfig is missing both fields, and qmd.ts doesn't pass them to setEmbeddingConfig:

// collections.ts
  openai?: {
    api_key?: string;
    model?: string;
+   expansion_model?: string;
+   base_url?: string;
  };

// qmd.ts
  openai: {
    apiKey: embeddingYamlConfig.openai?.api_key,
    embedModel: embeddingYamlConfig.openai?.model,
+   expansionModel: embeddingYamlConfig.openai?.expansion_model,
+   baseURL: embeddingYamlConfig.openai?.base_url,
  },

2 - generateEmbeddings ignoring the OpenAI provider

getLlm() always returns LlamaCpp, and the embedding loop always wraps in withLLMSessionForLlm which expects LlamaCpp. When isUsingOpenAI() is true, it needs to bypass the session wrapper and call the OpenAI LLM directly.

---

My yaml config file:

collections:
# [...]
embedding:
  provider: openai
  openai:
    api_key: ollama
    base_url: "http://ollama:11434/v1"
    model: qwen3-embedding:8b
    expansion_model: qwen3.5:9b

[alexleach]

Just catching up on progress, as I received several notifications yesterday. Looks like this will need to be rebased again, since https://github.com/tobi/qmd/releases/tag/v2.1.0 was released 8 hours ago (~2026-04-06:T00:00.00Z).

One note of interest on the release, which may conflict:-

GPU: catch initialization failures and fall back to CPU instead of crashing.

---

@jonesj38 Thanks again for your work on this and rebasing it to main last week. I've one comment on this PR as is:

adds configurable OPENAI_BASE_URL

I feel this would be better if it was renamed to QMD_OPENAI_BASE_URL, as OPENAI_BASE_URL is used by the official openai-node SDK and may conflict. For QMD, I want to use my host's GPU (so I would set OPENAI_BASE_URL to http://model-runner.docker.internal:12434), but then I want openclaw to use Azure Foundry for gpt- models, for which I would set OPENAI_BASE_URL to my Azure Foundry's resource URL. Of course this can also be set in the yaml config file, and actually, having just checked "Files Changed", there is no reference to OPENAI_BASE_URL, so this would have to be set in the yaml config anyway:-

  constructor(config: OpenAIConfig = {}) {
    this.client = new OpenAI({ 
      apiKey: config.apiKey || process.env.OPENAI_API_KEY,
      baseURL: config.baseURL,
    });

Sorry for not re-submitting my PRs, my code base became a mess after trying to rebase and I ended up deleting a lot of branches...

---

@tobi please can you comment on one of the many open PRs and Issues? A lot of people are pretty desperate to use remote models and it can be a lot of work having to rebase to main each time there's a new release. If you have any feedback, we would love to hear it! 🙂

[jonesj38]

Just catching up on progress, as I received several notifications yesterday. Looks like this will need to be rebased again, since https://github.com/tobi/qmd/releases/tag/v2.1.0 was released 8 hours ago (~2026-04-06:T00:00.00Z).

One note of interest on the release, which may conflict:-

GPU: catch initialization failures and fall back to CPU instead of crashing.

---

@jonesj38 Thanks again for your work on this and rebasing it to main last week. I've one comment on this PR as is:

adds configurable OPENAI_BASE_URL

I feel this would be better if it was renamed to QMD_OPENAI_BASE_URL, as OPENAI_BASE_URL is used by the official openai-node SDK and may conflict. For QMD, I want to use my host's GPU (so I would set OPENAI_BASE_URL to http://model-runner.docker.internal:12434), but then I want openclaw to use Azure Foundry for gpt- models, for which I would set OPENAI_BASE_URL to my Azure Foundry's resource URL. Of course this can also be set in the yaml config file, and actually, having just checked "Files Changed", there is no reference to OPENAI_BASE_URL, so this would have to be set in the yaml config anyway:-

  constructor(config: OpenAIConfig = {}) {
    this.client = new OpenAI({ 
      apiKey: config.apiKey || process.env.OPENAI_API_KEY,
      baseURL: config.baseURL,
    });

Sorry for not re-submitting my PRs, my code base became a mess after trying to rebase and I ended up deleting a lot of branches...

---

@tobi please can you comment on one of the many open PRs and Issues? A lot of people are pretty desperate to use remote models and it can be a lot of work having to rebase to main each time there's a new release. If you have any feedback, we would love to hear it! 🙂

Thanks Alex, agree with renaming OPENAI_API_KEY to QMD_OPENAI_API_KEY . I'll get this branch rebased and have a look at your pull request today/tomorrow

Cheers

[jonesj38]

@tobi Rebased on to main

Cheers

[Kaspre]

Use case: split embedding + chat backends (local GPU + cloud)

We're running QMD 2.1.0 on WSL2 (Ubuntu 24.04) with no GPU access in WSL — the GPU (AMD RX 580) is only reachable via a llama-server instance on the Windows host serving nomic-embed-text over HTTP (/v1/embeddings). For query expansion and reranking, we use Ollama Cloud (gemma3:4b via https://ollama.com/v1/chat/completions).

This PR got us from "completely broken hybrid search" (node-llama-cpp triggering a 2+ minute cmake build on every query) to 4s end-to-end hybrid queries — expansion via cloud, embeddings via local GPU, reranking via cloud.

One small patch we needed on top of this PR: split base URLs.

The current base_url config is shared between embeddings and chat (expansion/reranking). When your embedding server and chat server are at different URLs (common with llama-server which only serves one model per instance), you need separate endpoints.

We added:

• chat_base_url / QMD_OPENAI_CHAT_BASE_URL — separate base URL for chat completions (falls back to base_url)
• chat_api_key / QMD_OPENAI_CHAT_API_KEY — separate API key for the chat endpoint (falls back to api_key)
• A second OpenAI client instance in OpenAIEmbedding that routes expandQuery() and rerank() to the chat URL while embed() / embedBatch() use the embedding URL

Config example:

embedding:
  provider: openai
  openai:
    api_key: "dummy"
    base_url: "http://172.30.192.1:8081/v1"       # llama-server on Windows (nomic-embed-text, GPU)
    model: "nomic-embed-text"
    chat_base_url: "https://ollama.com/v1"          # Ollama Cloud for expansion/reranking
    expansion_model: "gemma3:4b"

The patch is ~20 lines across openai-llm.ts, collections.ts, and qmd.ts. Happy to open a PR against this branch if useful.

Also: we had to disable node-llama-cpp entirely (mv node_modules/node-llama-cpp ...disabled) to prevent it from triggering cmake compilation at import time, even when OpenAI mode is active. The lazy await import("node-llama-cpp") in llm.ts appears to fire during the embed command path despite isUsingOpenAI() returning true. Might be worth guarding that import with an explicit OpenAI check, or making the embed CLI command skip LLM session initialization entirely when in OpenAI mode.

[jarvisaoieong]

I found the openai setting only work in cli mode.
It doesn't work in SDK case.

[jonesj38]

Pushed a follow-up to address this.

What was fixed:

1. SDK/library mode now applies OpenAI embedding config

   • Root cause: the CLI was calling setEmbeddingConfig(...), but createStore() in src/index.ts was not.
   • Result: embedding.provider: openai worked in CLI mode but SDK consumers silently stayed on the default local provider.
   • Fix: createStore() now initializes the embedding provider from inline config, YAML config, and QMD_OPENAI_* env vars the same way the CLI does.

2. Unrelated CLI build break fixed

   • While validating the branch, npm run build was failing with src/cli/qmd.ts(3410,1): error TS1005: '}' expected.
   • Root cause was a missing closing brace in showStatus() after the local-model else branch.
   • Fixed that as well so the branch builds cleanly again.

Validation:

• npm run build ✅
• npx vitest run test/sdk.test.ts ✅ (80/80)

Commits pushed:

• 8eab131 — fix: apply OpenAI embedding config in SDK mode
• 6d06cf8 — fix: close showStatus local model branch

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​tiktoken@​1.0.22
	npm/​openai@​4.104.0

View full report

[tobi]

Closing as part of the post-v2.5.1 backlog cleanup: this item is over two months old and has either been superseded by newer QMD releases or is too stale to keep actionable. If it still reproduces on v2.5.1 or newer, please open a fresh focused issue with current repro steps.

[tobi]

Closing as part of the post-v2.5.1 backlog cleanup: this PR is over two months old and has either been superseded by newer QMD releases or is too stale to keep actionable. If this is still relevant on v2.5.1 or newer, please open a fresh focused PR against current main.