You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Docs: add usage guidance for litellm_model and disambiguate model servers #1666
PR #990 adds a standalone responses_api_models/litellm_model server for LiteLLM proxy endpoints. It extends openai_model's SimpleModelServer and uses the native /v1/responses path, normalizing LiteLLM proxy quirks (chat.completion hybrid responses, reasoning.effort="none" → null).
PR #990 ships with only a minimal in-repo README. The broader cross-server documentation is intentionally deferred to this follow-up so it doesn't couple #990 to the in-flight inference_provider server (renamed from chat_completions_model, currently on a branch).
Scope
Add documentation that disambiguates the model servers and explains when to use each. This spans multiple servers + the Fern docs site, so it belongs in its own PR once inference_provider lands on main.
Fern model-server docs (fern/versions/latest/pages/model-server/): add a "when to use which model server" section covering:
inference_provider — any hosted provider via OpenAI-compatible /v1/chat/completions (Responses synthesized client-side)
openai_model — direct OpenAI, native /v1/responses
litellm_model — a LiteLLM proxy fronting providers, using native /v1/responses with quirk normalization
Expand responses_api_models/litellm_model/README.md with a "When to Use This vs Other Model Servers" section (mirror the inference_provider README pattern).
Example config + ng_run invocation for routing through a LiteLLM proxy.
One ambiguity worth addressing explicitly is why inference_provider is needed when openai_model and vllm_model already accept configurable OpenAI-compatible base URLs.
Suggested clarification for the Fern documentation:
openai_model is a passthrough for an upstream that natively implements /v1/responses; its local /v1/responses route forwards to the upstream Responses API.
inference_provider targets hosted services that may expose only /v1/chat/completions; it translates Gym Responses requests to Chat Completions and synthesizes a Responses object from the result.
vllm_model can perform a similar Responses-to-Chat-Completions conversion, but is specialized for self-hosted vLLM and includes training and vLLM-specific behavior such as token IDs/logprobs, /tokenize, chat-template controls, endpoint/session routing, and vLLM error handling.
The docs should also acknowledge that vllm_model may technically work with some hosted OpenAI-compatible endpoints when its specialized options are disabled. The reason to prefer inference_provider is therefore the smaller hosted-provider contract and separation from vLLM-specific assumptions—not simply that conversion is otherwise unavailable.
A capability-based selection guide would make this actionable:
Upstream natively supports Responses API → openai_model
Hosted upstream supports Chat Completions only → inference_provider
Self-hosted vLLM or RL training requiring token-level data → vllm_model
LiteLLM proxy with native Responses support and normalization needs → litellm_model
It would also help to state that the YAML files configure these distinct server implementations; they are not interchangeable aliases for the same abstraction.
This clarification supports the self-service goal described in #1452 by reducing the internal architectural knowledge required to choose a model-server configuration.
Activity
ffrujeri commentedon Jul 23, 2026
One ambiguity worth addressing explicitly is why
inference_provideris needed whenopenai_modelandvllm_modelalready accept configurable OpenAI-compatible base URLs.Suggested clarification for the Fern documentation:
openai_modelis a passthrough for an upstream that natively implements/v1/responses; its local/v1/responsesroute forwards to the upstream Responses API.inference_providertargets hosted services that may expose only/v1/chat/completions; it translates Gym Responses requests to Chat Completions and synthesizes a Responses object from the result.vllm_modelcan perform a similar Responses-to-Chat-Completions conversion, but is specialized for self-hosted vLLM and includes training and vLLM-specific behavior such as token IDs/logprobs,/tokenize, chat-template controls, endpoint/session routing, and vLLM error handling.The docs should also acknowledge that
vllm_modelmay technically work with some hosted OpenAI-compatible endpoints when its specialized options are disabled. The reason to preferinference_provideris therefore the smaller hosted-provider contract and separation from vLLM-specific assumptions—not simply that conversion is otherwise unavailable.A capability-based selection guide would make this actionable:
openai_modelinference_providervllm_modellitellm_modelIt would also help to state that the YAML files configure these distinct server implementations; they are not interchangeable aliases for the same abstraction.
This clarification supports the self-service goal described in #1452 by reducing the internal architectural knowledge required to choose a model-server configuration.