Skip to content

Docs: add usage guidance for litellm_model and disambiguate model servers #1666

Open

Description

@cwing-nvidia

Background

PR #990 adds a standalone responses_api_models/litellm_model server for LiteLLM proxy endpoints. It extends openai_model's SimpleModelServer and uses the native /v1/responses path, normalizing LiteLLM proxy quirks (chat.completion hybrid responses, reasoning.effort="none" → null).

PR #990 ships with only a minimal in-repo README. The broader cross-server documentation is intentionally deferred to this follow-up so it doesn't couple #990 to the in-flight inference_provider server (renamed from chat_completions_model, currently on a branch).

Scope

Add documentation that disambiguates the model servers and explains when to use each. This spans multiple servers + the Fern docs site, so it belongs in its own PR once inference_provider lands on main.

  • Fern model-server docs (fern/versions/latest/pages/model-server/): add a "when to use which model server" section covering:
    • inference_provider — any hosted provider via OpenAI-compatible /v1/chat/completions (Responses synthesized client-side)
    • openai_model — direct OpenAI, native /v1/responses
    • litellm_model — a LiteLLM proxy fronting providers, using native /v1/responses with quirk normalization
    • vllm_model — self-hosted vLLM (training + eval, supports token IDs)
  • Expand responses_api_models/litellm_model/README.md with a "When to Use This vs Other Model Servers" section (mirror the inference_provider README pattern).
  • Example config + ng_run invocation for routing through a LiteLLM proxy.

Dependencies

Activity

  1. added
    documentationImprovements to documentation
    modelsinfra for different models and inference providers; middleware to standardize to OAI responses
    on Jun 23, 2026
  2. ffrujeri commented on Jul 23, 2026

    @ffrujeri
    Contributor

    One ambiguity worth addressing explicitly is why inference_provider is needed when openai_model and vllm_model already accept configurable OpenAI-compatible base URLs.

    Suggested clarification for the Fern documentation:

    • openai_model is a passthrough for an upstream that natively implements /v1/responses; its local /v1/responses route forwards to the upstream Responses API.
    • inference_provider targets hosted services that may expose only /v1/chat/completions; it translates Gym Responses requests to Chat Completions and synthesizes a Responses object from the result.
    • vllm_model can perform a similar Responses-to-Chat-Completions conversion, but is specialized for self-hosted vLLM and includes training and vLLM-specific behavior such as token IDs/logprobs, /tokenize, chat-template controls, endpoint/session routing, and vLLM error handling.

    The docs should also acknowledge that vllm_model may technically work with some hosted OpenAI-compatible endpoints when its specialized options are disabled. The reason to prefer inference_provider is therefore the smaller hosted-provider contract and separation from vLLM-specific assumptions—not simply that conversion is otherwise unavailable.

    A capability-based selection guide would make this actionable:

    • Upstream natively supports Responses API → openai_model
    • Hosted upstream supports Chat Completions only → inference_provider
    • Self-hosted vLLM or RL training requiring token-level data → vllm_model
    • LiteLLM proxy with native Responses support and normalization needs → litellm_model

    It would also help to state that the YAML files configure these distinct server implementations; they are not interchangeable aliases for the same abstraction.

    This clarification supports the self-service goal described in #1452 by reducing the internal architectural knowledge required to choose a model-server configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    RolloutdocumentationImprovements to documentationmodelsinfra for different models and inference providers; middleware to standardize to OAI responses

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions