Skip to content

Route-aware evaluation for user-provided routers and gateways #3416

Open

Description

@cwing-nvidia

Define route-aware evaluation for user-provided routers and gateways

Problem

A user may operate a model router or gateway—such as Kong, RAMP, or an internal system—that NeMo Gym does not yet recognize by name. If it exposes a compatible model API, the user may already be able to run Gym benchmarks against it through a generic model server. However, Gym then treats it as an opaque model endpoint rather than a system whose routing behavior is being evaluated.

Using Switchyard should not be a prerequisite for evaluating routing. Users should be able to evaluate routing behavior where it is already implemented—whether in LiteLLM, OpenRouter, Kong, RAMP, or a custom gateway—without first reproducing that strategy in Switchyard or adding a dedicated Gym model server.

Users want to:

  • Compare fixed-model baselines and routing algorithms or configurations on the same tasks.
  • Compare router and gateway stacks, including direct and composed deployments such as Switchyard through LiteLLM.
  • Route across heterogeneous targets—such as a model served by Gym's vllm_model and an external OpenAI endpoint—without changing the benchmark or agent harness.
  • Measure reward, selected models, total usage or cost, latency, and failures.
  • Record enough routing-condition and deployment provenance to reproduce a comparison.
  • Preserve target identity and relevant backend-specific capabilities, such as token capture, when routing through different model-server types.
  • Understand explicitly which attribution or accounting is unavailable when a router exposes only a black-box model API.

Related work

  • #2088, completed by #2141, added routing-condition evaluation, attribution, statistics, and provenance for the dedicated Switchyard model server. It does not establish the same experience for other routers or generic model endpoints.
  • #3298 covers a vendor-neutral routing-evaluation documentation workflow, using Switchyard as the most complete example and a generic endpoint such as OpenRouter Auto Router as a black-box example. It explicitly excludes adding or redesigning integrations, so it can document capability gaps but does not close them.
  • #990 added LiteLLM as a model endpoint, while #1666 covers guidance for choosing and configuring model servers. Neither defines route-aware comparison, attribution, or accounting for routers exposed through LiteLLM.
  • #1285, completed by #1286, added the generic inference_provider, including an OpenRouter configuration; the earlier #157 covered using OpenRouter as an inference endpoint. The existing configuration can send openrouter/auto to the OpenRouter Auto Router, but this is only black-box connectivity: on Gym's Responses conversion path, the response records the requested router slug rather than OpenRouter's returned selected model, losing route attribution. There is also no OpenRouter-specific routing-condition provenance or metadata workflow.
  • #2553 proposes an OrcaRouter configuration for inference_provider and demonstrates that an unlisted OpenAI-compatible router can use the generic provider configuration. It enables connectivity, but does not provide routing-algorithm comparison, selected-model attribution, router-internal usage, or routing-condition provenance.
  • #1564 tracks shared compatibility across model API shapes, not routing semantics or evaluation.
  • #2891 covers documentation of model-call capture and rollout observability. The closed #1483 describes the broader token, latency, and model-call metrics contract. These cover evidence visible at Gym's model-server boundary, but not router-internal decisions, calls, or overhead that an external system does not expose there.

Remaining gap

The existing work provides a rich Switchyard-specific evaluation path and generic connectivity for compatible endpoints. It does not define a consistent route-aware evaluation experience for:

  • A router or gateway that has not been added to Gym.
  • Comparing routing algorithms across different router implementations.
  • Comparing direct and composed gateway/router deployment paths.
  • Composing one routing target pool from existing Gym model servers and external provider endpoints.
  • Preserving target identity, observability, and backend-specific capabilities across heterogeneous targets.
  • Reporting comparable results when different systems expose different levels of routing metadata.

It is also unclear which capabilities should work through generic configuration and when a router-specific Gym integration is warranted.

Desired outcome

Define the route-aware capability contract and integration model for user-provided routers and gateways. A user should be able to run controlled comparisons without first adding their router to Gym, while optionally using richer attribution, accounting, provenance, and backend-specific capabilities when their system exposes them.

Use the existing Switchyard integration as a reference to determine which capabilities can be generalized, how routing composes with existing Gym model servers and external endpoints, and what extension model should support other routers and gateways. The vendor-neutral user workflow and documentation are tracked separately in #3298.

Validate the resulting experience against:

  1. A fixed-model baseline and multiple routing conditions on equivalent tasks.
  2. An existing rich integration such as direct Switchyard.
  3. OpenRouter Auto Router through the existing inference_provider configuration.
  4. An unlisted router reached through a generic model endpoint.
  5. A composed path such as Switchyard through LiteLLM.
  6. A heterogeneous target pool containing both a Gym-served vLLM model and an external hosted model endpoint.

The result should make unavailable metrics explicit rather than silently presenting a partial comparison as complete.

Non-goals

  • Adding Kong, RAMP, or another specific router as part of this issue.
  • Standardizing routing algorithms or vendor configuration formats.
  • Requiring routers to run inside Gym.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions