Skip to content

Add LiteLLM provider into ‎responses_api_models - #990

Merged
sephmard merged 15 commits into
NVIDIA-NeMo:mainfrom
imxj:feat/support_litellm_responses
Aug 10, 2026
Merged

sephmard merged 15 commits into
NVIDIA-NeMo:mainfrom
imxj:feat/support_litellm_responses

Conversation

@imxj

@imxj imxj commented Apr 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add new responses_api_models/litellm_model server for LiteLLM proxy endpoints
  • LiteLLMModelServer extends openai_model's SimpleModelServer, overriding only responses() to normalize LiteLLM proxy quirks
  • Normalize chat.completion hybrid responses to standard Responses API format (LiteLLM may downgrade /v1/responses internally)
  • Fix reasoning.effort="none" (string) to null Pydantic validation error for native Responses API responses
  • openai_model remains clean -- no proxy-specific workarounds

Test

  • Tested with GPT-5.4 via LiteLLM-backed endpoint -- native response format, reasoning fix applied
  • Tested with Opus 4.6 via LiteLLM-backed endpoint -- chat.completion hybrid normalized
  • End-to-end rollout collection verified with both models
  • 11 unit tests for litellm_model (normalization + server integration)
  • openai_model existing tests still pass after revert

@copy-pr-bot

copy-pr-bot Bot commented Apr 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@imxj
imxj force-pushed the feat/support_litellm_responses branch 2 times, most recently from 805aaef to f0558e5 Compare April 1, 2026 00:42
@imxj imxj changed the title Add LiteLLM response normalization for Responses API proxy compatibility Add LiteLLM provider into ‎responses_api_models Apr 7, 2026
imxj and others added 2 commits April 6, 2026 18:52
LiteLLM proxies may return chat.completion format or hybrid response
objects when called via /v1/responses. This normalizes those responses
to the expected Responses API shape so downstream NeMoGymResponse
validation succeeds. Also fixes reasoning.effort="none" validation
error for native Responses API responses.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Jin Xu <jinx@nvidia.com>
Revert openai_model/app.py to its clean state and move the LiteLLM
response normalization logic into a new responses_api_models/litellm_model
that extends SimpleModelServer. This keeps proxy-specific workarounds
(reasoning.effort="none" fix, chat.completion->response normalization)
isolated from the native OpenAI model server.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Jin Xu <jinx@nvidia.com>
@imxj
imxj force-pushed the feat/support_litellm_responses branch from 105e7c6 to 307b580 Compare April 7, 2026 01:52
@kajalj22
kajalj22 self-requested a review June 23, 2026 18:10
@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Jun 23, 2026

@kajalj22 kajalj22 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests are fully mocked so they never hit a real LiteLLM proxy, but runtime security is untested/potentially impacted with server url (could use any version).

The risk is on the proxy server side, if it ran a compromised version, the API keys it uses to call upstream LLMs may have been exfiltrated.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Jun 23, 2026
Adds a startup HTTP check against the LiteLLM proxy's /health endpoint
to detect and hard-fail on v1.82.7 and v1.82.8, which contained
credential-harvesting malware (supply chain incident, March 2026).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22 and others added 2 commits June 24, 2026 18:55
/health/readiness/details is the correct endpoint that exposes
litellm_version — verified against v1.89.3. /health does not
include version info.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
- Add TestVersionCheck with three cases: compromised version blocks
  startup, safe version passes, unreachable proxy only warns
- Add mock_safe_version fixture auto-applied to all server integration
  tests so they don't make real HTTP calls during the startup check

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

Copy link
Copy Markdown
Contributor

Raised a PR addressing my comment here: imxj#1

@anwithk anwithk moved this from Dev Todo to Review Todo in NeMo Gym 0.4.0 - July 1 Jun 25, 2026
@imxj
imxj requested a review from kajalj22 June 26, 2026 07:30
@imxj

imxj commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. I merged the follow up fixes into this pr branch

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Jun 26, 2026
@kajalj22
kajalj22 requested a review from a team June 26, 2026 20:52
kajalj22
kajalj22 previously approved these changes Jun 26, 2026
imxj and others added 4 commits June 27, 2026 02:33
Signed-off-by: Jin Xu <jin.xu999@gmail.com>
…ersion check errors

- Add x-litellm-key to _SENSITIVE_HEADER_RE so the newly introduced
  auth header is redacted alongside Authorization in all error paths
- Wrap the exception interpolation in _check_proxy_version with
  _sanitize_error() so API keys cannot leak into RuntimeError messages
  if aiohttp surfaces request headers in the exception

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
feat(litellm): block compromised proxy versions on startup
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Jul 3, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Jul 10, 2026
Signed-off-by: Jin Xu <jin.xu999@gmail.com>
@imxj
imxj force-pushed the feat/support_litellm_responses branch from f5c4df1 to f376b31 Compare August 1, 2026 16:46
…m_responses

Signed-off-by: Jin Xu <jin.xu999@gmail.com>
@imxj
imxj requested a review from ffrujeri August 1, 2026 16:53
@imxj

imxj commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Since the dismissed approval: fixed the ruff-format issue in tests/test_app.py, added the missing DCO sign-off (amend, no functional changes), and merged latest main. DCO is green; lint should pass once CI is re-triggered on faae30f.

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Aug 1, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Aug 3, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Aug 3, 2026
…m_responses

Signed-off-by: Jin Xu <jin.xu999@gmail.com>
ffrujeri
ffrujeri previously approved these changes Aug 10, 2026
@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Aug 10, 2026
imxj added 2 commits August 10, 2026 13:42
setup_webserver() on main now installs model-call capture from
server_client.global_config_dict; mirror the openai_model test setup.

Signed-off-by: Jin Xu <jin.xu999@gmail.com>
@sephmard
sephmard merged commit abcb0a0 into NVIDIA-NeMo:main Aug 10, 2026
8 checks passed
@github-project-automation github-project-automation Bot moved this from Dev Todo to Done in NeMo Gym 0.6.0 - Sept 2 Aug 10, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Issue reported or requested by someone from the community

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants