Skip to content

feat(telemetry): let fixed-base-URL clients name their rollout in the request URL - #4969

Merged
marcromeyn merged 11 commits into
mainfrom
romeyn/rollout-id-header
Nov 4, 2026
Merged

marcromeyn merged 11 commits into
mainfrom
romeyn/rollout-id-header

Conversation

@marcromeyn

@marcromeyn marcromeyn commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Auto-Ontology's agent runs inside a resources server's verify() and holds HTTP clients with a fixed base URL. Gym
attributes model calls to a rollout only through the /ng-rollout/<id>/ URL prefix, so the integration wrapped
httpx.Client.send, httpx.AsyncClient.send and requests.Session.send globally to rewrite every request
(#4724, part of #4720).

This PR gives those clients a supported way to add the same prefix, with no new server-side carrier:

  • with_rollout_prefix(url, base_urls) (in server_utils, beside rollout_path_prefix and
    apply_rollout_prefix) inserts /ng-rollout/<current rollout> after the matching base URL.
    • Base URLs are compared by scheme, case-insensitive host, effective port and whole path segments, as httpx and
      requests normalize them. A single string instead of a list is refused.
    • A URL that already carries a rollout prefix below the base is left alone, so a client addressing another key
      (an environment server's sub-episode, a token-capture prefix) is unaffected.
    • With no current rollout the URL is unchanged and a debug message is logged; the model server forwards the call
      without capture.
  • rollout_prefix_hook / async_rollout_prefix_hook wrap it as httpx request event hooks; the OpenAI SDK takes
    such a client through http_client. The docs show a requests adapter mounted for http:// and https://.
  • Docs. "Clients with a fixed base URL" in model-call capture, including the ContextVar caveat: a synchronous
    client sees the rollout through asyncio.to_thread or contextvars.copy_context().run, not through
    run_in_executor or a thread pool.

Because the result is an ordinary prefixed URL, every layer that reads the prefix (capture, the policy checkpoint
gate, switchyard's session affinity, the token-capture segment) treats these calls exactly like Gym's own.

An earlier version of this PR added an x-ng-rollout-id header read by the capture middleware and
RolloutContextMiddleware. Review showed that the checkpoint gate, switchyard and agent restart tracking read only
the prefix, so a header-only call skipped attempt fencing; that version is replaced and no server code changes.

Closes #4724.

Validation

  • With the telemetry extra installed: pytest -n 8 tests/unit_tests/test_rollout_url_prefix.py tests/unit_tests/test_server_utils.py tests/unit_tests/test_rollout_correlation.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_fern_docs_links.py: 344 passed,
    24 subtests passed.
  • New tests (test_rollout_url_prefix.py, 20):
    • URL matching: base paths, a trailing slash, host case, default ports both ways (http://h and http://h:80,
      https://h:443), segment and port lookalikes, another scheme or host, and an existing prefix (with and without
      training-token-capture) left alone;
    • no current rollout: unchanged and logged; a single base-URL string is refused;
    • the sync httpx hook and the documented requests adapter against a real uvicorn server wrapped in
      RequestContextIsolationMiddleware, as in production; a sync client in asyncio.to_thread;
    • 20 concurrent rollouts through one shared httpx.AsyncClient (ASGI transport, isolated per request) each reach
      the server under their own rollout.
  • With with_rollout_prefix made a no-op, the client tests fail.
  • pre-commit run --files <changed files>: passed (the guardwords hook skipped itself: private catalog unavailable).

Rollout evidence

gym env start --resources-server mcqa --model-type openai_model ++observability_enabled=true against
gpt-6-luna, then real requests to the policy model server:

  • the OpenAI SDK with base_url=<server>/v1 and http_client=httpx.Client(event_hooks=[rollout_prefix_hook(...)])
    inside rollout smoke-sdk;
  • one shared httpx.AsyncClient with async_rollout_prefix_hook, two concurrent requests under smoke-0 and
    smoke-1, and one request outside a rollout.

All four returned 200. The three in a rollout were sent to /ng-rollout/<rollout>/v1/responses and produced one
capture record each, in smoke-sdk.capture.jsonl, smoke-0.capture.jsonl and smoke-1.capture.jsonl. The request
outside a rollout was sent unprefixed and produced none.

Compatibility

Additive: three new functions and a docs section. No server behaviour changes.

Follow-up

#4630's verifier invocation and later session IDs could travel through the same client helper, so a client tags one
place for all of Gym's correlation context; left out until that design settles.

Linear: FEP-1834 (part of FEP-1829)

@copy-pr-bot

copy-pr-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 23, 2026
…eader

Long-lived clients with one base URL could not add the /ng-rollout/<id>/
prefix per request, so integrations patched httpx and requests globally to
get their calls captured. Model servers and resources servers now also
accept x-ng-rollout-id, refuse a header that is malformed or
contradicts the prefix, and add_rollout_id_header tags requests from the
current rollout for listed Gym servers only.

Closes #4724.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test b53a75d

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@marcromeyn
marcromeyn enabled auto-merge October 23, 2026 15:20
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 26, 2026
…ep its path intact

- A request object sent again (retry or replay) kept the rollout prefix of
  its first send, because the hook rewrote request.url in place and the old
  prefix then looked explicit. The hook keeps the original URL in the
  request's extensions and rewrites from it on every send.
- Matching ignored empty path segments while the rewrite cut the original
  path by character count, so `//api/...` under base `/api` corrupted the
  path and the rollout id. The prefix now goes right after the base's
  segments in the path as sent.
- An explicit port 0 no longer matches the default port.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
httpx copies request extensions onto a redirect request, so the hook took
the original URL back and requested it again until TooManyRedirects. The
hook now keeps the original URL together with the URL it last wrote, and
treats any other incoming URL, such as a redirect target, as new.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@marcromeyn
marcromeyn force-pushed the romeyn/rollout-id-header branch from f2081e0 to 531f592 Compare October 29, 2026 13:02
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 531f592

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>

# Conflicts:
#	nemo_gym/server_utils.py
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 86269c1

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>

# Conflicts:
#	nemo_gym/server_runtime/base_server.py
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 9693b7f

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 97ac6b4

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>

# Conflicts:
#	nemo_gym/server_runtime/__init__.py
#	nemo_gym/server_runtime/base_server.py
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test dc73207

@ffrujeri

ffrujeri commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Some context from the Auto Ontology side, and a few questions about the httpx examples.

Is the motivation still current? #5633 (draft) runs Auto Ontology as an Agent Server on the BIRD-Interact Environment Server, so it no longer runs inside a resources server's verify() or patches HTTP clients globally. Auto Ontology still builds its model clients once for each process, with a fixed base URL, so the problem remains. Should the first paragraph of the description state the problem without the verify() integration?

Should the docs recommend httpx here? The new section builds an httpx.AsyncClient and an httpx.Client with the hook. AGENTS.md asks Gym code to send async HTTP through the global aiohttp client rather than httpx.AsyncClient, and to give a wrapped library that uses httpx an aiohttp transport adapter, as tavily_search does. Is the httpx.AsyncClient example meant as an exception to that rule?

Is ServerClient enough for code inside a Gym server? As far as I can tell, ServerClient already adds /ng-rollout/<current_rollout_id()> to a model server's /v1/responses, /v1/chat/completions and /v1/messages when observability or checkpointing is on (nemo_gym/client/server_client.py). If so, a transport that hands the SDK's request to ServerClient needs no hook. That is what we are considering for Auto Ontology: its ChatOpenAI would get an httpx.Client whose transport sends through ServerClient, so httpx opens no connections. Would it help to mention this route in the section, and keep the hook for clients that must keep their own transport? Or is there a case where the hook is the better choice even inside a Gym server?

Did the thread pools affect #4724 too? The ContextVar caveat you added applies to Auto Ontology: one of its thread pools (coordinator.py:1053) does not copy the context, so its calls carry no rollout. We plan to propose ContextThreadPoolExecutor there to the Auto Ontology team. Did the #4724 integration lose those calls as well?

@ffrujeri ffrujeri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. The design is good: a fixed-base-URL client now sends the same /ng-rollout/<id>/ URL as Gym's own clients, so no server layer changes. Two questions inline; neither blocks.

sequenceDiagram
  participant R as Rollout runner
  participant V as Resources server verify()
  participant H as httpx client + rollout_prefix_hook
  participant M as Model server
  R->>V: POST /verify (sets current rollout ContextVar)
  V->>H: client.post("/v1/responses")
  H->>H: with_rollout_prefix(url, base_urls) reads current_rollout_id()
  H->>M: POST /ng-rollout/<id>/v1/responses
  Note over M: capture, checkpoint gate and session affinity read the prefix
Loading
flowchart LR
  subgraph server_runtime
    BS["base_server.py (new: with_rollout_prefix, hooks)"]
    TC["request_handling/trace_context.py (_ROLLOUT_PREFIX_RE)"]
  end
  subgraph client
    SC["server_client.py (rollout_path_prefix, apply_rollout_prefix)"]
  end
  subgraph core
    EC["episode_context.current_rollout_id"]
  end
  BS --> SC
  BS --> EC
  BS -. private import .-> TC
Loading

return index


def with_rollout_prefix(url: str, base_urls: Sequence[str]) -> str:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this helper go in nemo_gym/client/server_client.py, next to rollout_path_prefix and apply_rollout_prefix? It is client code, and base_server.py holds server code. client can import current_rollout_id from core. If _ROLLOUT_PREFIX_PATTERN also moves down to client, line 92 does not need the private import from trace_context. A public name is easier to move now than after a release.

rest = "/" + "/".join(segments[len(base_segments) :])
if _ROLLOUT_PREFIX_RE.match(rest):
return url
rollout_id = current_rollout_id()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you leave out the trace context segment on purpose? apply_rollout_prefix also adds /ng-trace/<traceparent>. This hook runs for each request, so the current span is correct here. Without the segment, do calls from verify() start a new trace? If this is out of scope, can you add it to the follow-up?

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41925813852

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41927931388

This branch was previously deployed

1 inactive deployment
public — dc732072 Deployed Nov 4, 2026 by copy-pr-bot[bot] via release / finalize / notify #9963
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide supported rollout tagging for external model clients

4 participants