Repository navigation
feat(telemetry): let fixed-base-URL clients name their rollout in the request URL - #4969
Conversation
…eader Long-lived clients with one base URL could not add the /ng-rollout/<id>/ prefix per request, so integrations patched httpx and requests globally to get their calls captured. Model servers and resources servers now also accept x-ng-rollout-id, refuse a header that is malformed or contradicts the prefix, and add_rollout_id_header tags requests from the current rollout for listed Gym servers only. Closes #4724. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
a62f430 to
ced9e36
Compare
|
/ok to test b53a75d |
|
🌿 Preview your docs: https://nvidia-preview-pr-4969-dc73207252c4.docs.buildwithfern.com/nemo/gym |
…ep its path intact - A request object sent again (retry or replay) kept the rollout prefix of its first send, because the hook rewrote request.url in place and the old prefix then looked explicit. The hook keeps the original URL in the request's extensions and rewrites from it on every send. - Matching ignored empty path segments while the rewrite cut the original path by character count, so `//api/...` under base `/api` corrupted the path and the rollout id. The prefix now goes right after the base's segments in the path as sent. - An explicit port 0 no longer matches the default port. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
httpx copies request extensions onto a redirect request, so the hook took the original URL back and requested it again until TooManyRedirects. The hook now keeps the original URL together with the URL it last wrote, and treats any other incoming URL, such as a redirect target, as new. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
f2081e0 to
531f592
Compare
|
/ok to test 531f592 |
Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> # Conflicts: # nemo_gym/server_utils.py
|
/ok to test 86269c1 |
Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> # Conflicts: # nemo_gym/server_runtime/base_server.py
|
/ok to test 9693b7f |
Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
|
/ok to test 97ac6b4 |
Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> # Conflicts: # nemo_gym/server_runtime/__init__.py # nemo_gym/server_runtime/base_server.py
|
/ok to test dc73207 |
|
Some context from the Auto Ontology side, and a few questions about the httpx examples. Is the motivation still current? #5633 (draft) runs Auto Ontology as an Agent Server on the BIRD-Interact Environment Server, so it no longer runs inside a resources server's Should the docs recommend httpx here? The new section builds an Is Did the thread pools affect #4724 too? The |
ffrujeri
left a comment
There was a problem hiding this comment.
Approve. The design is good: a fixed-base-URL client now sends the same /ng-rollout/<id>/ URL as Gym's own clients, so no server layer changes. Two questions inline; neither blocks.
sequenceDiagram
participant R as Rollout runner
participant V as Resources server verify()
participant H as httpx client + rollout_prefix_hook
participant M as Model server
R->>V: POST /verify (sets current rollout ContextVar)
V->>H: client.post("/v1/responses")
H->>H: with_rollout_prefix(url, base_urls) reads current_rollout_id()
H->>M: POST /ng-rollout/<id>/v1/responses
Note over M: capture, checkpoint gate and session affinity read the prefix
flowchart LR
subgraph server_runtime
BS["base_server.py (new: with_rollout_prefix, hooks)"]
TC["request_handling/trace_context.py (_ROLLOUT_PREFIX_RE)"]
end
subgraph client
SC["server_client.py (rollout_path_prefix, apply_rollout_prefix)"]
end
subgraph core
EC["episode_context.current_rollout_id"]
end
BS --> SC
BS --> EC
BS -. private import .-> TC
| return index | ||
|
|
||
|
|
||
| def with_rollout_prefix(url: str, base_urls: Sequence[str]) -> str: |
There was a problem hiding this comment.
Can this helper go in nemo_gym/client/server_client.py, next to rollout_path_prefix and apply_rollout_prefix? It is client code, and base_server.py holds server code. client can import current_rollout_id from core. If _ROLLOUT_PREFIX_PATTERN also moves down to client, line 92 does not need the private import from trace_context. A public name is easier to move now than after a release.
| rest = "/" + "/".join(segments[len(base_segments) :]) | ||
| if _ROLLOUT_PREFIX_RE.match(rest): | ||
| return url | ||
| rollout_id = current_rollout_id() |
There was a problem hiding this comment.
Do you leave out the trace context segment on purpose? apply_rollout_prefix also adds /ng-trace/<traceparent>. This hook runs for each request, so the current span is correct here. Without the segment, do calls from verify() start a new trace? If this is out of scope, can you add it to the follow-up?
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41925813852 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41927931388 |
What and why
Auto-Ontology's agent runs inside a resources server's
verify()and holds HTTP clients with a fixed base URL. Gymattributes model calls to a rollout only through the
/ng-rollout/<id>/URL prefix, so the integration wrappedhttpx.Client.send,httpx.AsyncClient.sendandrequests.Session.sendglobally to rewrite every request(#4724, part of #4720).
This PR gives those clients a supported way to add the same prefix, with no new server-side carrier:
with_rollout_prefix(url, base_urls)(inserver_utils, besiderollout_path_prefixandapply_rollout_prefix) inserts/ng-rollout/<current rollout>after the matching base URL.requests normalize them. A single string instead of a list is refused.
(an environment server's sub-episode, a token-capture prefix) is unaffected.
without capture.
rollout_prefix_hook/async_rollout_prefix_hookwrap it as httpx request event hooks; the OpenAI SDK takessuch a client through
http_client. The docs show arequestsadapter mounted forhttp://andhttps://.ContextVarcaveat: a synchronousclient sees the rollout through
asyncio.to_threadorcontextvars.copy_context().run, not throughrun_in_executoror a thread pool.Because the result is an ordinary prefixed URL, every layer that reads the prefix (capture, the policy checkpoint
gate, switchyard's session affinity, the token-capture segment) treats these calls exactly like Gym's own.
An earlier version of this PR added an
x-ng-rollout-idheader read by the capture middleware andRolloutContextMiddleware. Review showed that the checkpoint gate, switchyard and agent restart tracking read onlythe prefix, so a header-only call skipped attempt fencing; that version is replaced and no server code changes.
Closes #4724.
Validation
telemetryextra installed:pytest -n 8 tests/unit_tests/test_rollout_url_prefix.py tests/unit_tests/test_server_utils.py tests/unit_tests/test_rollout_correlation.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_fern_docs_links.py: 344 passed,24 subtests passed.
test_rollout_url_prefix.py, 20):http://handhttp://h:80,https://h:443), segment and port lookalikes, another scheme or host, and an existing prefix (with and withouttraining-token-capture) left alone;requestsadapter against a real uvicorn server wrapped inRequestContextIsolationMiddleware, as in production; a sync client inasyncio.to_thread;httpx.AsyncClient(ASGI transport, isolated per request) each reachthe server under their own rollout.
with_rollout_prefixmade a no-op, the client tests fail.pre-commit run --files <changed files>: passed (the guardwords hook skipped itself: private catalog unavailable).Rollout evidence
gym env start --resources-server mcqa --model-type openai_model ++observability_enabled=trueagainstgpt-6-luna, then real requests to the policy model server:base_url=<server>/v1andhttp_client=httpx.Client(event_hooks=[rollout_prefix_hook(...)])inside rollout
smoke-sdk;httpx.AsyncClientwithasync_rollout_prefix_hook, two concurrent requests undersmoke-0andsmoke-1, and one request outside a rollout.All four returned 200. The three in a rollout were sent to
/ng-rollout/<rollout>/v1/responsesand produced onecapture record each, in
smoke-sdk.capture.jsonl,smoke-0.capture.jsonlandsmoke-1.capture.jsonl. The requestoutside a rollout was sent unprefixed and produced none.
Compatibility
Additive: three new functions and a docs section. No server behaviour changes.
Follow-up
#4630's verifier invocation and later session IDs could travel through the same client helper, so a client tags one
place for all of Gym's correlation context; left out until that design settles.
Linear: FEP-1834 (part of FEP-1829)