Repository navigation
Conversation
Let training frameworks own capture roots and continuation decisions using request replay evidence and committed coordinates. Preserve completion identity, buffered Responses capture, explicit retry controls, and exact measurement through the ordinary vLLM server. Keep the existing compaction agent and model intact in this infrastructure layer. The native agent migration is a separate stacked change. Validation: 618 core/model cases and 56 retained legacy agent/model cases passed on Mac CPU across the documented runs. Combined-head GPU smoke evidence is recorded separately. Co-authored-by: alexandery <alexandery@nvidia.com> Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
5d32ffa to
66ce5c0
Compare
| return manifest.model_dump(mode="json") | ||
|
|
||
|
|
||
| def _validate_sequential_intent(rollout_id: str, rows: list[dict], expected_latest_call_id: str | None) -> None: |
There was a problem hiding this comment.
nemo_gym/token_id_capture/lineage.py:222
Why does sequential capture need to be mandated here? Some harnesses enqueue concurrent model calls (parallel tool calls, speculative branches), and with this check the first overlap is refused and the rollout is lost. The ledger already records siblings (same parent_call_id) and main's fingerprint lineage commits them today.
Two questions: is call order a hard requirement to maintain? And if it is, could the intent row in the manifest serve as the order marker (intents are appended in admission order, and admitted_at is on each record) instead of refusing calls whose commit order differs?
There was a problem hiding this comment.
Agreed—the sequential restriction was too strong. In the published update, independent calls can overlap, and RL uses validated parent ancestry rather than manifest completion order to reconstruct rows. The receipt identifies the scored response separately from selection order. Before finalization, RL atomically closes new admissions and drains already-admitted calls, preventing a late writer from racing cleanup. Tests cover both ledgers and both completion orders; the controlled GPU smoke covered overlapping calls, closure with calls pending, late-admission rejection, and once-only selected loss. This enables concurrent capture/reconstruction; it does not choose an all-captured training policy for subagents.
| # Exact prefix supply and capture share this decision. | ||
| if current_capture_context() is not None: | ||
| context = current_capture_context() | ||
| parent_header = request.headers.get(CAPTURE_PARENT_HEADER) |
There was a problem hiding this comment.
nemo_gym/base_responses_api_model.py:536
Is it necessary to carry x-nemo-gym-capture-parent as a field? Black-box harnesses cannot send it, so the gateway already has to be correct from text inference alone; a second, optional lineage path that only Gym-aware callers use adds a contract surface (and is currently read on Responses but not on Chat) without changing what the black-box path must handle.
There was a problem hiding this comment.
Agreed. Removed the header, decoder, explicit-parent override, and selection enum in the published update. Both Responses and Chat use ordinary request-history matching; equal-length ambiguous candidates still fail rather than guessing. Updated the separate native-agent client test to remove the deleted header contract.
Published in Gym f4565363. Native-agent test adjustment: 507d15c3.
| if parent_header is None: | ||
| await resolve_parent(request_messages) | ||
| else: | ||
| await resolve_parent(request_messages, parent_response_id=_decode_capture_parent(parent_header)) | ||
| await register_call_intent() |
There was a problem hiding this comment.
nemo_gym/base_responses_api_model.py:543
ValueErrors raised by _admit_candidate for client-attributable conditions (missing/ambiguous candidate, incomplete capture) surface here — and from measure_context — as bare 500s with no detail. Gym's own agent clients retry 500 (RETRY_ERROR_CODES includes 500), yet once an intent is pending the retry is guaranteed to fail, so each refusal costs a backoff cycle before the harness sees it. Mapping these to 409/422 would make them non-retryable and distinguishable from a gateway fault:
| if parent_header is None: | |
| await resolve_parent(request_messages) | |
| else: | |
| await resolve_parent(request_messages, parent_response_id=_decode_capture_parent(parent_header)) | |
| await register_call_intent() | |
| try: | |
| if parent_header is None: | |
| await resolve_parent(request_messages) | |
| else: | |
| await resolve_parent(request_messages, parent_response_id=_decode_capture_parent(parent_header)) | |
| except ValueError as error: | |
| raise HTTPException(status_code=409, detail=str(error)) from error | |
| await register_call_intent() |
There was a problem hiding this comment.
Implemented in the published update: expected admission conflicts use CaptureAdmissionError and return HTTP 409 on Responses, Chat, and measurement. Intent-registration conflicts are covered too; unrelated internal ValueErrors remain server faults. Rejected requests that never registered an intent no longer create a spurious failed-call record. HTTP tests cover the refusal behavior, including buffered Responses streaming before the stream is committed. Healthy pending sibling calls are now allowed rather than classified as incomplete-capture failures.
Published in Gym f4565363.
| context.capture_admission = CaptureAdmission( | ||
| **{**candidate.model_dump(exclude={"request_replay", "candidate_replay"}), "mode": "token_in"} | ||
| ) | ||
| handler.prepare_request({"model": "model", "messages": items}) |
There was a problem hiding this comment.
tests/unit_tests/test_framework_context_capture.py:45
Could a test be added where the worker continues the previous call instead of starting a new one? Every worker stub in the new tests roots: _serve acks with parent None / prev_len 0, and the HTTP harness converts each candidate to mode="text" unless decide_input is supplied (unused in Gym). So the ordinary-turn branch — external_capture.py committing a candidate as token_in with the chain extended, and _admit_candidate rebuilding a staging chain longer than one element — is never exercised. A splice=True flag on _serve that echoes the candidate's parent_call_id/prev_len into the coords covers it; this passes on both ledgers:
async def test_accepted_candidate_chains_records_and_rebuilds_the_full_staging_chain(ledger):
answer = [{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "answer"}]}]
history = [{"role": "user", "content": "task"}]
await _serve(ledger, "c1", history)
history = history + answer + [{"role": "user", "content": "more"}]
second, admission = await _serve(ledger, "c2", history, splice=True)
assert admission.parent_call_id == "c1" and admission.prev_len == 3
row = (await ledger.manifest("rollout-1"))["records"][1]
assert (row["parent_call_id"], row["mode"], row["prev_len"], row["cum_len"]) == ("c1", "token_in", 3, 6)
history = history + answer + [{"role": "user", "content": "again"}]
_, third = await _serve(ledger, "c3", history, splice=True)
assert third.staging_chain == ["rollout-1/c1", "rollout-1/c2"]There was a problem hiding this comment.
Added test_continuation_rebuilds_full_staging_chain in the published update, parametrized across in-memory and file-backed ledgers. The worker fixture can accept the candidate and commit token_in coordinates; three calls verify the parent/length metadata and reconstruction of the two-record staging chain. The paired RL/Gym tests also cover root -> append -> rewritten root -> append and compare the reconstructed tokens.
| rebuild_response: bool = True | ||
| # The framework compares source history and chooses physical capture roots. | ||
| # This is serving configuration, never a harness-provided segment decision. | ||
| framework_owned_context: bool = False |
There was a problem hiding this comment.
nemo_gym/token_id_capture/config.py:155
Could this surface be documented in fern/versions/latest/pages/model-server/model-call-capture.mdx (which already covers external_staging)? Specifically: framework_owned_context and its prerequisites (external_staging, external_staging_backend: vllm_worker, rebuild_response: false), what it refuses (/v1/messages, previous_response_id/conversation, Chat stream/n != 1), that a pending or failed call ends the attempt by design, POST /context/{rollout_id}/measure (called unprefixed), and run_examples(retry_requests=False). external-agent-harnesses.mdx is where harness authors would look for the framework-owned option.
There was a problem hiding this comment.
Updated both Fern pages with the framework-owned configuration prerequisites, supported/rejected API forms, the unprefixed measurement route, and run_examples(retry_requests=False). The concurrency wording reflects the revised implementation: healthy pending calls may overlap; failed captures poison the attempt, and finalization closes admissions before draining pending calls.
Published in Gym f4565363.
| def is_process_shared(self) -> bool: | ||
| return False | ||
|
|
||
| async def begin_call(self, rollout_id: str, model_call_id: str, expected_latest_call_id: str | None) -> None: |
There was a problem hiding this comment.
nemo_gym/token_id_capture/lineage.py:494
New public surface without docstrings: InMemoryLineageStore.begin_call here and FileLineageStore.begin_call at L930 (the only implementations of a new ledger operation), and retry_requests in openai_utils.py:1388 lacks the Field(description=...) its neighbours carry.
| async def begin_call(self, rollout_id: str, model_call_id: str, expected_latest_call_id: str | None) -> None: | |
| async def begin_call(self, rollout_id: str, model_call_id: str, expected_latest_call_id: str | None) -> None: | |
| """Append a durable intent row for ``model_call_id``. | |
| Fails if any call in the rollout is pending or failed, or if the ledger head is not | |
| ``expected_latest_call_id`` (another call committed since admission). | |
| """ |
There was a problem hiding this comment.
Added begin_call/close_admissions docstrings for both ledger implementations and a Field description for retry_requests in the published update. The begin_call contract now explicitly permits independent overlapping calls; the old expected-latest-call argument has been removed.
Published in Gym f4565363.
| "output_fingerprint", | ||
| "continuation_fingerprint", | ||
| "fingerprint_version", | ||
| "output_items", |
There was a problem hiding this comment.
nemo_gym/token_id_capture/lineage.py:90
A few small things, none blocking:
_CUSTODY_FIELDShas no readers (grepfinds only the definition) and now omitsreplay, which_custody_columnsdoes write — add it or drop the constant.lineage.py:121:record.replay.model_dump()/item.model_dump()embedschema_version: 3inside the on-diskreplayandoutput_itemscolumns, so the next bump turns v3 rows into aValidationErrorfrommanifest()rather than the fail-closed poison used for other columns;model_dump(exclude={"schema_version"})avoids coupling nested columns to the version.base_responses_api_model.py:410:body.get("n", 1) != 1rejects an explicit"n": null, which_validate_single_choice_token_requestaccepts —body.get("n") not in (None, 1).vllm_model/app.py:467:_resolve_clientis called inside the framework-owned context, so the idempotent/tokenizefor measurement also runs withretry_requests=False; resolving the client beforeset_token_sinkkeeps retries for measurement.rollout_collection.py:3675:**({"_retry": False} if not retry_requests else {})only exists to keep the mocked call signature stable;_retry=retry_requestsreads more clearly (adjust the test).
There was a problem hiding this comment.
All five cleanups are in the published update: removed the unused custody constant; excluded nested schema_version fields from persisted replay/output evidence and tested reload; accepted Chat n:null; resolved the measurement client before installing capture context so tokenization retains retries; and passed _retry=retry_requests directly with adjusted mocks.
Published in Gym f4565363.
Suggestion: one owner per check in the continuity decisionRight now the decision about whether a call continues a previous context is split. Gym proposes a parent and the worker (
Simplification: Gym decides everything it can see. The worker only confirms what it alone can see.
The conditions for splicing are the same as today, and the worker can still only downgrade. The difference is that each check runs in one place, and what Gym admits matches what gets recorded. Optionally, a |
|
@pthombre Following your continuity-ownership suggestion, the published update gives each check one owner: Gym matches source history and compares effective render options, emitting a direct text root when either cannot continue. Candidates carry an explicit flag when matching required null/empty tool-content equivalence. RL checks the converted-message boundary and actual renderer qualification, and can only downgrade the candidate to a root. Managed roots retain prefix stripping, capture/replay bookkeeping, and media validation. I left the optional root_reason metadata out to keep this change narrow. Validation includes paired lifecycle tests, a controlled concurrent GPU smoke, and passing 20-step manufactured identity/image-compaction and custom OpenCode compaction runs. The unchanged custom OpenCode identity repeat also passed all 20 steps with maximum TMPE 1.00178 and exact token/mask/logprob and selected-call coverage checks. An earlier identity run had a 1.1787 spike that remains unexplained; the bounded old/new comparison did not reproduce it, so I am not claiming a numerical bug was fixed. This evidence qualifies the tested fixtures and pinned fork, not arbitrary multi-agent harnesses. The changes are now pushed to this PR and RL #4124. I have left the review threads open for your review. Published in Gym f4565363. Paired RL changes: dae5a4ad5. Native-agent test adjustment: 507d15c3. |
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Preserve concurrent framework capture alongside upstream rollout timing, outbound telemetry, tool-call attribution, and canonical finish reasons. Keep retry suppression effective with metrics and tracing enabled. Validation: 2,475 affected Gym CPU tests and 184 paired RL compatibility tests passed; two optional custom OpenCode exporter cases skipped. Full pre-commit passed. Fern check passed with the unauthenticated redirect check skipped. Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
| raise CaptureAdmissionError("Capture admissions are closed for this rollout") | ||
| if any(row.get("model_call_id") == model_call_id for row in rows): | ||
| raise CaptureAdmissionError(f"Capture call ID already registered: {model_call_id}") | ||
| if any(row.get("failure_reason") is not None for row in rows): |
There was a problem hiding this comment.
Once any call in an attempt has a failure row, this check refuses every later begin_call in the same attempt with a 409. _admit_candidate refuses the same calls again at sink.py#L355-L356, with a different message ("incomplete capture" there, "failed capture" here). The refused calls are never generated, so they are missing from the ledger entirely.
In practice one failed or cancelled call ends the rollout. With parallel tool calls or subagents, a transport error on one branch refuses every other branch's next call, including calls that start new contexts and have nothing to do with the failed one:
sequenceDiagram
participant Child as "Subagent call"
participant Parent as "Parent call"
participant Gym as "Gym admission"
participant Ledger as "Ledger"
Child->>Gym: "request"
Gym->>Ledger: "intent row"
Note over Child,Gym: "worker or transport fails"
Gym->>Ledger: "failure row"
Parent->>Gym: "next request, unrelated context"
Gym-->>Parent: "409, never generated"
The context-overflow and request-check cases in the other comments become dead rollouts for the same reason.
The refusal is not needed for correctness. A later call that would have continued the failed call cannot match its prefix, because the failed call has no committed replay, so it is admitted as a new root. The failure row is already in the manifest, and NeMo RL already masks attempts that have failures or pending calls.
Could both checks be removed, keeping the failure rows as recorded outcomes? The trainer can then decide whether to mask the whole attempt or drop only the failed chain.
There was a problem hiding this comment.
Removed both attempt-wide admission blocks. Later independent calls can proceed; failed and pending calls remain in the manifest rather than being discarded. NeMo-RL still masks an incomplete attempt, so this does not introduce selective failed-chain training. Regression tests.
There was a problem hiding this comment.
Confirmed at 09adc12: a failure row no longer refuses later calls. _validate_call_intent now checks only closed admissions and duplicate ids (lineage.py#L234-L238). Our original reproduction now shows the failed subagent call's row in the manifest while the parent's next call and a fresh root both return 200.
| await self._external_capture_handler.finalize_response(_jsonable(response)) | ||
| context = current_capture_context() | ||
| if context is not None and context.framework_owned_context and not context.committed: | ||
| raise RuntimeError("Framework-owned context requires a committed capture before serving a response") |
There was a problem hiding this comment.
When the prompt exceeds the context window, the worker refuses the request before generating or staging anything. With the default propagate_context_overflow_errors: false, the model server then builds a local response with finish_reason="length", and this guard raises because nothing was committed, so the harness receives a 500. The middleware writes a failure row for the call, and because of the check at lineage.py#L259-L260 the harness's next call gets a 409:
sequenceDiagram
participant Harness as "Harness"
participant Gym as "Gym vLLM model server"
participant Worker as "Inference worker"
participant Ledger as "Ledger"
Harness->>Gym: "call with long history"
Gym->>Ledger: "intent row"
Gym->>Worker: "generate"
Worker-->>Gym: "context length exceeded, nothing staged"
Gym-->>Harness: "500 from the committed-capture guard"
Gym->>Ledger: "failure row"
Harness->>Gym: "compacted retry"
Gym-->>Harness: "409, attempt refused"
That next call is usually the compacted retry. Codex, OpenCode and Gym's built-in compaction agent all compact when they see a context-length error, so with framework-owned context the compaction flow this PR is meant to support ends the rollout instead. Setting propagate_context_overflow_errors: true does not help: the harness gets the 400, but the failure row still blocks the retry. A streaming request gets response.failed and then the same 409.
Could the guard let the response through when the worker refused before generating, and could that call be recorded as not generated rather than as a failure? With the check at lineage.py#L259 removed, the compacted retry would then be admitted as a new root.
There was a problem hiding this comment.
Fixed the overflow/retry path: framework mode preserves the recognizable HTTP 400 context-length refusal, regardless of propagate_context_overflow_errors. A matching worker acknowledgement settles the pre-generation intent while retaining its audit row; unacknowledged errors remain incomplete. The paired worker acknowledgement is in NVIDIA-NeMo/RL#4124; harnesses need no new header. Regression tests.
There was a problem hiding this comment.
Confirmed at 09adc12: with the worker acknowledgement, the overflow returns a 400 context_length_exceeded for both propagation settings and both streaming modes. The call is settled as refused_before_generation, and the compacted retry commits as a new root (app.py#L539-L556). We also checked the NeMo RL side in NVIDIA-NeMo/RL#4124 end to end.
Optional: the refused call does not appear in RolloutManifest, and the retry reads root_reason="first_call", so a trainer cannot see that a compaction followed an overflow. A refused_call_ids list would make that visible.
There was a problem hiding this comment.
Keeping refused_call_ids out of this update: it would be additional trainer-visible diagnostics, and the confirmed pre-generation refusal already settles safely and permits the compacted retry. The ledger audit row is retained. root_reason="first_call" continues to mean no committed predecessor, not proof that no earlier request was refused; the root reasons are documented as observations rather than compaction attribution.
| if current_capture_context() is not None: | ||
| context = current_capture_context() | ||
| if context is not None: | ||
| request_messages = _request_messages(params) |
There was a problem hiding this comment.
Framework-owned admission builds its replay evidence from _request_messages(params), and that helper (L219-L226) only reads input when it is a list. A Responses request with a string input, which is the most common form, therefore contributes no user item to the evidence.
This has two effects. If the first call sends input: "old question", its committed replay contains only the assistant answer, so a follow-up that sends the full history as a list no longer matches and silently becomes a new root. Ordinary capture chains the same requests correctly.
The second effect is worse. If the first call sends input: "question A" and a later call sends [<first call's output>, {"role": "user", "content": "unrelated"}], the later request matches the first call's replay and is proposed as a candidate continuing it. NeMo RL's decide_capture_input checks the message roles at the boundary, not the prefix content, so it accepts the candidate and splices the first call's prompt tokens, including "question A", into a context the harness never sent. This is the shape a compaction harness produces when it keeps only the last answer plus a short continuation prompt.
Could _request_messages turn a string input into [{"role": "user", "content": input}]? The measurement route uses the same helper, so it is fixed too.
There was a problem hiding this comment.
Fixed _request_messages to normalize string input into a user message. Measurement and generation now use the same evidence; the regression checks both a valid continuation and requests that omit or change the original question, which must start new roots. Regression tests.
There was a problem hiding this comment.
Confirmed at 09adc12: a string input becomes a user message (base_responses_api_model.py#L216-L218). The full-history follow-up continues as a candidate, and the request that omits the original question now starts a new root.
| admission = context.capture_admission | ||
| if admission is None: | ||
| return request_payload | ||
| if context.framework_owned_context: |
There was a problem hiding this comment.
These checks run inside the model handler, after register_call_intent has already written the intent row. When one of them raises, the call ends with a 500, and the middleware records request_finished_without_staged_coordinates even though the worker was never called. Every later call in the attempt then gets a 409.
Two paths reach these checks today. A server configured with continue_final_assistant_message: true fails on any request that ends with an assistant message, and a request whose metadata.extra_body carries truncate_prompt_tokens fails on its first call. The direct body fields are already rejected with a 422 by the request schema, so only these indirect paths get through.
Could these checks move before intent registration and return a 422, next to the stream and history gates in the dispatch methods? A config validator that rejects continue_final_assistant_message and extra_body.truncate_prompt_tokens together with framework_owned_context would also catch the server-level case at startup.
There was a problem hiding this comment.
Moved request-option validation before intent registration, returning HTTP 422 for unsupported metadata.extra_body options. Incompatible server-level options now fail at startup. Tests verify no worker call or capture intent is created, and a corrected request can proceed in the same attempt. Regression tests.
There was a problem hiding this comment.
Confirmed at 09adc12: metadata.extra_body truncation returns a 422 with no intent and no worker call (app.py#L518-L530), and continue_final_assistant_message and truncation in server options now fail at startup. Request conversion errors still happen after the intent; the new comment on app.py#L767 covers that case.
| @model_validator(mode="after") | ||
| def _validate(self) -> "TokenIdCaptureConfig": | ||
| block = self.token_id_capture | ||
| if block.framework_owned_context and ( |
There was a problem hiding this comment.
This validator checks the staging backend, but it does not reject framework_owned_context together with a checkpoint policy on the same model server, and that combination fails every call.
With the checkpoint gate attached, finalize_response defers the ledger row until the response starts, so committed is still false when the committed-capture guard in vllm_model/app.py runs. Every non-streaming call returns a 500. The failure is not recorded either, because _fail_uncommitted_external_call skips calls with a deferred commit, so the call stays in pending_call_ids forever.
Streaming fails differently but is also wrong. The harness gets response.failed, while the gate still publishes the deferred row, so the ledger records an answer the harness never received, and a retry adds a second root.
A restored attempt has a related problem: checkpoint export keeps the {"admissions_closed": true} row (_checkpoint/model.py#L629-L636), so the restored attempt refuses its first call.
NeMo RL refuses checkpointing in this mode on its side, but nothing in Gym enforces it. Could the configuration reject the combination, with one line in the docs, until deferred commits work with framework-owned context?
There was a problem hiding this comment.
Fixed deferred publication and restore rather than rejecting this combination. The guard recognizes a deferred capture, the checkpoint gate publishes the ledger record before forwarding the first answer chunk, and restore does not inherit closed admissions. Regression tests cover streaming/non-streaming delivery, publication failure, pending state, and a restored closed attempt; these are component checks, not a claim of arbitrary distributed checkpoint compatibility.
There was a problem hiding this comment.
Confirmed at 09adc12: non-streaming and streaming calls behind the checkpoint gate return 200, with the ledger row published before the first answer chunk (_checkpoint/model.py#L930-L936). A failed publication sends no answer, and a restored attempt admits its next call, including from an older export that still carries the closure row. Supporting the combination is better than rejecting it, thanks.
| elif kind == "function_call_output": | ||
| role = "tool" | ||
| else: | ||
| raise ValueError(f"Unsupported replay item kind: {kind}") |
There was a problem hiding this comment.
A Responses request containing an item kind this function does not know, such as web_search_call or custom_tool_call, raises a plain ValueError here. The client receives a 500, which SDKs retry. Ordinary capture also fails on these items, so this is not a regression, but a 422 would tell the harness the request will never be accepted. Could it return a 422, and could the docs table list the unsupported item kinds?
There was a problem hiding this comment.
Unsupported replay item kinds are now mapped to HTTP 422 before intent registration instead of leaking a ValueError as HTTP 500. Regression tests verify the response, no generation/capture writes, and subsequent reuse of the attempt. The supported-input table lists the unsupported kinds.
There was a problem hiding this comment.
Confirmed at 09adc12: unsupported item kinds return a 422 before the intent, and the docs table lists them.
| for record in manifest.records: | ||
| replay = record.replay | ||
| if replay is None: | ||
| raise ValueError("Captured call is missing source replay evidence") |
There was a problem hiding this comment.
If the ledger contains a committed row without replay, every later admission in the attempt raises this plain ValueError. That happens after restoring a ledger written without framework-owned context, or after turning the flag on for an existing ledger. The client gets a 500, nothing is recorded because the intent is not registered yet, and SDKs keep retrying. The cyclic and missing-ancestor errors further down behave the same way. Could records without replay evidence be skipped as candidates, or could these raise CaptureAdmissionError?
There was a problem hiding this comment.
Converted missing replay, cyclic-chain, and missing-ancestor cases to CaptureAdmissionError (HTTP 409) before intent registration. I chose explicit refusal over skipping evidence and guessing a different parent; the ledger is preserved for diagnosis. Regression tests verify repeated dispatch/measurement refusals without worker calls or ledger mutation.
There was a problem hiding this comment.
Confirmed at 09adc12: these now return a 409 before the intent, with the ledger unchanged (sink.py#L367-L369). Refusing loudly on a mixed or corrupt ledger is fine.
| then drains already-admitted calls through the manifest endpoint. Closure refuses new generation but | ||
| allows admitted calls to finish. Missing acknowledgements remain pending; failure is not permission | ||
| to delete data that a worker may still be writing. NeMo RL masks an incomplete attempt. | ||
| Closed, failed, or ambiguous admission returns HTTP 409 with a reason instead of a retryable 500. |
There was a problem hiding this comment.
Docs suggestion: several framework-mode refusals still return a 500 rather than a 409, namely a context-length overflow, the truncation and prefill checks in prepare_request, unsupported input item kinds, and a ledger row without replay evidence. The OpenAI, Anthropic and Vercel AI SDKs also retry a 409. Could this sentence say which cases return a 409? The section could also state that framework-owned context cannot be combined with the checkpoint gate, and that the measurement route sits outside the rollout prefix without the control bearer token.
There was a problem hiding this comment.
Updated the documentation to distinguish HTTP 422 invalid/unsupported input, HTTP 409 closed/ambiguous/unusable history, and acknowledged HTTP 400 context-length refusals, including SDK retries of 409. It also describes the now-supported checkpoint gate and rollout-prefixed measurement route; measurement is a model route, whereas the control endpoints require the control bearer token.
There was a problem hiding this comment.
Confirmed at 09adc12: the section now separates the 422, 409 and acknowledged 400 cases, warns about SDK retries of 409, and describes the checkpoint gate and the measurement route (model-call-capture.mdx#L148-L180).
| await next(RolloutCollectionHelper().run_examples([failing_row()])) | ||
|
|
||
| @pytest.mark.parametrize("retry_requests", [True, False]) | ||
| async def test_run_examples_controls_transport_replay( |
There was a problem hiding this comment.
post is mocked at the ServerClient level, which is above the retry loop, so it is awaited once whether retry_requests is True or False. The True case therefore passes even when retries are disabled, and the test only shows that the keyword is forwarded. Could it use a fake that fails once at the transport layer and assert one attempt for False and two for True? The docstring also says "CC callers", which would read better as "framework callers".
There was a problem hiding this comment.
Replaced the ServerClient.post mock with a fake underlying transport that disconnects once and then succeeds, exercising the real ServerClient retry loop. The test now asserts one transport attempt with retries disabled and two when enabled, and uses “framework callers” wording. Updated test.
There was a problem hiding this comment.
Confirmed at 09adc12: the test goes through the real retry loop, and it fails when _retry is ignored.
|
|
||
| @pytest.mark.parametrize("route", ["responses", "responses-stream", "chat/completions", "measure"]) | ||
| @pytest.mark.parametrize("failure", ["ambiguous", "failed"]) | ||
| def test_capture_refusals_are_non_retryable(monkeypatch, tmp_path, route, failure): |
There was a problem hiding this comment.
The name says these refusals are non-retryable, but the OpenAI and Anthropic Python SDKs and the Vercel AI SDK that OpenCode uses all retry a 409; only Gym's NeMoGymAsyncOpenAI treats it as final. The refusals tested here happen before an intent is written, so a retry is harmless. Could the name say that instead, for example test_capture_refusals_happen_before_generation?
There was a problem hiding this comment.
Removed the non-retryable claim from the tests. The ambiguous-admission test verifies refusal before generation, with separate closed-admission coverage. The docs warn that external SDKs may retry HTTP 409; these refusals create no intent and do not dispatch inference.
There was a problem hiding this comment.
Confirmed at 09adc12: the names now describe refusal before generation, and the docs cover SDK retries.
Normalize string input as a user message before capture matching and measurement. Verify unchanged history continues while omitted or changed initial questions start new roots. Validation: 840 affected Gym CPU tests, three HTTP prefix checks using the actual RL input decision, and pre-commit checks passed. Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
ananthsub
left a comment
There was a problem hiding this comment.
@aroshanghias-nvd please address the merge conflicts as well
| chat_completion_create_params = self._converter.responses_to_chat_completion_create_params( | ||
| body, drop_unrepresentable_request_fields=self.config.drop_unrepresentable_request_fields | ||
| ) | ||
| chat_completion_create_params = self._responses_to_chat_request(body) |
There was a problem hiding this comment.
Framework generation converts the Responses body here, after dispatch has already registered the call intent at base_responses_api_model.py#L736. A conversion error therefore reaches the client as a 500, and the middleware records request_finished_without_staged_coordinates for a call the worker never saw.
Two requests show it. A user message with an input_file part raises NotImplementedError("Unsupported part param type: input_file"), and tool_choice: "required" without tools raises ValueError. In both cases the manifest ends with a pending call plus a failure row, so NeMo RL masks the attempt, and an SDK that retries the 500 adds another intent and failure row each time. Measurement maps the same errors to a 422 before anything is written (app.py#L621-L625).
Could framework mode run _responses_to_chat_request inside _validate_capture_request, before the intent, and map ValueError and NotImplementedError to a 422? A test that asserts the 422 and an empty manifest would pin it down.
There was a problem hiding this comment.
Moved Responses-to-Chat conversion into _validate_capture_request, before intent registration, and map ValueError/NotImplementedError to HTTP 422. Regression coverage checks both examples through buffered/streaming Responses and measurement, with and without existing history: the manifest and worker-call count stay unchanged, and a corrected request can reuse the attempt. Published in c566cd5.
| raise CaptureAdmissionError("Capture admissions are closed for this rollout") | ||
| if any(row.get("model_call_id") == model_call_id for row in rows): | ||
| raise CaptureAdmissionError(f"Capture call ID already registered: {model_call_id}") | ||
| if any(row.get("failure_reason") is not None for row in rows): |
There was a problem hiding this comment.
Confirmed at 09adc12: a failure row no longer refuses later calls. _validate_call_intent now checks only closed admissions and duplicate ids (lineage.py#L234-L238). Our original reproduction now shows the failed subagent call's row in the manifest while the parent's next call and a fresh root both return 200.
| await self._external_capture_handler.finalize_response(_jsonable(response)) | ||
| context = current_capture_context() | ||
| if context is not None and context.framework_owned_context and not context.committed: | ||
| raise RuntimeError("Framework-owned context requires a committed capture before serving a response") |
There was a problem hiding this comment.
Confirmed at 09adc12: with the worker acknowledgement, the overflow returns a 400 context_length_exceeded for both propagation settings and both streaming modes. The call is settled as refused_before_generation, and the compacted retry commits as a new root (app.py#L539-L556). We also checked the NeMo RL side in NVIDIA-NeMo/RL#4124 end to end.
Optional: the refused call does not appear in RolloutManifest, and the retry reads root_reason="first_call", so a trainer cannot see that a compaction followed an overflow. A refused_call_ids list would make that visible.
| if current_capture_context() is not None: | ||
| context = current_capture_context() | ||
| if context is not None: | ||
| request_messages = _request_messages(params) |
There was a problem hiding this comment.
Confirmed at 09adc12: a string input becomes a user message (base_responses_api_model.py#L216-L218). The full-history follow-up continues as a candidate, and the request that omits the original question now starts a new root.
| admission = context.capture_admission | ||
| if admission is None: | ||
| return request_payload | ||
| if context.framework_owned_context: |
There was a problem hiding this comment.
Confirmed at 09adc12: metadata.extra_body truncation returns a 422 with no intent and no worker call (app.py#L518-L530), and continue_final_assistant_message and truncation in server options now fail at startup. Request conversion errors still happen after the intent; the new comment on app.py#L767 covers that case.
| elif kind == "function_call_output": | ||
| role = "tool" | ||
| else: | ||
| raise ValueError(f"Unsupported replay item kind: {kind}") |
There was a problem hiding this comment.
Confirmed at 09adc12: unsupported item kinds return a 422 before the intent, and the docs table lists them.
| for record in manifest.records: | ||
| replay = record.replay | ||
| if replay is None: | ||
| raise ValueError("Captured call is missing source replay evidence") |
There was a problem hiding this comment.
Confirmed at 09adc12: these now return a 409 before the intent, with the ledger unchanged (sink.py#L367-L369). Refusing loudly on a mixed or corrupt ledger is fine.
| then drains already-admitted calls through the manifest endpoint. Closure refuses new generation but | ||
| allows admitted calls to finish. Missing acknowledgements remain pending; failure is not permission | ||
| to delete data that a worker may still be writing. NeMo RL masks an incomplete attempt. | ||
| Closed, failed, or ambiguous admission returns HTTP 409 with a reason instead of a retryable 500. |
There was a problem hiding this comment.
Confirmed at 09adc12: the section now separates the 422, 409 and acknowledged 400 cases, warns about SDK retries of 409, and describes the checkpoint gate and the measurement route (model-call-capture.mdx#L148-L180).
| await next(RolloutCollectionHelper().run_examples([failing_row()])) | ||
|
|
||
| @pytest.mark.parametrize("retry_requests", [True, False]) | ||
| async def test_run_examples_controls_transport_replay( |
There was a problem hiding this comment.
Confirmed at 09adc12: the test goes through the real retry loop, and it fails when _retry is ignored.
|
|
||
| @pytest.mark.parametrize("route", ["responses", "responses-stream", "chat/completions", "measure"]) | ||
| @pytest.mark.parametrize("failure", ["ambiguous", "failed"]) | ||
| def test_capture_refusals_are_non_retryable(monkeypatch, tmp_path, route, failure): |
There was a problem hiding this comment.
Confirmed at 09adc12: the names now describe refusal before generation, and the docs cover SDK retries.
|
🌿 Preview your docs: https://nvidia-preview-pr-4128-a058736f12b4.docs.buildwithfern.com/nemo/gym |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40419833248 |
…r than 64 bits The fingerprint fallback for integers beyond orjson's 64-bit range landed on main in #4128. This keeps the test that a tool call whose arguments hold such an integer still fingerprints, matches its reserialized form, and changes when the value changes. Signed-off-by: Christian Munley <cmunley@nvidia.com>
* fix(capture): accept capture key names from before the rename #4128 renamed rollout_id to capture_key on the capture wire models and the old CallRecord.capture_key to source_capture_key. Frameworks still on the old names, including NeMo RL main, now fail every rollout with AttributeError or a rejected extra field. Accept the old names as validation aliases and expose them as read-only properties. Serialization still writes only the new names, and a payload that carries both names is rejected. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * fix(capture): warn once when pre-rename capture names are used Emit a FutureWarning the first time a process passes or reads rollout_id or CallRecord.capture_key, naming the field that replaced it, so frameworks still on the old names learn to migrate without flooding their logs. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…r than 64 bits The fingerprint fallback for integers beyond orjson's 64-bit range landed on main in #4128. This keeps the test that a tool call whose arguments hold such an integer still fingerprints, matches its reserialized form, and changes when the value changes. Signed-off-by: Christian Munley <cmunley@nvidia.com>
…64 bits (NVIDIA-NeMo#5049) * test(token_id_capture): fingerprint tool arguments with integers wider than 64 bits The fingerprint fallback for integers beyond orjson's 64-bit range landed on main in NVIDIA-NeMo#4128. This keeps the test that a tool call whose arguments hold such an integer still fingerprints, matches its reserialized form, and changes when the value changes. Signed-off-by: Christian Munley <cmunley@nvidia.com> * fix(server_utils): send request bodies that hold integers wider than 64 bits The outbound JSON body in request() is encoded with orjson, which rejects integers outside the 64-bit range. A tool call whose arguments carry one therefore failed before it was sent, the same way the token-capture fingerprint did. Fall back to the stdlib encoder, compact and non-ASCII preserved, which accepts them. Signed-off-by: Christian Munley <cmunley@nvidia.com> * fix(server_utils): escape non-ASCII text in the fallback request body With ensure_ascii=False the fallback still raised UnicodeEncodeError for a lone surrogate, which orjson also rejects and a model can write into a tool argument. The default escaping sends it, and any other non-ASCII text, as valid JSON. Signed-off-by: Christian Munley <cmunley@nvidia.com> --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
QA verified: PR 4128 coverage — feat(capture): support framework-owned context reconstruction
Existing coverage
Contract
Doc check (2026-11-12, workflow wf_554f4197-ab2, Gym 5d22439)model-call-capture.mdx L65-196 + external-agent-harnesses.mdx L405-423. 5/5 checkers returned (26/34/30/24/34 Decisions
Design (Gym-only case
|
| # | Gym | Cases | Result | Run |
|---|---|---|---|---|
| 1 | 0a5110a6c |
both | ENV: h100 uv 0.12.20 < Gym main's required 0.13.4; upgraded bootstrap venv uv → 0.13.8 | 20261112T184424576448Z-4a650ffc |
| 2 | 0a5110a6c |
both | BLOCKED before Gym ran: vllm serve exits in 15 s, ModuleNotFoundError: Could not import module 'MistralCommonBackend' ← pycountry missing from the local_vllm_model venv. Same for token_capture_external_staging (not caused by this change) |
20261112T184514318207Z-57419a36 |
| 3 | 0a5110a6c |
both | BLOCKED: engine init No module named 'torchvision' (pycountry workaround in) |
20261112T190328741843Z-69ed0590 |
| 4 | 0a5110a6c |
both | harness PASS; external_staging PASS; README FAIL = test bug: ledger is <key>.lineage.jsonl, case globbed *.tokens.jsonl. Real data: qa-sh-4 tool call → continuation admitted as candidate (parent = first call) → worker_rooted; 4 rollouts answered without tools. Fixed glob; 5 repeats per example |
20261112T190745945428Z-ea473c29 |
| 5 | efaa56a17 |
both | harness PASS; external_staging PASS; README FAIL = test bug in the stand-in's verify: it passed all 39 staged rows (25 rollouts) for a 2-row manifest (row_count_mismatch). Fixed: filter staged rows by capture_key. Evidence: 10/25 rollouts called tools, 14 candidate admissions |
20261112T191327438857Z-cb87ec63 |
| 6 | efaa56a17 |
all 3 | PASS (README 73.3 s, harness 45.1 s, external_staging 41.8 s), pycountry + torchvision workaround | 20261112T191640267263Z-f8f15a82 |
| control | efaa56a17 |
source_history case with parent_resolution: fingerprint (1-line copy, direct run) |
FAIL ×2 on behavior: first call has no root_reason; first continuation is a Gym-chosen token_in staging_chain admission, which the source-history worker refuses (CaptureError … prefix_token_ids) → 500 |
direct 20261112T191949Z, 20261112T192231Z (kept) |
Registry: status: fail, nvbugs: ["7087161"] (open), reason in the docstring. Branch commit c1990fee on
nmfw_tests main a9f72574; MR body mr-body.md; NVBug comment draft
../nvb-7087161-local-vllm-pycountry/nvbug-comment-torchvision.html.
Blocker: vLLM 0.29 cannot start from a Gym-main local_vllm_model venv (2026-11-12)
Filed as NVBug 7087161 (board bugs/nvb-7087161-local-vllm-pycountry.md). Workaround per user: uv pip install --no-config --no-deps "pycountry==26.2.16" into h100's checkout venv for 0a5110a6cb (survives Gym's --inexact
pre-warm). Runs from 3 on carry this workaround; the MR says so. Run 3 then died on torchvision (same
never-install group, imported at engine init); from run 4 the venv also has torchvision==0.28.0+cu130 (cu130 index,
--no-deps).
- Cause (observed): Gym build(env): AUT-2574 lock managed environments and use uv sync #4837 (
1203b0be21, 2026-11-04) added"pycountry; sys_platform == 'never'"to
responses_api_models/local_vllm_model/pyproject.tomloverride-dependencies(legal exclusion, security: remove dependencies flagged in legal review #2492). vLLM 0.29
backend_guidance→transformers.MistralCommonBackend→mistral_common→pydantic_extra_types.language_code
→pycountry. Reproduced in isolation on h100:python -c "from vllm.v1.structured_output.backend_guidance import GuidanceBackend"in that venv fails the same way. - Gym's own GPU e2e works around it:
tests/e2e/gpu_e2e_test.sh:143-146installspycountry==26.2.16"even for
non-Mistral models".docker/install_excluded_deps.sh(the documented opt-in) says pycountry is for VLMs and is
documented for containers only;local_vllm_model/README.mddoes not mention it. - Suite:
lib/image_requirements.pyruns that installer only whenNEMO_GYM_CONTAINER=1; checkout-lane vLLM cases
get nothing, so everyvllmcase on the h100 lane is blocked on Gym main since 11-04 (board trap
environment-traps.md:238recorded it for the nightly image only, when main still had pycountry).
Review (fresh agent, 2026-11-12): Minor fix needed
- Needs run evidence on main + a control (pending; blocked above).
- Case 1
continued > 0depends on Qwen3-0.6B calling a tool on ≥1 of 5 tasks → decide after the first real run. - Measure assertion
prompt_tokens == worker countcould not fail → dropped; the served-prompt equality stays.
Worker protocol, root reasons, manifest fields, measure path and the external-staging refactor: no issue found.
Handoff
2026-11-12: MR !539 open (branch
qiaochuz/nemo-gym-pr4128-framework-context, commit c1990fee on nmfw_tests main a9f72574). NVBug 7087161 comment
#1 posted. Next:
/gym-review-bots !539once Greptile/CodeRabbit have posted.- When NVBug 7087161 is fixed: rerun on h100 without the workaround, then flip the registry to
passand drop
nvbugs(a/gym-merged-style follow-up MR). - NeMo-RL compaction run against Gym main: proposed as its own session (card "NeMo-RL compaction run vs Gym main").
- h100 state left behind: uv 0.13.8 in
~/.local/share/nemo-qa-bootstrap; pycountry + torchvision in the
local_vllm_modelvenvs of checkouts0a5110a6cb…andefaa56a175…; landing worktree
~/nemo/workdir/pr4128-framework-context/nmfw_tests(itsconfig/config.pyis modified by the runner).
Training frameworks need to distinguish an exact captured-prefix continuation from a rewritten context without requiring harnesses to define training segments. Gym now carries replay evidence, completion identity, and committed capture coordinates through its shared ledger and ordinary vLLM server. Harnesses can own compaction while the training framework owns reconstruction and training-call selection.
Framework-owned capture validates requests before recording call intents, supports overlapping independent calls, and exposes exact context measurement under the rollout route. Buffered Responses and Chat streaming commit capture before returning answer chunks, including behind the checkpoint gate. Confirmed context-length refusals before generation are settled so a harness can compact and retry; cancellation preserves uncertain-call failure evidence. Ambiguous source-prefix matches start fresh roots with
root_reason="ambiguous_candidate"and no guessed parent. Training selection must still validate accepted-call membership independently.Captured generation and rollout requests reuse the existing connection-safe retry control: connections that were never established may retry, while requests that may have been delivered are not resent. The duplicate
_retrymechanism is removed. Ordinary tokenization retains its retries.Compatibility: capture's staging wire schema changes to version 3. Gym and every external-staging worker must upgrade together, including for ordinary capture. Version 2 admissions, staged rows, receipts, and checkpoints are incompatible; no migration is provided. Finish or retire existing attempts before upgrading, and do not resume their persisted version 2 data.
Part 1 of 2: Gym #3368 provides the optional compaction-agent migration and depends on this PR. NeMo-RL #4124 uses this shared infrastructure; external harnesses that own compaction do not need #3368. The existing compaction agent/model remain in this lower layer until the migration merges.
Validation of the current update:
pre-commit run --all-filespassed. The private guardword catalog was unavailable during the infrastructure check; the hook skipped that private scan.ci: bump cherry-pick workflow to v1.12.2, alert Gym channel (4119)intor0.7.0#4124 completed a two-optimizer-step GPU smoke: four rollouts with 20 internal turns each, 80 selected calls, and 40 training rows. Saved-evidence audits passed exact token/prefix/delta, mask, media, reward, and exact-once selected-output checks. Both optimizer updates had finite nonzero gradients. Peak TMPE was 1.000004971. W&B evidence.The smoke uses manufactured image observations and synthetic rewards with real model generation and learning. It is a bounded integration check, not a new 20-optimizer-step qualification or a concurrent-harness benchmark. The optional agent layer separately passed 254 native/parity/checkpoint CPU tests, including six real transport retry regressions. Repository CI remains required before merge.
Merged Gym main at
ed830b41, preserving upstream rollout tracing and retry behavior. No dependency, lockfile, or secrets-baseline changes relative to that main revision are included.