Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
66ce5c0
feat(capture): support framework-owned context reconstruction
aroshanghias-nvd Oct 6, 2026
f456536
fix(capture): allow concurrent calls and own continuity checks
aroshanghias-nvd Oct 13, 2026
92e4300
merge: reconcile capture infrastructure with current Gym main
aroshanghias-nvd Oct 13, 2026
b53e7b7
merge: reconcile capture infrastructure with current Gym main
aroshanghias-nvd Oct 16, 2026
dd713d2
refactor(capture): remove unused custody field list restored by merge
aroshanghias-nvd Oct 16, 2026
0bc517d
fix(capture): preserve Responses string input in context evidence
aroshanghias-nvd Oct 19, 2026
5560478
fix(capture): address framework context review boundaries
aroshanghias-nvd Oct 19, 2026
52f4335
merge: preserve capture review fixes on current Gym main
aroshanghias-nvd Oct 19, 2026
81b8129
merge: preserve capture retry controls with rollout request IDs
aroshanghias-nvd Oct 20, 2026
09adc12
merge: retain capture single-attempt policy with provider backoff
aroshanghias-nvd Oct 20, 2026
c566cd5
fix(capture): validate admission and preserve safe recovery
aroshanghias-nvd Oct 21, 2026
0526500
Merge main into capture infrastructure
aroshanghias-nvd Oct 26, 2026
dd98800
refactor(capture): name source-history parent resolution explicitly
aroshanghias-nvd Oct 26, 2026
33f9232
test(capture): preserve tracing and replay controls across dispatch
aroshanghias-nvd Oct 26, 2026
86dff9f
Merge main refusal capture while preserving source-history acknowledg…
aroshanghias-nvd Oct 26, 2026
a9052b3
merge: preserve capture refusals with upstream overflow classification
aroshanghias-nvd Oct 26, 2026
a058736
refactor(capture): distinguish capture ownership from rollout identity
aroshanghias-nvd Oct 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 57 additions & 1 deletion fern/versions/latest/pages/model-server/model-call-capture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ original request and response payloads.

### Buffered SSE with external token staging

With `token_id_capture.external_staging: true`, captured requests can use `stream: true`
With `token_id_capture.external_staging: true` and `framework_owned_context: false`, captured requests can use `stream: true`
on Chat Completions, Responses (including the compaction adapter), and Anthropic Messages.
Gym requests a complete, non-streaming answer from the inference worker. The worker stages
training tokens before returning commit coordinates. Gym removes internal token fields,
Expand All @@ -84,6 +84,62 @@ the staged rows must run the same Gym version. A framework whose Gym predates a
server writes, such as `capture_key` or `finish_reason`, rejects every manifest row as
`invalid_manifest_row` and masks the rollout.

### Framework-owned context and compaction

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming suggestion, not a blocker: elsewhere on this page "framework-owned" means the training framework owns storage ("Framework-owned media staging"), which matches external_staging. In this section Gym makes the continuity decision itself, so "Framework-owned context" suggests the opposite ownership. If the setting is renamed (see the comment on config.py), a heading such as "Resolve parents from the source history" or "Source-history continuity and context compaction" would describe the behavior directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the heading to “Source-history continuity and context compaction,” updated the configuration examples and incoming anchor link, and clarified Gym’s source-history matching versus the worker’s exact-token verification. Published in 86dff9f. Fern check passed; the authenticated redirect audit was skipped because the local Fern CLI was not authenticated.


A compatible training framework can own exact-token prefix reuse and training-row reconstruction.
Gym matches source history and effective render options, sending a root when they do not match.
For a candidate parent, the inference worker verifies the converted-message boundary and any
required null/empty tool-content equivalence before restoring exact tokens. The harness sends its chosen
message history, including any compaction. It does not send tokens, segment IDs, or a parent header.

Framework candidates carry an explicit source-match proof. Deploy compatible Gym and RL worker
revisions together: a missing proof is an error, not permission to assume an exact match.
Direct roots retain replay evidence and framework safety checks so later calls can continue them.

The framework must configure the worker, shared capture ledger and control bearer token as well as
these run-wide settings:

```yaml
token_id_capture:
enabled: true
external_staging: true
external_staging_backend: vllm_worker
rebuild_response: false
framework_owned_context: true
```

This requires a framework worker implementing Gym's capture protocol; stock vLLM alone is insufficient.
Keep the usual per-agent capture opt-in or `all_agents: true`. `rebuild_response: false` preserves
ordinary rollout output for framework reconstruction instead of Gym's single-chain rebuilding.

| Request | Framework-owned behavior |
|---|---|
| Responses with explicit history | Supported, including buffered SSE. |
| Chat Completions with explicit history | Non-streaming, one choice; omitted or null `n` means one. |
| `/v1/messages` | Refused with HTTP 422. |
| Responses `previous_response_id` or `conversation` | Refused with HTTP 422; full input must be visible. |
| Chat `stream: true` or `n` other than null or 1 | Refused with HTTP 422. |

Independent calls may overlap. Each registers an intent before inference. Failed capture prevents
further admission in that attempt; a pending call alone does not. After the harness returns, the
framework calls the bearer-protected `POST /training-token-capture/control/rollouts/{rollout_id}/close`
then drains already-admitted calls through the manifest endpoint. Closure refuses new generation but
allows admitted calls to finish. Missing acknowledgements remain pending; failure is not permission
to delete data that a worker may still be writing. NeMo RL masks an incomplete attempt.
Closed, failed, or ambiguous admission returns HTTP 409 with a reason instead of a retryable 500.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs suggestion: several framework-mode refusals still return a 500 rather than a 409, namely a context-length overflow, the truncation and prefill checks in prepare_request, unsupported input item kinds, and a ledger row without replay evidence. The OpenAI, Anthropic and Vercel AI SDKs also retry a 409. Could this sentence say which cases return a 409? The section could also state that framework-owned context cannot be combined with the checkpoint gate, and that the measurement route sits outside the rollout prefix without the control bearer token.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the documentation to distinguish HTTP 422 invalid/unsupported input, HTTP 409 closed/ambiguous/unusable history, and acknowledged HTTP 400 context-length refusals, including SDK retries of 409. It also describes the now-supported checkpoint gate and rollout-prefixed measurement route; measurement is a model route, whereas the control endpoints require the control bearer token.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed at 09adc12: the section now separates the 422, 409 and acknowledged 400 cases, warns about SDK retries of 409, and describes the checkpoint gate and the measurement route (model-call-capture.mdx#L148-L180).

Use a fresh attempt ID when restarting a rollout.

The model server also exposes `POST /context/{rollout_id}/measure`. Call it **without** the
`/ng-rollout/.../training-token-capture` prefix, with a non-streaming Responses request containing
explicit history. It returns `{"prompt_token_count": N}` using the same candidate and rendering
path as generation, without registering an intent or writing capture records. This is a count of
that request, not a reservation against concurrent changes. Idempotent worker tokenization retains
ordinary retries; captured generation disables them to avoid replaying an uncertain call.

Framework callers should use `run_examples(retry_requests=False)` to avoid automatically replaying
an entire `/run` after transport failure. This does not prevent a harness from explicitly making a
new call after a definite, successfully captured response.

### Rollout-record attachment

`rollout_collection` attaches available model-call data to each rollout. This representative example
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,26 @@ asyncio.run(main())

For a multi-process deployment, run the check against the real shared backend rather than a local stand-in.

### Provide unique rollout ids
## Let the framework own context reconstruction

For compaction-aware training with a compatible framework worker, enable
`token_id_capture.framework_owned_context: true` together with `external_staging: true`,
`external_staging_backend: vllm_worker`, and `rebuild_response: false`.
See [Framework-owned context and compaction](/main/model-server/model-call-capture#framework-owned-context-and-compaction)
for configuration prerequisites, supported APIs, measurement, and admission closure.

The harness keeps using rollout-correlated Responses or Chat requests with explicit history.
It owns compaction of the next input and needs no special parent header or segment definition.
Return the complete accepted execution history or existing call references, including actions
before compaction. The shortened next input alone is not that history. NeMo RL joins the returned
evidence to captured calls and reconstructs their context chains; Gym does not replace `response.output`.
Concurrent capture is permitted, but selecting subagent calls for training remains the framework's
responsibility. Capture alone cannot establish which calls the harness accepted.

The single-chain rebuilding steps and metrics above describe Gym's local capture path. Framework-owned
reconstruction has its own finalization and validity checks; it does not use that delivery function.

## Provide unique rollout ids

Gym normally derives the capture id from the task and rollout indices. If a custom training loop restarts those indices on each step, explicitly provide a unique `_ng_rollout_id`:

Expand Down
9 changes: 8 additions & 1 deletion nemo_gym/_checkpoint/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -626,7 +626,14 @@ def export_model_records(
records = []
for episode_id in episode_ids:
key = episode_id.capture_key
rows = [row for row in ledger.export_rows(key) if row.get("model_call_id") not in undelivered]
# Admission intents belong to the live attempt, not to its delivered history.
# A restored attempt reissues undelivered calls with new IDs; carrying their
# intents over would leave captures pending forever, including across servers.
rows = [
row
for row in ledger.export_rows(key)
if not row.get("intent") and row.get("model_call_id") not in undelivered
]
cuts = list(cuts_by_key.get(key, []))
# A restored cut that no re-issued call has consumed yet is still this episode's prefix,
# unless a newer call of the episode was cut since.
Expand Down
78 changes: 54 additions & 24 deletions nemo_gym/base_responses_api_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,13 +112,14 @@
resolve_parent,
set_token_sink,
)
from nemo_gym.token_id_capture.completion import completion_metadata

# The store factory needs Gym's server stack.
# The leaf package does not re-export it.
from nemo_gym.token_id_capture.config import NonGeneratingRequest, token_id_capture_config
from nemo_gym.token_id_capture.control_routes import install_rollout_control_routes
from nemo_gym.token_id_capture.lineage import FileLineageStore, InMemoryLineageStore
from nemo_gym.token_id_capture.protocols import CaptureLedger, LineageResolver
from nemo_gym.token_id_capture.protocols import CaptureAdmissionError, CaptureLedger, LineageResolver
from nemo_gym.token_id_capture.records import UNCOMMITTED_CALL_REASON
from nemo_gym.token_id_capture.store import make_token_store

Expand Down Expand Up @@ -316,6 +317,7 @@ class SimpleResponsesAPIModel(BaseResponsesAPIModel, SimpleServer):
# Subclasses can declare successful metadata or health routes here.
# Unknown successful routes fail closed during training-token capture.
non_generating_model_routes: ClassVar[frozenset[tuple[str, str]]] = frozenset()
_capture_lineage: LineageResolver | None = PrivateAttr(default=None)
# Main process of a multi-worker policy server under checkpointing: coordinates the workers.
_policy_coordinator: Optional[PolicyCoordinator] = PrivateAttr(default=None)

Expand Down Expand Up @@ -356,7 +358,7 @@ async def upstream_error(request: Request, error: ClientResponseError) -> Respon

self.setup_session_middleware(app)
capture_config = ModelCallCaptureConfig.model_validate(self.server_client.global_config_dict)
capture_ledger = install_model_call_capture(
self._capture_lineage = capture_ledger = install_model_call_capture(
app,
capture_config,
model_server_name=self.config.name,
Expand Down Expand Up @@ -496,6 +498,17 @@ async def responses_dispatch(self, request: Request, body: dict = Body()):
retain their status before the stream starts. Other ``responses()`` failures become terminal
``response.failed`` events; bad-request validation still fails eagerly.
"""
context = current_capture_context()
if context is not None and context.framework_owned_context:
# ``stream: true`` is allowed: the streaming branch below makes exactly one
# non-streaming ``responses()`` call and re-emits the completed response as
# synthesized SSE, so the engine-bound request stays non-streaming (external
# capture refuses a streamed backend request) and ``_stream_served_response``
# finalizes the capture before the stream is committed. SSE-only harnesses
# such as the Codex CLI thereby run under framework-owned context. Server-held
# history is still refused: it hides part of the context from capture.
if body.get("previous_response_id") or body.get("conversation"):
raise HTTPException(status_code=422, detail="Framework context requires explicit history")
if not body.get("stream"):
params = _validate_responses_params(body)
response = await self._invoke_responses(request, params)
Expand Down Expand Up @@ -557,6 +570,13 @@ async def chat_completions_dispatch(self, request: Request, body: dict = Body())
(e.g. ``"false"`` or ``1``) stays on the strict non-streaming path, which rejects the
malformed ``stream`` with the same 422 as before.
"""
context = current_capture_context()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chat stream: true is refused here, but the Responses gate above allows streaming because the server makes one non-streaming worker call and only synthesizes the server-sent events after capture is finalized. The Chat streaming path below uses the same buffered approach through _stream_served_response, so the same reasoning applies.

Harnesses that only speak streaming Chat, such as OpenClaw or Codex with the Chat wire API, get a 422 on their first call and cannot run in framework mode at all.

Could this gate refuse only n other than 1, with a test that streams a Chat continuation?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the blanket streaming-Chat refusal; only multi-choice requests (n other than 1/default) are rejected. Chat uses one buffered backend generation and commits capture before emitting the answer chunks. Regression tests cover continuation, reasoning, keepalive, and cancellation with uncertain capture retained.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed at 09adc12 that streaming Chat is admitted and commits capture before the first answer chunk.

The new test_cancelled_chat_keepalive_retains_uncertain_capture fails every time when run alone, at both 09adc12 and 5560478, and only sometimes passes inside a larger run. When the client disconnects during the keepalive, the request's cancellation also cancels await ledger.record_failure(...) in _fail_uncommitted_external_call (base_responses_api_model.py#L1358-L1362) before the executor starts it. The call then stays pending with no failure row, and NeMo RL's drain waits its full 300 seconds for that rollout.

Wrapping the write in asyncio.shield, as at base_responses_api_model.py#L1754, makes the test pass on every run. Only lint, DCO, copyright and secrets checks have run on this head so far, so CI has not caught it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shielded the failure write so request cancellation cannot prevent it from reaching the ledger. The cancellation regression now also delays the async write before executor submission: that case failed before the fix and passes after it, retaining the pending call and failure row without exposing an answer. Published in c566cd5.

if (
context is not None
and context.framework_owned_context
and (body.get("stream") or body.get("n") not in (None, 1))
):
raise HTTPException(status_code=422, detail="Framework context requires one non-streaming Chat completion")
if body.get("stream") is not True:
params = _validate_chat_params(body)
response = await self._invoke_chat_completions(request, params)
Expand Down Expand Up @@ -659,8 +679,11 @@ async def _invoke_chat_completions(
# Exact prefix supply and capture share this decision.
if current_capture_context() is not None:
request_messages = _request_messages(params)
await resolve_parent(request_messages)
await register_call_intent()
try:
await resolve_parent(request_messages)
await register_call_intent()
except CaptureAdmissionError as error:
raise HTTPException(status_code=409, detail=str(error)) from error
else:
request_messages = None
if "request" in inspect.signature(self.chat_completions).parameters:
Expand Down Expand Up @@ -703,10 +726,14 @@ async def _invoke_responses(
# all of them.
# Resolve the parent from the received request before dispatch.
# Exact prefix supply and capture share this decision.
if current_capture_context() is not None:
context = current_capture_context()
if context is not None:
request_messages = _request_messages(params)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Framework-owned admission builds its replay evidence from _request_messages(params), and that helper (L219-L226) only reads input when it is a list. A Responses request with a string input, which is the most common form, therefore contributes no user item to the evidence.

This has two effects. If the first call sends input: "old question", its committed replay contains only the assistant answer, so a follow-up that sends the full history as a list no longer matches and silently becomes a new root. Ordinary capture chains the same requests correctly.

The second effect is worse. If the first call sends input: "question A" and a later call sends [<first call's output>, {"role": "user", "content": "unrelated"}], the later request matches the first call's replay and is proposed as a candidate continuing it. NeMo RL's decide_capture_input checks the message roles at the boundary, not the prefix content, so it accepts the candidate and splices the first call's prompt tokens, including "question A", into a context the harness never sent. This is the shape a compaction harness produces when it keeps only the last answer plus a short continuation prompt.

Could _request_messages turn a string input into [{"role": "user", "content": input}]? The measurement route uses the same helper, so it is fixed too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed _request_messages to normalize string input into a user message. Measurement and generation now use the same evidence; the regression checks both a valid continuation and requests that omit or change the original question, which must start new roots. Regression tests.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed at 09adc12: a string input becomes a user message (base_responses_api_model.py#L216-L218). The full-history follow-up continues as a candidate, and the request that omits the original question now starts a new root.

await resolve_parent(request_messages)
await register_call_intent()
try:
await resolve_parent(request_messages)
await register_call_intent()
except CaptureAdmissionError as error:
raise HTTPException(status_code=409, detail=str(error)) from error
else:
request_messages = None
if "request" in inspect.signature(self.responses).parameters:
Expand Down Expand Up @@ -1139,23 +1166,7 @@ def build_model_call_record(exchange: dict[str, Any], *, call_index: int) -> Mod
tool_calls, reasoning_content = _tool_calls_and_reasoning(response)
raw_request = exchange.get("request")
request = raw_request if isinstance(raw_request, dict) else {}
choices = response.get("choices")
first_choice = choices[0] if isinstance(choices, list) and choices and isinstance(choices[0], dict) else {}
incomplete_details = response.get("incomplete_details")
if not isinstance(incomplete_details, dict):
incomplete_details = {}
finish_reason = next(
(
value
for value in (
response.get("stop_reason"),
first_choice.get("finish_reason"),
incomplete_details.get("reason"),
)
if isinstance(value, str)
),
None,
)
_, finish_reason = completion_metadata(response)
model = response.get("model") or request.get("model")
return ModelCallRecord(
model_call_id=exchange.get("model_call_id"),
Expand Down Expand Up @@ -1647,6 +1658,7 @@ async def _fail_uncommitted_external_call(context: CaptureContext | None) -> Non
or not context.external_staging
or context.capture_admission is None
or context.committed
or (context.framework_owned_context and not context.intent_registered)
# A checkpoint gate owns a deferred row: it publishes it when the response starts,
# and a call whose response never starts made no answer to record, so it is not poisoned.
or context.deferred_commit is not None
Expand Down Expand Up @@ -1699,6 +1711,7 @@ def __init__(
lineage_store: LineageResolver | None = None,
delta_records: bool = False,
external_staging: bool = False,
framework_owned_context: bool = False,
token_capture_enabled: bool = False,
assistant_message_header: bytes | None = None,
non_generating_requests: frozenset[tuple[str, str]] = frozenset(),
Expand All @@ -1718,6 +1731,7 @@ def __init__(
# A framework inference worker owns record staging; the lineage store
# doubles as the per-rollout capture ledger.
self._external_staging = external_staging
self._framework_owned_context = framework_owned_context
self._capture_ledger: CaptureLedger | None = (
lineage_store if external_staging and isinstance(lineage_store, CaptureLedger) else None
)
Expand Down Expand Up @@ -1762,6 +1776,18 @@ async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None
# Installed sinks are resolved for each request.
token_sink = self._configured_sink or installed_token_sink() or self._token_store
capture_wanted = token_capture_requested and (token_sink is not None or self._token_capture_enabled)
if (
capture_wanted
and self._framework_owned_context
and dialect is not None
and dialect not in ("responses", "chat")
):
await Response(
content='{"detail":"Framework-owned context currently requires Responses or Chat"}',
status_code=422,
media_type="application/json",
)(scope, receive, send)
return
if token_capture_requested and dialect is None and token_sink is not None:
marked_incomplete = False
response_started = False
Expand Down Expand Up @@ -1844,6 +1870,7 @@ async def _send_unobserved(message: dict[str, Any]) -> None:
lineage_store=self._capture_ledger if self._external_staging else self._lineage_store,
delta_records=self._delta_records,
external_staging=self._external_staging,
framework_owned_context=self._framework_owned_context,
admitted_at=time.time(),
)
sink_token = set_token_sink(capture_context)
Expand Down Expand Up @@ -2149,6 +2176,9 @@ async def _capture_lifespan(application):
lineage_store=lineage_store,
delta_records=(capture_settings.token_id_capture.delta_records if capture_settings is not None else False),
external_staging=external_staging,
framework_owned_context=(
capture_settings.token_id_capture.framework_owned_context if capture_settings is not None else False
),
token_capture_enabled=capture_settings.enabled if capture_settings is not None else False,
non_generating_requests=non_generating_requests,
)
Expand Down
9 changes: 9 additions & 0 deletions nemo_gym/openai_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -1600,6 +1600,11 @@ class NeMoGymAsyncOpenAI(BaseModel): # pragma: no cover

max_http_attempts: int = Field(default=MAX_NUM_TRIES, ge=1)

retry_requests: bool = Field(
default=True,
description="Retry transport and HTTP failures. Disable when replaying a request may duplicate generation.",
)

default_headers: Dict[str, str] = Field(
default_factory=dict,
description="Extra headers to include in every request.",
Expand Down Expand Up @@ -1681,6 +1686,7 @@ async def _request(self, **request_kwargs: Dict) -> ClientResponse:
"_internal": self.internal,
"_max_num_tries": self.max_num_tries,
"_max_connection_retries": self.max_connection_retries,
"_retry": self.retry_requests,
}
if self.request_timeout_seconds is not None:
request_kwargs["timeout"] = ClientTimeout(
Expand All @@ -1692,6 +1698,9 @@ async def _request(self, **request_kwargs: Dict) -> ClientResponse:
async def _request_with_retry(self, **request_kwargs: Dict) -> ClientResponse:
if self._permanent_trip is not None:
self._raise_permanent_error()
if not self.retry_requests:
return await request(**request_kwargs)

max_num_tries = self.max_num_tries or self.max_http_attempts
tries = 0
while tries < max_num_tries:
Expand Down
Loading
Loading