Repository navigation
feat(capture): support framework-owned context reconstruction #4128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
66ce5c0
f456536
92e4300
b53e7b7
dd713d2
0bc517d
5560478
52f4335
81b8129
09adc12
c566cd5
0526500
dd98800
33f9232
86dff9f
a9052b3
a058736
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -64,7 +64,7 @@ original request and response payloads. | |
|
|
||
| ### Buffered SSE with external token staging | ||
|
|
||
| With `token_id_capture.external_staging: true`, captured requests can use `stream: true` | ||
| With `token_id_capture.external_staging: true` and `framework_owned_context: false`, captured requests can use `stream: true` | ||
| on Chat Completions, Responses (including the compaction adapter), and Anthropic Messages. | ||
| Gym requests a complete, non-streaming answer from the inference worker. The worker stages | ||
| training tokens before returning commit coordinates. Gym removes internal token fields, | ||
|
|
@@ -84,6 +84,62 @@ the staged rows must run the same Gym version. A framework whose Gym predates a | |
| server writes, such as `capture_key` or `finish_reason`, rejects every manifest row as | ||
| `invalid_manifest_row` and masks the rollout. | ||
|
|
||
| ### Framework-owned context and compaction | ||
|
|
||
| A compatible training framework can own exact-token prefix reuse and training-row reconstruction. | ||
| Gym matches source history and effective render options, sending a root when they do not match. | ||
| For a candidate parent, the inference worker verifies the converted-message boundary and any | ||
| required null/empty tool-content equivalence before restoring exact tokens. The harness sends its chosen | ||
| message history, including any compaction. It does not send tokens, segment IDs, or a parent header. | ||
|
|
||
| Framework candidates carry an explicit source-match proof. Deploy compatible Gym and RL worker | ||
| revisions together: a missing proof is an error, not permission to assume an exact match. | ||
| Direct roots retain replay evidence and framework safety checks so later calls can continue them. | ||
|
|
||
| The framework must configure the worker, shared capture ledger and control bearer token as well as | ||
| these run-wide settings: | ||
|
|
||
| ```yaml | ||
| token_id_capture: | ||
| enabled: true | ||
| external_staging: true | ||
| external_staging_backend: vllm_worker | ||
| rebuild_response: false | ||
| framework_owned_context: true | ||
| ``` | ||
|
|
||
| This requires a framework worker implementing Gym's capture protocol; stock vLLM alone is insufficient. | ||
| Keep the usual per-agent capture opt-in or `all_agents: true`. `rebuild_response: false` preserves | ||
| ordinary rollout output for framework reconstruction instead of Gym's single-chain rebuilding. | ||
|
|
||
| | Request | Framework-owned behavior | | ||
| |---|---| | ||
| | Responses with explicit history | Supported, including buffered SSE. | | ||
| | Chat Completions with explicit history | Non-streaming, one choice; omitted or null `n` means one. | | ||
| | `/v1/messages` | Refused with HTTP 422. | | ||
| | Responses `previous_response_id` or `conversation` | Refused with HTTP 422; full input must be visible. | | ||
| | Chat `stream: true` or `n` other than null or 1 | Refused with HTTP 422. | | ||
|
|
||
| Independent calls may overlap. Each registers an intent before inference. Failed capture prevents | ||
| further admission in that attempt; a pending call alone does not. After the harness returns, the | ||
| framework calls the bearer-protected `POST /training-token-capture/control/rollouts/{rollout_id}/close` | ||
| then drains already-admitted calls through the manifest endpoint. Closure refuses new generation but | ||
| allows admitted calls to finish. Missing acknowledgements remain pending; failure is not permission | ||
| to delete data that a worker may still be writing. NeMo RL masks an incomplete attempt. | ||
| Closed, failed, or ambiguous admission returns HTTP 409 with a reason instead of a retryable 500. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Docs suggestion: several framework-mode refusals still return a 500 rather than a 409, namely a context-length overflow, the truncation and prefill checks in
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated the documentation to distinguish HTTP 422 invalid/unsupported input, HTTP 409 closed/ambiguous/unusable history, and acknowledged HTTP 400 context-length refusals, including SDK retries of 409. It also describes the now-supported checkpoint gate and rollout-prefixed measurement route; measurement is a model route, whereas the control endpoints require the control bearer token.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed at 09adc12: the section now separates the 422, 409 and acknowledged 400 cases, warns about SDK retries of 409, and describes the checkpoint gate and the measurement route (model-call-capture.mdx#L148-L180). |
||
| Use a fresh attempt ID when restarting a rollout. | ||
|
|
||
| The model server also exposes `POST /context/{rollout_id}/measure`. Call it **without** the | ||
| `/ng-rollout/.../training-token-capture` prefix, with a non-streaming Responses request containing | ||
| explicit history. It returns `{"prompt_token_count": N}` using the same candidate and rendering | ||
| path as generation, without registering an intent or writing capture records. This is a count of | ||
| that request, not a reservation against concurrent changes. Idempotent worker tokenization retains | ||
| ordinary retries; captured generation disables them to avoid replaying an uncertain call. | ||
|
|
||
| Framework callers should use `run_examples(retry_requests=False)` to avoid automatically replaying | ||
| an entire `/run` after transport failure. This does not prevent a harness from explicitly making a | ||
| new call after a definite, successfully captured response. | ||
|
|
||
| ### Rollout-record attachment | ||
|
|
||
| `rollout_collection` attaches available model-call data to each rollout. This representative example | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -112,13 +112,14 @@ | |
| resolve_parent, | ||
| set_token_sink, | ||
| ) | ||
| from nemo_gym.token_id_capture.completion import completion_metadata | ||
|
|
||
| # The store factory needs Gym's server stack. | ||
| # The leaf package does not re-export it. | ||
| from nemo_gym.token_id_capture.config import NonGeneratingRequest, token_id_capture_config | ||
| from nemo_gym.token_id_capture.control_routes import install_rollout_control_routes | ||
| from nemo_gym.token_id_capture.lineage import FileLineageStore, InMemoryLineageStore | ||
| from nemo_gym.token_id_capture.protocols import CaptureLedger, LineageResolver | ||
| from nemo_gym.token_id_capture.protocols import CaptureAdmissionError, CaptureLedger, LineageResolver | ||
| from nemo_gym.token_id_capture.records import UNCOMMITTED_CALL_REASON | ||
| from nemo_gym.token_id_capture.store import make_token_store | ||
|
|
||
|
|
@@ -316,6 +317,7 @@ class SimpleResponsesAPIModel(BaseResponsesAPIModel, SimpleServer): | |
| # Subclasses can declare successful metadata or health routes here. | ||
| # Unknown successful routes fail closed during training-token capture. | ||
| non_generating_model_routes: ClassVar[frozenset[tuple[str, str]]] = frozenset() | ||
| _capture_lineage: LineageResolver | None = PrivateAttr(default=None) | ||
| # Main process of a multi-worker policy server under checkpointing: coordinates the workers. | ||
| _policy_coordinator: Optional[PolicyCoordinator] = PrivateAttr(default=None) | ||
|
|
||
|
|
@@ -356,7 +358,7 @@ async def upstream_error(request: Request, error: ClientResponseError) -> Respon | |
|
|
||
| self.setup_session_middleware(app) | ||
| capture_config = ModelCallCaptureConfig.model_validate(self.server_client.global_config_dict) | ||
| capture_ledger = install_model_call_capture( | ||
| self._capture_lineage = capture_ledger = install_model_call_capture( | ||
| app, | ||
| capture_config, | ||
| model_server_name=self.config.name, | ||
|
|
@@ -496,6 +498,17 @@ async def responses_dispatch(self, request: Request, body: dict = Body()): | |
| retain their status before the stream starts. Other ``responses()`` failures become terminal | ||
| ``response.failed`` events; bad-request validation still fails eagerly. | ||
| """ | ||
| context = current_capture_context() | ||
| if context is not None and context.framework_owned_context: | ||
| # ``stream: true`` is allowed: the streaming branch below makes exactly one | ||
| # non-streaming ``responses()`` call and re-emits the completed response as | ||
| # synthesized SSE, so the engine-bound request stays non-streaming (external | ||
| # capture refuses a streamed backend request) and ``_stream_served_response`` | ||
| # finalizes the capture before the stream is committed. SSE-only harnesses | ||
| # such as the Codex CLI thereby run under framework-owned context. Server-held | ||
| # history is still refused: it hides part of the context from capture. | ||
| if body.get("previous_response_id") or body.get("conversation"): | ||
| raise HTTPException(status_code=422, detail="Framework context requires explicit history") | ||
| if not body.get("stream"): | ||
| params = _validate_responses_params(body) | ||
| response = await self._invoke_responses(request, params) | ||
|
|
@@ -557,6 +570,13 @@ async def chat_completions_dispatch(self, request: Request, body: dict = Body()) | |
| (e.g. ``"false"`` or ``1``) stays on the strict non-streaming path, which rejects the | ||
| malformed ``stream`` with the same 422 as before. | ||
| """ | ||
| context = current_capture_context() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Chat Harnesses that only speak streaming Chat, such as OpenClaw or Codex with the Chat wire API, get a 422 on their first call and cannot run in framework mode at all. Could this gate refuse only
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed the blanket streaming-Chat refusal; only multi-choice requests (
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed at 09adc12 that streaming Chat is admitted and commits capture before the first answer chunk. The new Wrapping the write in
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Shielded the failure write so request cancellation cannot prevent it from reaching the ledger. The cancellation regression now also delays the async write before executor submission: that case failed before the fix and passes after it, retaining the pending call and failure row without exposing an answer. Published in c566cd5. |
||
| if ( | ||
| context is not None | ||
| and context.framework_owned_context | ||
| and (body.get("stream") or body.get("n") not in (None, 1)) | ||
| ): | ||
| raise HTTPException(status_code=422, detail="Framework context requires one non-streaming Chat completion") | ||
| if body.get("stream") is not True: | ||
| params = _validate_chat_params(body) | ||
| response = await self._invoke_chat_completions(request, params) | ||
|
|
@@ -659,8 +679,11 @@ async def _invoke_chat_completions( | |
| # Exact prefix supply and capture share this decision. | ||
| if current_capture_context() is not None: | ||
| request_messages = _request_messages(params) | ||
| await resolve_parent(request_messages) | ||
| await register_call_intent() | ||
| try: | ||
| await resolve_parent(request_messages) | ||
| await register_call_intent() | ||
| except CaptureAdmissionError as error: | ||
| raise HTTPException(status_code=409, detail=str(error)) from error | ||
| else: | ||
| request_messages = None | ||
| if "request" in inspect.signature(self.chat_completions).parameters: | ||
|
|
@@ -703,10 +726,14 @@ async def _invoke_responses( | |
| # all of them. | ||
| # Resolve the parent from the received request before dispatch. | ||
| # Exact prefix supply and capture share this decision. | ||
| if current_capture_context() is not None: | ||
| context = current_capture_context() | ||
| if context is not None: | ||
| request_messages = _request_messages(params) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Framework-owned admission builds its replay evidence from This has two effects. If the first call sends The second effect is worse. If the first call sends Could
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed at 09adc12: a string |
||
| await resolve_parent(request_messages) | ||
| await register_call_intent() | ||
| try: | ||
| await resolve_parent(request_messages) | ||
| await register_call_intent() | ||
| except CaptureAdmissionError as error: | ||
| raise HTTPException(status_code=409, detail=str(error)) from error | ||
| else: | ||
| request_messages = None | ||
| if "request" in inspect.signature(self.responses).parameters: | ||
|
|
@@ -1139,23 +1166,7 @@ def build_model_call_record(exchange: dict[str, Any], *, call_index: int) -> Mod | |
| tool_calls, reasoning_content = _tool_calls_and_reasoning(response) | ||
| raw_request = exchange.get("request") | ||
| request = raw_request if isinstance(raw_request, dict) else {} | ||
| choices = response.get("choices") | ||
| first_choice = choices[0] if isinstance(choices, list) and choices and isinstance(choices[0], dict) else {} | ||
| incomplete_details = response.get("incomplete_details") | ||
| if not isinstance(incomplete_details, dict): | ||
| incomplete_details = {} | ||
| finish_reason = next( | ||
| ( | ||
| value | ||
| for value in ( | ||
| response.get("stop_reason"), | ||
| first_choice.get("finish_reason"), | ||
| incomplete_details.get("reason"), | ||
| ) | ||
| if isinstance(value, str) | ||
| ), | ||
| None, | ||
| ) | ||
| _, finish_reason = completion_metadata(response) | ||
| model = response.get("model") or request.get("model") | ||
| return ModelCallRecord( | ||
| model_call_id=exchange.get("model_call_id"), | ||
|
|
@@ -1647,6 +1658,7 @@ async def _fail_uncommitted_external_call(context: CaptureContext | None) -> Non | |
| or not context.external_staging | ||
| or context.capture_admission is None | ||
| or context.committed | ||
| or (context.framework_owned_context and not context.intent_registered) | ||
| # A checkpoint gate owns a deferred row: it publishes it when the response starts, | ||
| # and a call whose response never starts made no answer to record, so it is not poisoned. | ||
| or context.deferred_commit is not None | ||
|
|
@@ -1699,6 +1711,7 @@ def __init__( | |
| lineage_store: LineageResolver | None = None, | ||
| delta_records: bool = False, | ||
| external_staging: bool = False, | ||
| framework_owned_context: bool = False, | ||
| token_capture_enabled: bool = False, | ||
| assistant_message_header: bytes | None = None, | ||
| non_generating_requests: frozenset[tuple[str, str]] = frozenset(), | ||
|
|
@@ -1718,6 +1731,7 @@ def __init__( | |
| # A framework inference worker owns record staging; the lineage store | ||
| # doubles as the per-rollout capture ledger. | ||
| self._external_staging = external_staging | ||
| self._framework_owned_context = framework_owned_context | ||
| self._capture_ledger: CaptureLedger | None = ( | ||
| lineage_store if external_staging and isinstance(lineage_store, CaptureLedger) else None | ||
| ) | ||
|
|
@@ -1762,6 +1776,18 @@ async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None | |
| # Installed sinks are resolved for each request. | ||
| token_sink = self._configured_sink or installed_token_sink() or self._token_store | ||
| capture_wanted = token_capture_requested and (token_sink is not None or self._token_capture_enabled) | ||
| if ( | ||
| capture_wanted | ||
| and self._framework_owned_context | ||
| and dialect is not None | ||
| and dialect not in ("responses", "chat") | ||
| ): | ||
| await Response( | ||
| content='{"detail":"Framework-owned context currently requires Responses or Chat"}', | ||
| status_code=422, | ||
| media_type="application/json", | ||
| )(scope, receive, send) | ||
| return | ||
| if token_capture_requested and dialect is None and token_sink is not None: | ||
| marked_incomplete = False | ||
| response_started = False | ||
|
|
@@ -1844,6 +1870,7 @@ async def _send_unobserved(message: dict[str, Any]) -> None: | |
| lineage_store=self._capture_ledger if self._external_staging else self._lineage_store, | ||
| delta_records=self._delta_records, | ||
| external_staging=self._external_staging, | ||
| framework_owned_context=self._framework_owned_context, | ||
| admitted_at=time.time(), | ||
| ) | ||
| sink_token = set_token_sink(capture_context) | ||
|
|
@@ -2149,6 +2176,9 @@ async def _capture_lifespan(application): | |
| lineage_store=lineage_store, | ||
| delta_records=(capture_settings.token_id_capture.delta_records if capture_settings is not None else False), | ||
| external_staging=external_staging, | ||
| framework_owned_context=( | ||
| capture_settings.token_id_capture.framework_owned_context if capture_settings is not None else False | ||
| ), | ||
| token_capture_enabled=capture_settings.enabled if capture_settings is not None else False, | ||
| non_generating_requests=non_generating_requests, | ||
| ) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Naming suggestion, not a blocker: elsewhere on this page "framework-owned" means the training framework owns storage ("Framework-owned media staging"), which matches
external_staging. In this section Gym makes the continuity decision itself, so "Framework-owned context" suggests the opposite ownership. If the setting is renamed (see the comment onconfig.py), a heading such as "Resolve parents from the source history" or "Source-history continuity and context compaction" would describe the behavior directly.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated the heading to “Source-history continuity and context compaction,” updated the configuration examples and incoming anchor link, and clarified Gym’s source-history matching versus the worker’s exact-token verification. Published in 86dff9f. Fern check passed; the authenticated redirect audit was skipped because the local Fern CLI was not authenticated.