Repository navigation
fix(anyswe_agent): request the training-token-capture prefix and run the inner harness in capture mode - #4016
Open
linj-glitch wants to merge 5 commits into
Open
linj-glitch wants to merge 5 commits into
linj-glitch wants to merge 5 commits into
Conversation
linj-glitch
marked this pull request as draft
October 5, 2026 06:11
linj-glitch
force-pushed
the
linj/anyswe-token-capture
branch
from
October 8, 2026 00:15
a382130 to
e04a68c
Compare
…the inner harness in capture mode `_setup_params` built the in-sandbox model URL with `apply_rollout_prefix(url, rollout_id)` and no `token_capture=` keyword, so the harness only ever got the eval prefix `/ng-rollout/<id>`. With `token_id_capture.enabled: true` the run looks healthy, but every rollout reports `n_calls == 0` and is masked. NVIDIA-NeMo#3397 fixed the same bug in anyterminal_agent. Pass `self._token_id_capture_enabled()`. Two more gaps kept captured rollouts from being attributed: - The in-sandbox runner built the inner agent's global config without a `token_id_capture` block, so the inner agent ran in eval mode. When the model URL carries the training-token-capture segment, the runner now enables capture for the inner agent. - The outer response replaced the inner agent's id with `anyswe-<instance_id>`. Under capture, keep the inner id (the served id the inner agent reports) so terminal attribution can match it. Eval mode keeps the synthetic id. Signed-off-by: Lin Jia <linj@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
linj-glitch
force-pushed
the
linj/anyswe-token-capture
branch
from
October 8, 2026 00:20
e04a68c to
4402c2f
Compare
linj-glitch
marked this pull request as ready for review
October 13, 2026 23:08
pthombre
self-requested a review
October 15, 2026 20:26
pthombre
reviewed
Oct 16, 2026
Contributor
|
/ok to test 4402c2f |
Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
Contributor
|
/ok to test 1d4013d |
pthombre
previously approved these changes
Oct 19, 2026
Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
Collaborator
Author
|
/ok to test 391f8a6 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
anyswe_agentruns an inner harness in a SWE task image. Training-token capture previously broke at three handoffs: the outer agent omitted the capture URL prefix, the inner runner did not enable capture, and the outer response replaced the inner served-response ID. This left rollouts with zero captured calls or no usable attribution.The model URL now carries the training capture prefix when requested, the runner enables capture from that URL, and the outer response retains the inner ID only when the actual sandbox URL contains the capture segment. Direct requests without rollout indices and custom URLs without that segment retain evaluation behavior. The main integration also preserves #4836's trace context on URLs without a rollout ID.
The related tool-call-ID attribution change #4014 merged on October 16 and is included from main; it is no longer an outstanding merge dependency. No separate issue is needed for this focused repair.
Validation
Validated head:
391f8a62a774dad38858956bdc90ce2d18f496f8, including an additive merge of mainf36e126547866601bee9c0671c9b59484ef83723. Python 3.13.14; before running pytest, the runner asserted thatnemo_gym,responses_api_agents.anyswe_agent.app, andagent_runnerwere imported from the reviewed checkout.pytest -q -o addopts= --disable-warnings responses_api_agents/anyswe_agent/tests: 82 passed.pre-commit run --all-files: passed; final PR diff whitespace checks passed and the worktree was clean.The suite exercises the real
run()→ parameter setup → sandbox execution path, checks the URL given to the harness and the returned response ID with capture on/off, covers direct requests and custom URLs, and runs the actual runner construction path to verify the inner agent's capture setting. Existing model-name tests pass without aserver_clienton the test agent. Additional tests preserve trace context with and without a rollout ID.Rollouts and compatibility
Historical author-reported Claude Code GRPO use on R2E-Gym: before these fixes, rollouts were masked because capture or attribution was missing; the combined fixes were subsequently used over several thousand training rollouts. No fresh model or sandbox rollout was run for this integration update, and no new end-to-end training-readiness claim is made.
Capture-off response IDs remain
anyswe-<instance_id>. Capture-enabled responses use the inner served-response ID. Current-head CI and renewed maintainer review remain merge gates.