Repository navigation
Conversation
Correct the AGENTS.md lines that disagree with the code on main: - The YAML config wires an Environment Server (single_agent_turn or single_agent_turn_legacy) that runs each episode. - Every server endpoint must be async; an agent's own /run runs only behind the legacy_agent relay. - ServerClient retries wait a fixed 0.5 s; connection errors and calls between Gym servers retry until a caller bound; the 3-try cap covers other exceptions on external calls; HTTP error statuses are never retried. - On agent sessions the Environment Server holds the seed reply's cookies; request.cookies is the fallback outside a session. - Add the status-code rule (422/409/404 helpers; 408/425/429/5xx and connection errors retryable, other 4xx terminal). - ttl_s defaults to None. - Only mask_sample removes a row from the score; inconclusive_verification_retries is a per-server field. - The author sets verified: true after the add-benchmark exit checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 835e1cc |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
AGENTS.md(whichCLAUDE.mdlinks to) is read by every coding agent working in this repository, and several of its serving rules no longer match the code. This PR corrects those lines and touches no other file. Line references below are atb02f8d0909(currentmain).AGENTS.mdsingle_agent_turnwith a typedtasksetdataset, orsingle_agent_turn_legacyfor flat rows.environment_servers/single_agent_turn/configs/single_agent_turn.yaml;nemo_gym/config/config_types.py:563-567(taskset)/runendpoint must be async"/v1/responses; an Environment Server'srun()). An agent's own/runruns only behind thelegacy_agentrelay.nemo_gym/server_runtime/base_environment_server.py:466;nemo_gym/agents/base_responses_api_agent.py:787, 812-817;nemo_gym/config/global_config.py:166, 1243(agents with no Environment Server get a generatedlegacy_agentrelay)ServerClientretries_max_num_tries,_max_connection_retries); with none, calls between Gym servers retry without limit, otherrequest()calls stop after 3 tries except refused or dropped connections. An HTTP error status is never retried.nemo_gym/client/http_client.py:842-844(backoff),:735-748(response returned whatever its status),:749-799(connection errors),:800-834(other exceptions,MAX_NUM_TRIESonly when not_internal);nemo_gym/client/server_client.py:262(_internal=True)cookies=request.cookieson all downstream callsrequest.cookiesis the fallback outside an agent session.environment_servers/single_agent_turn/app.py:436-441, 532, 668;responses_api_agents/simple_agent/app.py:904ResourcesSeedRejectedError(422),ResourcesSessionConflictError(409) andResourcesSessionLostError(404).nemo_gym/server_runtime/base_environment_server.py:185-210;nemo_gym/server_runtime/base_resources_server.py:368-406ttl_sttl_s"None, so it must be set explicitly.nemo_gym/sandbox/config.py:226evaluation_completed=Falseinstead of a silent 0; names a sharedinconclusive_verification_retriesknob.mask_sample: true(optionally withevaluation_completed: false). Onlymask_sampleremoves a row from the score;evaluation_completed: falsealone is scored as 0. The retry count is each server's own config field.nemo_gym/metrics/reward_profile.py:892-902(_partition_on_mask);nemo_gym/server_runtime/base_resources_server.py:210-222; per-server fields with defaults 1 or 2, e.g.resources_servers/swe_rebench/app.py:96,resources_servers/swebench_pro/app.py:236verifiedflagverified: truein their own PR once theadd-benchmarkskill's exit checks pass, and links the run folder,quality_summary.jsonand baselining numbers..agents/skills/add-benchmark/SKILL.md:268. Nothing in code reads the flag.Left unchanged on purpose: the httpx ban and the rule that keeps helpers out of
nemo_gym.sandbox. They are policy, not statements about current code.Issue
None needed: these are documentation corrections to match the code on
main.Overlap with open PRs
Other open PRs edit some of the same
AGENTS.mdlines; whichever merges second needs a small rebase:ServerClientparagraph (and docs(skills): align repository skills with current contracts #3123 the cookie bullet), against the oldernemo_gym.server_utilspath.ServerClientsentence to "retries and connection pooling" and adds Environment Server rows and ownership text elsewhere.Validation
b02f8d0909.pre-commit run --files AGENTS.md: passed (guardwords check ran; the other hooks skip Markdown).AGENTS.mdorCLAUDE.md(grep -rlnovertests/,scripts/,.github/).Rollout evidence
N/A: docs-only change to agent instructions; no environment, agent or verifier behavior changes, so no model compute is needed.
Compatibility
None. No code, config or benchmark result changes.
🤖 Generated with Claude Code