Skip to content

docs(agents): align serving rules with the session code - #6028

Draft
yaoyu-33 wants to merge 1 commit into
mainfrom
yuya/docs-agents-md-serving-rules
Draft

yaoyu-33 wants to merge 1 commit into
mainfrom
yuya/docs-agents-md-serving-rules

Conversation

@yaoyu-33

Copy link
Copy Markdown
Contributor

What changed and why

AGENTS.md (which CLAUDE.md links to) is read by every coding agent working in this repository, and several of its serving rules no longer match the code. This PR corrects those lines and touches no other file. Line references below are at b02f8d0909 (current main).

AGENTS.md Before After Code
Creating Environments, item 5 The YAML config wires the resources server, agent and model server. It wires an Environment Server, which runs each episode, to the resources server and agent: single_agent_turn with a typed taskset dataset, or single_agent_turn_legacy for flat rows. environment_servers/single_agent_turn/configs/single_agent_turn.yaml; nemo_gym/config/config_types.py:563-567 (taskset)
Async endpoints "the /run endpoint must be async" Every server endpoint must be async (seed, verify, close, tools; agent session endpoints and /v1/responses; an Environment Server's run()). An agent's own /run runs only behind the legacy_agent relay. nemo_gym/server_runtime/base_environment_server.py:466; nemo_gym/agents/base_responses_api_agent.py:787, 812-817; nemo_gym/config/global_config.py:166, 1243 (agents with no Environment Server get a generated legacy_agent relay)
ServerClient retries "3 tries, exponential backoff" Fixed 0.5 s wait. A call that raises retries until the caller's bound (_max_num_tries, _max_connection_retries); with none, calls between Gym servers retry without limit, other request() calls stop after 3 tries except refused or dropped connections. An HTTP error status is never retried. nemo_gym/client/http_client.py:842-844 (backoff), :735-748 (response returned whatever its status), :749-799 (connection errors), :800-834 (other exceptions, MAX_NUM_TRIES only when not _internal); nemo_gym/client/server_client.py:262 (_internal=True)
Session cookies cookies=request.cookies on all downstream calls On agent sessions the Environment Server keeps the seed reply's cookies, sends them with verify and close, and grants them to the agent in its tool access. request.cookies is the fallback outside an agent session. environment_servers/single_agent_turn/app.py:436-441, 532, 668; responses_api_agents/simple_agent/app.py:904
Status codes (new bullet) none 408, 425, 429, 5xx, refused or dropped connections and timeouts are retryable; other 4xx are terminal. A resources server answers with ResourcesSeedRejectedError (422), ResourcesSessionConflictError (409) and ResourcesSessionLostError (404). nemo_gym/server_runtime/base_environment_server.py:185-210; nemo_gym/server_runtime/base_resources_server.py:368-406
ttl_s "Set ttl_s" Adds that it defaults to None, so it must be set explicitly. nemo_gym/sandbox/config.py:226
Inconclusive verification Report evaluation_completed=False instead of a silent 0; names a shared inconclusive_verification_retries knob. Return mask_sample: true (optionally with evaluation_completed: false). Only mask_sample removes a row from the score; evaluation_completed: false alone is scored as 0. The retry count is each server's own config field. nemo_gym/metrics/reward_profile.py:892-902 (_partition_on_mask); nemo_gym/server_runtime/base_resources_server.py:210-222; per-server fields with defaults 1 or 2, e.g. resources_servers/swe_rebench/app.py:96, resources_servers/swebench_pro/app.py:236
verified flag "means the benchmark has been baselined and reviewed" The author sets verified: true in their own PR once the add-benchmark skill's exit checks pass, and links the run folder, quality_summary.json and baselining numbers. .agents/skills/add-benchmark/SKILL.md:268. Nothing in code reads the flag.

Left unchanged on purpose: the httpx ban and the rule that keeps helpers out of nemo_gym.sandbox. They are policy, not statements about current code.

Issue

None needed: these are documentation corrections to match the code on main.

Overlap with open PRs

Other open PRs edit some of the same AGENTS.md lines; whichever merges second needs a small rebase:

Validation

  • Re-read every cited line on b02f8d0909.
  • pre-commit run --files AGENTS.md: passed (guardwords check ran; the other hooks skip Markdown).
  • Confirmed no test or script reads AGENTS.md or CLAUDE.md (grep -rln over tests/, scripts/, .github/).
  • Unit tests: N/A (docs-only).

Rollout evidence

N/A: docs-only change to agent instructions; no environment, agent or verifier behavior changes, so no model compute is needed.

Compatibility

None. No code, config or benchmark result changes.

🤖 Generated with Claude Code

Correct the AGENTS.md lines that disagree with the code on main:

- The YAML config wires an Environment Server (single_agent_turn or
  single_agent_turn_legacy) that runs each episode.
- Every server endpoint must be async; an agent's own /run runs only
  behind the legacy_agent relay.
- ServerClient retries wait a fixed 0.5 s; connection errors and
  calls between Gym servers retry until a caller bound; the 3-try cap
  covers other exceptions on external calls; HTTP error statuses are
  never retried.
- On agent sessions the Environment Server holds the seed reply's
  cookies; request.cookies is the fallback outside a session.
- Add the status-code rule (422/409/404 helpers; 408/425/429/5xx and
  connection errors retryable, other 4xx terminal).
- ttl_s defaults to None.
- Only mask_sample removes a row from the score;
  inconclusive_verification_retries is a per-server field.
- The author sets verified: true after the add-benchmark exit checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 835e1cc

@yaoyu-33 yaoyu-33 added docs Documentation-only updates or documentation debt area:docs Generic documentation without another dominant domain labels Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Generic documentation without another dominant domain docs Documentation-only updates or documentation debt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant