Skip to content

docs(agents): cover environment servers, ownership, and terminology - #5781

Closed
ananthsub wants to merge 1 commit into
mainfrom
ananthsub/agents-md-update
Closed

ananthsub wants to merge 1 commit into
mainfrom
ananthsub/agents-md-update

Conversation

@ananthsub

@ananthsub ananthsub commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

AGENTS.md (and CLAUDE.md, a symlink to it) is the entry point that Claude Code, Codex, and other coding agents read
first. It described Gym as datasets plus resources, agent, and model servers. It did not mention environment servers,
which own the episode today, or the manifest-backed way to contribute an environment, and it misstated how
ServerClient retries.

This is one complete update to AGENTS.md. It supersedes #5779 and includes that PR's Terminology
section.

Architecture:

  • Adds the environment server to the four-concept table and BaseEnvironmentServer to the base class hierarchy.
  • Adds a short "who owns what" list:
    • the environment server owns the episode, and a new interaction pattern gets its own environment server;
    • the resources, agent, and model servers own tools, state, and verification; the loop; and inference;
    • whoever creates a per-task resource releases it on every path.

Agent run() deprecation:

Conventions and workflow:

  • Every policy model call goes through the model server, because capture and training see only those calls.
  • The contribution path for an environment or benchmark: gym search, gym env init, gym env validate,
    gym env test, with pointers to the contributor docs, the harness guide, and the add-benchmark and add-harness
    skills.
  • gym eval run and gym eval profile in the common commands.

Terminology, from #5779:

  • The glossary is the source of truth, new concepts get a glossary entry, names are not reused, renames keep a
    deprecation path, components are spelled out, and metric names follow the metrics page.
  • docs(agents): make the terminology rules cover the whole glossary #5779's last bullet, about what reviewers flag, is left out. Review checks belong in the pr-review skill.
  • The Code Style rule "Name things by what they are" is deleted. It banned "legacy" with no exception, which
    contradicts the Terminology section's exception for the legacy_agent path and an agent's own /run.

Corrections:

  • The retry description said "3 tries, exponential backoff". nemo_gym/client/http_client.py waits a fixed 0.5 s
    between attempts, and the 3-attempt cap applies only to some errors, so the numbers are removed rather than
    restated.
  • The update-readme-table hook updates the environment list, not only a resources server table.

The new text stays at the level of ownership, conventions, and pointers. Lists of specific implementations, method
names, and flag-level details are left to the docs, so that code changes do not leave this file stale.

Issue

None. This follows an audit of AGENTS.md against the current code and docs, and it replaces #5779.

Validation

  • Checked every command, path, and class named in the new text against upstream/main: the gym command registry in
    nemo_gym/cli/main.py, BaseEnvironmentServer.run() in nemo_gym/server_runtime/base_environment_server.py, the
    retry loop in nemo_gym/client/http_client.py, and the docs pages and skills linked.
  • pre-commit run --files AGENTS.md CLAUDE.md passes.
  • DCO sign-off on the commit.

Rollouts

N/A: documentation for coding agents only. No code, configuration, or environment behavior changes.

Compatibility

None. Docs and glossary follow-ups, such as the same retry claim in about/architecture.mdx and new glossary entries,
come in later PRs.

@copy-pr-bot

copy-pr-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub force-pushed the ananthsub/agents-md-update branch from 77dac5f to 1ccb7c5 Compare November 7, 2026 00:16
@ananthsub
ananthsub marked this pull request as ready for review November 7, 2026 00:25
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 7, 2026 00:26
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 7, 2026 00:26
@svcnvidia-nemo-ci svcnvidia-nemo-ci added docs Documentation-only updates or documentation debt area:docs Generic documentation without another dominant domain needs-review PR is ready for code review and waiting on a reviewer labels Nov 7, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

Two non-blocking NOTEs about contributor guidance. I reviewed both changed files and traced the affected session, collector, CLI, terminology, and compatibility contracts. The Python change only adds a docstring; it preserves the abstract method and runtime behavior.

This was a static review. I did not execute tests.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread AGENTS.md
├── SimpleResourcesServer → implement verify()
├── SimpleResponsesAPIModel → implement chat_completions(), responses()
└── SimpleResponsesAPIAgent → implement responses(), run()
├── SimpleResponsesAPIAgent → implement responses(); run() is deprecated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE — Suggestion: keep the required run() override explicit.

The hierarchy no longer tells subclass authors to implement run(), but SimpleResponsesAPIAgent.run() remains abstract. A new subclass implementing responses() and the session hooks, but omitting run(), still cannot be instantiated. Implementing sessions does not satisfy that separate abstract method. This affects authors following the revised guidance, not existing agents.

Could you distinguish deprecation of whole-episode execution from the remaining subclass requirement? Suggested replacement for this diagram line:

Suggested change
├── SimpleResponsesAPIAgent → implement responses(); run() is deprecated
├── SimpleResponsesAPIAgent → implement responses(); deprecated run() still requires an override

The new docstring could also state: “Session-only subclasses must still override this abstract method and explicitly reject direct /run calls if unsupported.” Keep session logic in the session hooks and episode ordering in the environment server; don't remove the abstract method or existing compatibility implementations to resolve this documentation issue.

Acceptance, static checks: the guidance should distinguish implementing sessions from satisfying the abstract run() requirement. Preserve the ability to instantiate a subclass implementing both methods, as in test_SimpleResponsesAPIAgent, and retain the legacy relay test's assertions that the request reaches ("agent", "/run") with its body and cookies unchanged. These checks were not executed.

Comment thread AGENTS.md
4. **Model** — use any LLM endpoint via the Model Server (supports inference providers like OpenAI, and vLLM for local/open models), or manage inference in your own agent harness
5. **YAML config** — wires the resources server, agent, and model server together

To contribute an environment or benchmark, search for an existing one first (`gym search`), scaffold with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE — Suggestion: distinguish new datasets from custom resources-server contributions.

For a new dataset that needs no custom resources server, this unconditional onboarding recommendation sends contributors to gym env init, but _env_init() explicitly deprecates that command for new datasets. It directs them to gym dataset init <name> instead. That workflow creates task folders with a verifier and reference solution, then uses gym dataset validate, rather than manifest-fixture validation. The consequence is unnecessary work on a deprecated contribution path, not a runtime regression.

Could you split the recommendation? Replacing this line with the following keeps the subsequent custom-workload commands and documentation pointers intact:

Suggested change
To contribute an environment or benchmark, search for an existing one first (`gym search`), scaffold with
To contribute an environment or benchmark, search for an existing one first (`gym search`). For a new dataset
that needs no custom resources server, use `gym dataset init` and `gym dataset validate`. For a workload
that needs a custom resources server, scaffold with

Acceptance, static checks: the dataset branch should agree with the scaffold's printed next steps and the Harbor task guide. Preserve gym env init, gym env validate, and gym env test for custom resources-server workloads, and preserve the harness pointers. The existing CLI assertions should continue to dispatch both --environment and --benchmark to init_environment with their respective scaffold kinds. No runtime change is needed; these checks were not executed.

AGENTS.md described Gym as datasets plus resources, agent, and model
servers. It did not mention environment servers, which own the episode
today, or the manifest-backed contribution path, and it misstated
ServerClient retries.

Add the environment server to the architecture table and the base class
hierarchy, a short "who owns what" list, the rule that every policy
model call goes through the model server, the gym search / env init /
validate / test workflow with its docs and skills, and the gym eval
commands. Remove the incorrect retry details and correct the
update-readme-table hook description.

Say that an agent's own run() is deprecated: new agents implement
agent sessions, and custom episode logic belongs in an environment
server. The deprecation itself lands in the agent server base first.

Fold in the Terminology section from #5779, without its reviewer
bullet, since review checks live in the pr-review skill. Delete the
Code Style naming rule, which banned "legacy" outright and contradicts
the Terminology section's legacy_agent exception.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub

Copy link
Copy Markdown
Contributor Author

Replaced by #5782, which carries these AGENTS.md changes together with the review skill that enforces them, so the conventions and the checks are worded the same way.

@ananthsub ananthsub closed this Nov 16, 2026

This branch was previously deployed

1 inactive deployment
public — 1d8e8b08 Deployed Nov 7, 2026 by copy-pr-bot[bot] via release / finalize / notify #10793
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Generic documentation without another dominant domain docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants