Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
77dac5f to
1ccb7c5
Compare
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
Two non-blocking NOTEs about contributor guidance. I reviewed both changed files and traced the affected session, collector, CLI, terminology, and compatibility contracts. The Python change only adds a docstring; it preserves the abstract method and runtime behavior.
This was a static review. I did not execute tests.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| ├── SimpleResourcesServer → implement verify() | ||
| ├── SimpleResponsesAPIModel → implement chat_completions(), responses() | ||
| └── SimpleResponsesAPIAgent → implement responses(), run() | ||
| ├── SimpleResponsesAPIAgent → implement responses(); run() is deprecated |
There was a problem hiding this comment.
NOTE — Suggestion: keep the required run() override explicit.
The hierarchy no longer tells subclass authors to implement run(), but SimpleResponsesAPIAgent.run() remains abstract. A new subclass implementing responses() and the session hooks, but omitting run(), still cannot be instantiated. Implementing sessions does not satisfy that separate abstract method. This affects authors following the revised guidance, not existing agents.
Could you distinguish deprecation of whole-episode execution from the remaining subclass requirement? Suggested replacement for this diagram line:
| ├── SimpleResponsesAPIAgent → implement responses(); run() is deprecated | |
| ├── SimpleResponsesAPIAgent → implement responses(); deprecated run() still requires an override |
The new docstring could also state: “Session-only subclasses must still override this abstract method and explicitly reject direct /run calls if unsupported.” Keep session logic in the session hooks and episode ordering in the environment server; don't remove the abstract method or existing compatibility implementations to resolve this documentation issue.
Acceptance, static checks: the guidance should distinguish implementing sessions from satisfying the abstract run() requirement. Preserve the ability to instantiate a subclass implementing both methods, as in test_SimpleResponsesAPIAgent, and retain the legacy relay test's assertions that the request reaches ("agent", "/run") with its body and cookies unchanged. These checks were not executed.
| 4. **Model** — use any LLM endpoint via the Model Server (supports inference providers like OpenAI, and vLLM for local/open models), or manage inference in your own agent harness | ||
| 5. **YAML config** — wires the resources server, agent, and model server together | ||
|
|
||
| To contribute an environment or benchmark, search for an existing one first (`gym search`), scaffold with |
There was a problem hiding this comment.
NOTE — Suggestion: distinguish new datasets from custom resources-server contributions.
For a new dataset that needs no custom resources server, this unconditional onboarding recommendation sends contributors to gym env init, but _env_init() explicitly deprecates that command for new datasets. It directs them to gym dataset init <name> instead. That workflow creates task folders with a verifier and reference solution, then uses gym dataset validate, rather than manifest-fixture validation. The consequence is unnecessary work on a deprecated contribution path, not a runtime regression.
Could you split the recommendation? Replacing this line with the following keeps the subsequent custom-workload commands and documentation pointers intact:
| To contribute an environment or benchmark, search for an existing one first (`gym search`), scaffold with | |
| To contribute an environment or benchmark, search for an existing one first (`gym search`). For a new dataset | |
| that needs no custom resources server, use `gym dataset init` and `gym dataset validate`. For a workload | |
| that needs a custom resources server, scaffold with |
Acceptance, static checks: the dataset branch should agree with the scaffold's printed next steps and the Harbor task guide. Preserve gym env init, gym env validate, and gym env test for custom resources-server workloads, and preserve the harness pointers. The existing CLI assertions should continue to dispatch both --environment and --benchmark to init_environment with their respective scaffold kinds. No runtime change is needed; these checks were not executed.
AGENTS.md described Gym as datasets plus resources, agent, and model servers. It did not mention environment servers, which own the episode today, or the manifest-backed contribution path, and it misstated ServerClient retries. Add the environment server to the architecture table and the base class hierarchy, a short "who owns what" list, the rule that every policy model call goes through the model server, the gym search / env init / validate / test workflow with its docs and skills, and the gym eval commands. Remove the incorrect retry details and correct the update-readme-table hook description. Say that an agent's own run() is deprecated: new agents implement agent sessions, and custom episode logic belongs in an environment server. The deprecation itself lands in the agent server base first. Fold in the Terminology section from #5779, without its reviewer bullet, since review checks live in the pr-review skill. Delete the Code Style naming rule, which banned "legacy" outright and contradicts the Terminology section's legacy_agent exception. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
1ccb7c5 to
1d8e8b0
Compare
|
Replaced by #5782, which carries these |
What changed and why
AGENTS.md(andCLAUDE.md, a symlink to it) is the entry point that Claude Code, Codex, and other coding agents readfirst. It described Gym as datasets plus resources, agent, and model servers. It did not mention environment servers,
which own the episode today, or the manifest-backed way to contribute an environment, and it misstated how
ServerClientretries.This is one complete update to
AGENTS.md. It supersedes #5779 and includes that PR's Terminologysection.
Architecture:
BaseEnvironmentServerto the base class hierarchy.Agent
run()deprecation:AGENTS.mdclass hierarchy and ownership list say that an agent's ownrun()is deprecated: new agentsimplement agent sessions, and custom episode logic belongs in an environment server. The deprecation itself is
feat(agent): deprecate an agent's own run() and make it optional #5785, which should merge first.
Conventions and workflow:
gym search,gym env init,gym env validate,gym env test, with pointers to the contributor docs, the harness guide, and theadd-benchmarkandadd-harnessskills.
gym eval runandgym eval profilein the common commands.Terminology, from #5779:
deprecation path, components are spelled out, and metric names follow the metrics page.
pr-reviewskill.contradicts the Terminology section's exception for the
legacy_agentpath and an agent's own/run.Corrections:
nemo_gym/client/http_client.pywaits a fixed 0.5 sbetween attempts, and the 3-attempt cap applies only to some errors, so the numbers are removed rather than
restated.
update-readme-tablehook updates the environment list, not only a resources server table.The new text stays at the level of ownership, conventions, and pointers. Lists of specific implementations, method
names, and flag-level details are left to the docs, so that code changes do not leave this file stale.
Issue
None. This follows an audit of
AGENTS.mdagainst the current code and docs, and it replaces #5779.Validation
upstream/main: thegymcommand registry innemo_gym/cli/main.py,BaseEnvironmentServer.run()innemo_gym/server_runtime/base_environment_server.py, theretry loop in
nemo_gym/client/http_client.py, and the docs pages and skills linked.pre-commit run --files AGENTS.md CLAUDE.mdpasses.Rollouts
N/A: documentation for coding agents only. No code, configuration, or environment behavior changes.
Compatibility
None. Docs and glossary follow-ups, such as the same retry claim in
about/architecture.mdxand new glossary entries,come in later PRs.