Repository navigation
Description
NeMo Gym's current CLI is well-suited for RL training workflows. As we expand to serve agent developers focused on evaluation, there are several UX improvements we should make.
Benchmark-centric, need to improve UX for agent-centric
The current evaluation flow starts from "pick a benchmark" which bundles the agent, environment, and data together in a single config. Agent developers think the other way: "I have an agent, I want to evaluate it against environments." We need to improve the UX of this path.
Agent swapping requires config duplication
Running SWE-bench with OpenHands vs SWE Agent vs Mini SWE Agent requires three nearly identical YAML configs that differ by one line (the agent). Every agent-environment combination needs its own config file rather than being composable at the CLI level.
Config paths require familiarity with project structure
Users need to specify exact paths like resources_servers/gpqa/configs/gpqa.yaml. This requires familiarity with the internal project structure — you need to know that model configs live in responses_api_models/ and benchmark configs in resources_servers/ or benchmarks/.
3 commands to compose agentic eval
ng_run → ng_collect_rollouts → ng_reward_profile exposes the microservice infrastructure (start servers, collect, score) rather than abstracting it into the user's goal: "evaluate my agent on this task set." Ideally the user could specify tasks, model, agent, and environment in a single command. Ref: #1188
SWE-bench not in ng_list_benchmarks
SWE-bench is one of the most popular agentic benchmarks, yet it doesn't appear in ng_list_benchmarks because it's implemented at the agent level (in responses_api_agents/swe_agents/) rather than as a benchmark config. A user looking for "how do I run SWE-bench" won't find it through the primary discovery command. The benchmark registry only covers environments that follow the resources-server-with-simple-agent pattern, missing agent-centric benchmarks. #1249
CLI terminology is training-centric, not eval-centric
"Collect rollouts" and "reward profiling" are RL training terms — an agent developer who wants to evaluate their agent thinks in terms of "run evaluation" and "get metrics/scores." The current naming assumes the user's goal is to generate training data, not to assess agent performance. Similarly, responses_create_params, verifier_metadata, and resources_server are abstractions that don't map to how an agent developer thinks about their workflow ("tasks," "scores," "environments"). The help text explains the current concepts well, but users searching for evaluation workflows may not discover the right commands.
Acceptance Criteria
Agent-centric workflow
- A user with an existing agent can evaluate it against any supported environment without authoring a YAML config file
- The CLI supports an agent-first entry point (e.g.,
gym eval --agent <path-or-name> --env <environment>) in addition to the existing benchmark-first flow - Agent developers can point to their own agent implementation (local path or remote repo) without forking or modifying the Gym repo
Composability without config duplication
- Agent, environment, and model can be specified independently at the CLI level — no pre-authored YAML required for each combination
- Swapping the agent for a given environment (e.g., OpenHands → SWE Agent on SWE-bench) requires changing only the agent argument, not duplicating a config file
- Model provider/endpoint can be overridden at invocation time without editing YAML (e.g.,
--model gpt-4oor--model-url http://localhost:8000)
Single-command evaluation
- A user can go from "I want to evaluate my agent" to "I have scores" in a single CLI command
- The single command handles server lifecycle, rollout collection, and scoring internally — users don't need to know about microservice orchestration
- Progress is reported during execution (tasks completed, running score, ETA)
- Results are output in a human-readable summary (pass@1, pass@k, per-category breakdown) without requiring a separate profiling step
Discoverability
- All supported evaluations (including agent-level benchmarks like SWE-bench) are discoverable through a single listing command
- The listing command shows available environments with metadata: domain, difficulty signal, compatible agents, data availability
-
--helptext uses evaluation-centric terminology (e.g., "evaluate", "score", "tasks") alongside or instead of RL-specific terms ("rollouts", "reward", "verifier")
Terminology and naming
- User-facing CLI commands and
--helpoutput use evaluation-centric language that is understandable without RL background - Internal abstractions (
resources_server,verifier_metadata,responses_create_params) are not exposed in the user-facing eval command interface - Documentation maps between RL terminology and eval terminology for users who need to understand both layers
Path resolution
- Users can reference environments and agents by short name (e.g.,
workplace_assistant,gpqa) rather than internal file paths - Short names are auto-resolved from a registry or directory convention — no
resources_servers/gpqa/configs/gpqa.yamlin the CLI invocation
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
[-]feat: improve evaluation UX for agent developers[/-][+]epic: improve evaluation UX for agent developers[/+]sephmard commentedon Jul 29, 2026
Duplicates #1736. Closing #1736.