Repository navigation
Conversation
Signed-off-by: Cameron Afzal <cameron.afzal@gmail.com>
yaoyu-33
left a comment
There was a problem hiding this comment.
LGTM. The documented flat TaskData / legacy_location convention matches the implementation. I left three non-blocking inline suggestions to cover self-contained agent-owned schemas, clarify warn-vs-error validation behavior, and align the routing terminology with current collation.
| Task-owned fields such as `expected_answer` vary by resources server—the component that provides tools and reward signals. Inspect that server's typed schema instead of copying another environment's row shape: | ||
|
|
||
| ```bash | ||
| gym env schema --resources-server <name> | ||
| ``` |
There was a problem hiding this comment.
The resources-server command does not cover self-contained agents such as Harbor, Tau2, and OSWorld. Those schemas live under responses_api_agents/<name>/task_data.py, and gym env schema --resources-server harbor_agent currently reports that no schema exists. This wording keeps the command useful while making the second ownership path discoverable.
| Task-owned fields such as `expected_answer` vary by resources server—the component that provides tools and reward signals. Inspect that server's typed schema instead of copying another environment's row shape: | |
| ```bash | |
| gym env schema --resources-server <name> | |
| ``` | |
| Task-owned fields such as `expected_answer` vary by the component that owns the dataset—the resources server for a resource-backed agent, or the agent itself when it is self-contained. For a resources-server-backed dataset, inspect that server's typed schema instead of copying another environment's row shape: | |
| ```bash | |
| gym env schema --resources-server <name> | |
| ``` | |
| For a self-contained agent, inspect `responses_api_agents/<name>/task_data.py` directly; `gym env schema` currently accepts resources-server names only. |
| } | ||
| ``` | ||
|
|
||
| Do not duplicate a field at the top level and inside `verifier_metadata`. Conflicting duplicates fail validation; equal duplicates add no information. Use the placement required by the selected server until its runtime request contract is migrated. |
There was a problem hiding this comment.
In train_preparation, the default auto mode resolves to warn, so a conflict dirties the report but does not stop collation. This makes the user-visible behavior explicit.
| Do not duplicate a field at the top level and inside `verifier_metadata`. Conflicting duplicates fail validation; equal duplicates add no information. Use the placement required by the selected server until its runtime request contract is migrated. | |
| Do not duplicate a field at the top level and inside `verifier_metadata`. Conflicting duplicates make the task-data validation report invalid: `example_validation` fails collation, while the default `train_preparation` mode reports a warning and continues. Set `task_data_validation=error` to make training-data collation fail on these violations. Equal duplicates add no information. Use the placement required by the selected server until its runtime request contract is migrated. |
| During migration, a field read exclusively from `verifier_metadata` is marked `legacy_location: verifier_metadata` in the flat `TaskData` schema. This marker lets validation detect a top-level field that the current server would not receive. It does not make `verifier_metadata` the default for new fields. | ||
|
|
||
| <Note> | ||
| `verifier_metadata` describes task-owned inputs to verification, not rollout provenance or framework control data. Keep framework-owned fields such as `responses_create_params` and `agent_ref` outside it. |
There was a problem hiding this comment.
Current collation removes legacy agent_ref and stamps task_source, so the new convention should use the current routing field. The adjacent pre-existing Required Fields / agent_ref section should be updated consistently as well.
| `verifier_metadata` describes task-owned inputs to verification, not rollout provenance or framework control data. Keep framework-owned fields such as `responses_create_params` and `agent_ref` outside it. | |
| `verifier_metadata` describes task-owned inputs to verification, not rollout provenance or framework control data. Keep framework-owned fields such as `responses_create_params` and `task_source` outside it. `agent_ref` is a deprecated routing input that `gym dataset collate` removes from source rows. |
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
Summary
TaskDataas the authoritative field/type schemaverifier_metadatato existing wire contracts that read fields exclusively from that legacy containerlegacy_location, duplicate/conflict behavior, andgym env schemaCloses #1342.
Why this generalizes
The previous docs presented
expected_answer,question, andidas globally optional top-level fields while other guides usedverifier_metadata. That ambiguity applies to every verifier and forces users to inspect server implementation details.Current
mainalready provides the general contract through #2795, #2827, and #2800: every shipped resources server has a dependency-light flatTaskDataschema; normalization understands current top-level, legacy-nested, and futuretask_dataforms;legacy_locationrecords the current wire placement; and validation detects misplaced or conflicting fields. This PR documents those existing mechanics rather than adding a competing schema or changing environment behavior.Independent integration evidence also found that verifier behavior and inputs need explicit identity for reliable offline replay. A single typed task-data definition is a prerequisite for that work, but this PR does not prescribe any benchmark's scoring policy.
Compatibility
Documentation only. Existing datasets and request models are unchanged. The guidance explicitly preserves legacy
verifier_metadataplacement where a current server requires it.Validation
cd fern && npm run check— 0 errors (one pre-existing/unspecified warning)uv run --extra dev pytest tests/unit_tests/test_task_data.py -q— 305 passedgit diff --check— passeduv run pre-commit run --files <three changed MDX files>— passed (hooks skip MDX as configured)The focused unit suite validates the documented properties, including flat-schema normalization, legacy nested rows, conflicting duplicates, misplaced legacy fields, dependency-light schema imports, all shipped schemas, and all committed rows.