Skip to content

docs: standardize verification task data convention - #3133

Draft
cafzal wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
cafzal:cafzal/docs-verifier-metadata-convention
Draft

cafzal wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
cafzal:cafzal/docs-verifier-metadata-convention

Conversation

@cafzal

@cafzal cafzal commented Sep 5, 2026

Copy link
Copy Markdown

Summary

  • define one verification task-data convention for dataset and resources-server authors
  • document flat TaskData as the authoritative field/type schema
  • limit verifier_metadata to existing wire contracts that read fields exclusively from that legacy container
  • document legacy_location, duplicate/conflict behavior, and gym env schema

Closes #1342.

Why this generalizes

The previous docs presented expected_answer, question, and id as globally optional top-level fields while other guides used verifier_metadata. That ambiguity applies to every verifier and forces users to inspect server implementation details.

Current main already provides the general contract through #2795, #2827, and #2800: every shipped resources server has a dependency-light flat TaskData schema; normalization understands current top-level, legacy-nested, and future task_data forms; legacy_location records the current wire placement; and validation detects misplaced or conflicting fields. This PR documents those existing mechanics rather than adding a competing schema or changing environment behavior.

Independent integration evidence also found that verifier behavior and inputs need explicit identity for reliable offline replay. A single typed task-data definition is a prerequisite for that work, but this PR does not prescribe any benchmark's scoring policy.

Compatibility

Documentation only. Existing datasets and request models are unchanged. The guidance explicitly preserves legacy verifier_metadata placement where a current server requires it.

Validation

  • cd fern && npm run check — 0 errors (one pre-existing/unspecified warning)
  • uv run --extra dev pytest tests/unit_tests/test_task_data.py -q — 305 passed
  • git diff --check — passed
  • uv run pre-commit run --files <three changed MDX files> — passed (hooks skip MDX as configured)

The focused unit suite validates the documented properties, including flat-schema normalization, legacy nested rows, conflicting duplicates, misplaced legacy fields, dependency-light schema imports, all shipped schemas, and all committed rows.

Signed-off-by: Cameron Afzal <cameron.afzal@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 5, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 7, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 8, 2026
@yaoyu-33 yaoyu-33 added area:data Datasets, preparation, task formats, and materialization docs Documentation-only updates or documentation debt and removed community-request Issue reported or requested by someone from the community labels Sep 8, 2026
@ananthsub
ananthsub requested a review from adil-a September 9, 2026 12:16
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 9, 2026
@yaoyu-33 yaoyu-33 added needs-review PR is ready for code review and waiting on a reviewer community-request Issue reported or requested by someone from the community and removed waiting-on-maintainers Waiting on maintainers to respond community-request Issue reported or requested by someone from the community labels Sep 9, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 9, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-maintainers Waiting on maintainers to respond label Sep 9, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 10, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-maintainers Waiting on maintainers to respond label Sep 10, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 10, 2026
@yaoyu-33 yaoyu-33 added community-request Issue reported or requested by someone from the community and removed waiting-on-maintainers Waiting on maintainers to respond labels Sep 10, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 10, 2026
@yaoyu-33 yaoyu-33 removed waiting-on-maintainers Waiting on maintainers to respond community-request Issue reported or requested by someone from the community labels Sep 10, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 10, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 10, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The documented flat TaskData / legacy_location convention matches the implementation. I left three non-blocking inline suggestions to cover self-contained agent-owned schemas, clarify warn-vs-error validation behavior, and align the routing terminology with current collation.

Comment on lines +30 to +34
Task-owned fields such as `expected_answer` vary by resources server—the component that provides tools and reward signals. Inspect that server's typed schema instead of copying another environment's row shape:

```bash
gym env schema --resources-server <name>
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resources-server command does not cover self-contained agents such as Harbor, Tau2, and OSWorld. Those schemas live under responses_api_agents/<name>/task_data.py, and gym env schema --resources-server harbor_agent currently reports that no schema exists. This wording keeps the command useful while making the second ownership path discoverable.

Suggested change
Task-owned fields such as `expected_answer` vary by resources server—the component that provides tools and reward signals. Inspect that server's typed schema instead of copying another environment's row shape:
```bash
gym env schema --resources-server <name>
```
Task-owned fields such as `expected_answer` vary by the component that owns the dataset—the resources server for a resource-backed agent, or the agent itself when it is self-contained. For a resources-server-backed dataset, inspect that server's typed schema instead of copying another environment's row shape:
```bash
gym env schema --resources-server <name>
```
For a self-contained agent, inspect `responses_api_agents/<name>/task_data.py` directly; `gym env schema` currently accepts resources-server names only.

}
```

Do not duplicate a field at the top level and inside `verifier_metadata`. Conflicting duplicates fail validation; equal duplicates add no information. Use the placement required by the selected server until its runtime request contract is migrated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In train_preparation, the default auto mode resolves to warn, so a conflict dirties the report but does not stop collation. This makes the user-visible behavior explicit.

Suggested change
Do not duplicate a field at the top level and inside `verifier_metadata`. Conflicting duplicates fail validation; equal duplicates add no information. Use the placement required by the selected server until its runtime request contract is migrated.
Do not duplicate a field at the top level and inside `verifier_metadata`. Conflicting duplicates make the task-data validation report invalid: `example_validation` fails collation, while the default `train_preparation` mode reports a warning and continues. Set `task_data_validation=error` to make training-data collation fail on these violations. Equal duplicates add no information. Use the placement required by the selected server until its runtime request contract is migrated.

During migration, a field read exclusively from `verifier_metadata` is marked `legacy_location: verifier_metadata` in the flat `TaskData` schema. This marker lets validation detect a top-level field that the current server would not receive. It does not make `verifier_metadata` the default for new fields.

<Note>
`verifier_metadata` describes task-owned inputs to verification, not rollout provenance or framework control data. Keep framework-owned fields such as `responses_create_params` and `agent_ref` outside it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current collation removes legacy agent_ref and stamps task_source, so the new convention should use the current routing field. The adjacent pre-existing Required Fields / agent_ref section should be updated consistently as well.

Suggested change
`verifier_metadata` describes task-owned inputs to verification, not rollout provenance or framework control data. Keep framework-owned fields such as `responses_create_params` and `agent_ref` outside it.
`verifier_metadata` describes task-owned inputs to verification, not rollout provenance or framework control data. Keep framework-owned fields such as `responses_create_params` and `task_source` outside it. `agent_ref` is a deprecated routing input that `gym dataset collate` removes from source rows.

@yaoyu-33 yaoyu-33 added waiting-on-customer Waiting on the original author to respond ready-to-merge PR is approved, current, and waiting for final CI or merge and removed waiting-on-maintainers Waiting on maintainers to respond sla:review-overdue Review response is over the one-business-day SLA needs-review PR is ready for code review and waiting on a reviewer waiting-on-customer Waiting on the original author to respond labels Sep 12, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 12, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@yaoyu-33 yaoyu-33 removed the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 13, 2026
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:data Datasets, preparation, task formats, and materialization community-request Issue reported or requested by someone from the community complexity:low Localized change in one scope with a small, straightforward review surface docs Documentation-only updates or documentation debt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standardize verifier_metadata convention

3 participants