Repository navigation
feat(harbor): let a dataset name the reward.json key that is the score - #5733
zuraiz-anjum wants to merge 10 commits into
Conversation
A Harbor verifier that writes several metrics and no `reward` key is scored as the sum of every field, so a verifier that writes six 0 to 1 metrics plus `overall` scores up to about 7. `reward_key` in dataset.toml (`[gym.defaults]`, or `[gym.tasks."<id>".verifier]` for one task) names the field that is the score. The other keys stay in `verifier_rewards` as metrics, a file without the key is measured 0 with `harbor:invalid_reward`, and a step's scalar `min_reward` gates that key. A task.toml cannot set it (the loader ignores it there with the unknown-key warning, as Harbor does), and a task graded by a Python verifier refuses it, since the runner only writes a `reward` key or a single metric. Unset keeps the current rule (a `reward` key, else the sum) and existing task digests. Closes NVIDIA-NeMo#5641 Signed-off-by: Zuraiz Anjum <zuraizwork@gmail.com>
c8706e8 to
000034d
Compare
|
/ok to test 000034d |
|
/review |
|
🌿 Preview your docs: https://nvidia-preview-pr-5733-89e5b3af4224.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
I found two RISK findings and one documentation NOTE. I inspected all ten changed files and their directly affected configuration, preparation, scoring, and multi-turn paths. This was a static review; I didn't execute tests.
RISK — Ignore task-local reward_key consistently during preparation
Failure: An otherwise valid task with [verifier] reward_key = "" and no dataset-level setting is no longer prepared consistently. load_task deliberately removes that unknown task-local field, but the earlier dockerfile_base_images and plan_builds readers validate raw TOML against the newly constrained field. They catch the validation error and omit the task. With no existing records, a Dockerfile containing FROM ubuntu:24.04\nWORKDIR src\n gets no base-image record; one containing FROM ubuntu:24.04\nCOPY . /app\n gets no build entry. Discovery then rejects the task for missing preparation outputs. Before this PR, these readers ignored the unknown key. This affects task folders without opting into the feature.
Correction: In both preparation readers linked above, replace the raw-TOML validation statement with the following, keeping apply_dataset_config afterward:
data = tomllib.loads((child / TASK_FILE).read_text())
if isinstance(verifier_table := data.get("verifier"), dict):
data["verifier"] = {key: value for key, value in verifier_table.items() if key != "reward_key"}
config = HarborTaskConfig.model_validate(data)This restores the invariant that raw task-local reward_key is ignored before validation, while effective dataset settings remain validated. I traced this suggested replacement through both callers, but it wasn't executed; please verify it before adopting it.
Preservation and acceptance — proposed tests: With no image/build records, parameterize task-local values as "" and 1. The relative-WORKDIR fixture should yield {"ubuntu:24.04": ["a"]} from dockerfile_base_images; the COPY fixture should produce a build entry containing task a. Retain the existing configuration assertions: dataset defaults select overall, a task override selects score, an empty dataset key is rejected, a task-local key is ignored with a warning, opting in changes the digest, and leaving the key unset preserves the previous digest. Other verifier fields and environment overrides must retain their validation and application.
RISK — Provide the required real smoke-rollout evidence
The PR changes environment behavior when enabled, including the score-selection branch and the gate controlling whether another agent turn runs. The description reports FakeSandbox validation and explicitly no real model rollouts. The trusted AGENTS.md and CONTRIBUTING.md require a representative model rollout for behavior-changing environment work; opt-in behavior has no stated exemption. This is a validation gap, not a demonstrated runtime defect.
Could you run a small real two-step Harbor task with a model, reward_key = "overall", and a scalar first-step min_reward, then record the command, model/harness/provider, inspected agent work, verifier files, and resulting scores? For files containing {"accuracy": 1.0, "overall": 0.5} on both turns with min_reward = 0.5, acceptance is two turns scoring 0.5, episode mean 0.5, both metrics retained, and no components. An unset-key control without the scalar gate should still score 1.5 with both fields as components. Retain the added below-threshold case, where overall = 0.1 stops the episode, and keep mapping thresholds tied to their explicitly named keys. No scoring rewrite is requested; the correction is to supply and inspect the required integration evidence.
Inline comments (escaped v1):
-
fern/versions/latest/pages/data/harbor-tasks.mdxline 107: NOTE — Suggestion: qualify the existing reward-contract descriptions forreward_key.With
reward_key = "overall"andreward.json = {"accuracy": 1.0, "overall": 0.5}, this paragraph still promises components because there is norewardkey. The implementation instead scores0.5and names no components. The adjacent digest and fixed-objective guidance, result docstring, and README gate description also describe only the previous contract.Could you replace the components condition here with “when
reward_keyis unset and the verifier'sreward.jsonnamed norewardkey”? Apply that qualifier to the result docstring and fixed-objective/GDPO guidance, include defaultreward_keyalongside scaling in the digest explanation, and describe the scalar gate as “a number gatesreward_keywhen set, elsereward; a table gates each named key.” These are documentation-only corrections; retain the valid unset-key sum/components rule.Static acceptance: selected
overallmeans score0.5, no components, and a scalar gate readingoverall; unset means score1.5, both components, and a scalar gate readingreward. In both cases{accuracy = 0.5}still gatesaccuracy. Check the revised wording against these cases; these checks weren't executed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
The base-image scan and the build plan validated task.toml with a task-local `[verifier] reward_key` still in place, while the loader drops it with a warning. A value the field refuses (an empty string, a number) left the task out of both scans, and discovery then refused it for the missing records. The loader and both scans now read task.toml through `read_task_toml`, which drops the key before validation. The docs and docstrings now qualify the reward contract for `reward_key`: components are named only when it is unset, a scalar `min_reward` gates it when set, and it enters the task digest. Signed-off-by: Zuraiz Anjum <zuraizwork@gmail.com>
|
Pushed 35e5def for the review. Task-local Instead of copying the strip into each reader, I added one helper, The new test is parameterized over
Docs. Applied your wording in
The same qualifier also went into the Smoke rollout. I can't run a real model rollout myself because I don't have model access. The scoring and gating branches are covered with FakeSandbox through the server: the per-turn score under Smoke dataset, command and expected resultsRun in an empty folder; it writes #!/bin/bash
set -eu
root=reward-key-smoke
grader() { # grader <test.sh path> <file under /app> <word> <reward.json when the agent did the step>
cat > "$1" <<EOF
#!/bin/bash
if grep -qx $3 /app/$2 2>/dev/null; then
echo '$4' > /logs/verifier/reward.json
else
echo '{"accuracy": 0.0, "overall": 0.0}' > /logs/verifier/reward.json
fi
EOF
chmod +x "$1"
}
task() { # task <id> <plan step reward.json> <plan step min_reward line, or empty>
t=$root/$1
mkdir -p "$t/steps/plan/tests" "$t/steps/build/tests"
cat > "$t/task.toml" <<EOF
schema_version = "1.4"
[environment]
docker_image = "ubuntu:24.04"
workdir = "/app"
[agent]
timeout_sec = 180
[[steps]]
name = "plan"
$3
[[steps]]
name = "build"
EOF
echo 'Create the file /app/plan.txt whose only line is the word ready.' > "$t/steps/plan/instruction.md"
echo 'Create the file /app/build.txt whose only line is the word done.' > "$t/steps/build/instruction.md"
grader "$t/steps/plan/tests/test.sh" plan.txt ready "$2"
grader "$t/steps/build/tests/test.sh" build.txt done '{"accuracy": 1.0, "overall": 0.5}'
}
task keyed '{"accuracy": 1.0, "overall": 0.5}' 'min_reward = 0.5'
task control '{"accuracy": 1.0, "overall": 0.5}' ''
task below '{"accuracy": 1.0, "overall": 0.1}' 'min_reward = 0.5'
task mapped '{"accuracy": 0.0, "overall": 0.9}' 'min_reward = { accuracy = 0.5 }'
cat > "$root/dataset.toml" <<EOF
[gym.tasks."keyed".verifier]
reward_key = "overall"
[gym.tasks."below".verifier]
reward_key = "overall"
[gym.tasks."mapped".verifier]
reward_key = "overall"
EOFgym eval run ./reward-key-smoke --agent <harness> --sandbox docker \
--model-type openai_model --model <model> --model-url <url> --model-api-key <key>Expected episode result when the agent writes the file each step asks for (field names of
These values come from running these four tasks through the server with FakeSandbox, feeding each step the Every turn's |
Conflicts: - environment_servers/harbor/verifier.py: `_score` keeps main's exit-75 `ignored` message and scores with `select_reward(rewards, key=settings.reward_key)`. - environment_servers/harbor/README.md: main's trimmed README; the reward_key notes move to the reference pages in a later commit. - fern/versions/latest/pages/data/harbor-tasks.mdx: deleted, as on main (the page moved to data/task-folders/); its reward_key notes move there in a later commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
After the merge of main, `test_an_unset_reward_key_leaves_the_digest_of_a_ shaping_dataset_unchanged` failed. It rebuilt the digest from a hand-written copy of `[gym.defaults]`, and the verifier-only budget change on main (NVIDIA-NeMo#6225) added `verifier_timeout_multiplier` to those defaults. The code was right: the merged branch and main give the same digest for this dataset. The test now pins that digest (6895b51f..., what main computes today), so a change to the digest's inputs fails it instead of being copied into it. Dropping `exclude_none=True` from `task_digest` fails it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
Since the Agentic-VBench benchmark entry (NVIDIA-NeMo#4049), main's reward parser skips non-numeric keys (a `details` block, a `reason`) only beside a numeric `reward`; without one every key must be a number. The parser did not know the dataset's `reward_key`, so after the merge a file such as `{"overall": 0.5, "accuracy": 1.0, "details": {...}}` under `reward_key = "overall"` was refused whole: a measured 0 with `harbor:invalid_reward`. The same file with a `reward` key keeps its score. `parse_reward_file` now takes the score key, and `_collect_reward` passes the turn's `reward_key` on both verification paths: - A numeric value under that key makes the other keys metrics, as `reward` does, so non-numeric keys beside it are left out. - A file without the key holds no valid reward. A separately run test.sh that exits 75 is then masked as `harbor:verifier_unavailable`, and a run past its deadline is `harbor:verifier_timeout`, as for any file without a valid reward. Before, such a file parsed as valid and `_score` measured it as `harbor:invalid_reward`. `_score` no longer needs its own missing-key branch. Tests: a `details` block beside the key keeps the score, a file without the key is masked on exit 75 and a timeout past the deadline, and a file whose other keys sum to the score (`{"overall": 0.5}` or with a 0.0 penalty) names no `reward_components`. That last test fails if `reward_components` stops checking the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
`load_task` refused any `reward_key` on a task graded by a Python verifier. That covered more than the message and README said: a task's own `tests/verifier.py` (graded when there is no test.sh) as well as the dataset's `verifier.py`, and even `reward_key = "reward"`, the key the runner writes for a returned number. A dataset that mixes test.sh and Python-graded tasks and sets `[gym.defaults] reward_key` lost every Python-graded task to skipped.json, and no per-task setting could keep them. A Python-graded task now takes `reward_key = "reward"` (or none). Any other value is refused with the grading file named (`tests/verifier.py` or the dataset's `verifier.py`) and the fix spelled out: set `reward_key = "reward"` for that task under `[gym.tasks."<id>".verifier]`. verify() must then return a number or a dict with a `reward` key. The test covers both grading files, the message, and the per-task `"reward"` override beside a test.sh task that keeps the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
The PR's notes lived in the server README and in data/harbor-tasks.mdx; main trimmed the README and moved the page to data/task-folders/, so the merge took main's versions. Main's reference pages then contradicted the key: task-folder-format.mdx said Gym refuses any `[gym]` key it does not list, and multi-step-tasks.mdx said a scalar `min_reward` gates `reward`. - task-folder-format.mdx: a `reward_key` row in `[gym.defaults]`, the per-task override in `[gym.tasks]`, the key in the digest note, and that `gym dataset publish --harbor` drops it with the `[gym]` table. - task-folder-verifiers.mdx: how the key is read, the missing-key outcome (and exit 75 or a timeout), the reward.txt caveat, the task.toml rule, and the "reward"-only rule for Python verifiers. - multi-step-tasks.mdx: a scalar `min_reward` gates `reward_key` when set. - harbor-tasks.mdx and troubleshooting/task-folders.mdx: a pointer, and the missing key under `harbor:invalid_reward`. - `gym dataset init`: a commented `reward_key` line in `[gym.defaults]`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
No conflicts. Nothing main changed since the last merge touches the Harbor server, nemo_gym/data/tasks/harbor, nemo_gym/verifiers or the task folder docs pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
|
Thanks @zuraiz-anjum, this is a well-scoped fix and the tests cover the main paths. The branch had fallen behind main and no longer merged, so I merged main into it (your two commits are unchanged) and added four commits on top:
I then merged main once more. That merge had no conflicts and changed none of the files this PR touches. At the resulting head,
The same run covered three more cases. A separately run Nothing more is needed from you. |
|
/ok to test 308e8e9 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
The preparation failure and documentation NOTE from the previous review are fixed. Both preparation readers now use the loader's shared TOML reader, and the regression test covers empty-string and integer task-local values in both readers. The current references explain selected-key scoring without components, digest changes, and scalar versus named-key gates.
I checked the prior findings and subsequent changes against the previous-review tree, then traced their directly affected preparation and verification paths. I found no new finding eligible under the follow-up impact filter. This was static review; I did not execute tests or rollouts.
RISK: Supply the still-required representative model rollout
Unresolved prior finding: Review 5895799665 requested a representative model smoke rollout. The updated evidence reports real OpenSandbox execution, but explicitly uses one-step oracle tasks without a model. The two-step results still come from FakeSandbox. This does not satisfy the model-driven agent/verifier inspection required by trusted AGENTS.md and CONTRIBUTING.md. The gap covers the real agent-to-verifier path, including the selected-key gate deciding whether another turn runs. This is missing validation, not a demonstrated runtime defect.
Correction: Could a maintainer run the proposed two-step smoke dataset with a real model and supported harness? Ask the model to write /app/plan.txt containing ready, then /app/build.txt containing done. Each grader should check its requested file before writing the corresponding reward, rather than always emitting fixed metrics. Record the tested SHA, exact command, model, harness, sandbox provider, inspected agent work, verifier files and result fields. For example:
gym eval run <dataset> --agent <harness> --sandbox opensandbox \
--model-type openai_model --model <model> --model-url <url> --model-api-key <key>No scoring rewrite is requested. The invariant to validate is that real model work reaches verification and that selected-key scoring controls continuation without changing unset-key scoring or explicitly named thresholds.
Proposed acceptance checks: Inspect the model's file writes and corresponding verifier outputs. When the requested files are correct:
- With
reward_key = "overall", first-stepmin_reward = 0.5, and both graders writing{"accuracy": 1.0, "overall": 0.5}, both turns must run and score0.5; the episode mean must be0.5, both metrics must be retained, and components must be absent. - With the key unset and no scalar gate, the same reward files must score
1.5per turn and for the episode, with both fields as components. - With the selected key and first-step
overall = 0.1, the episode must stop after the first turn. - With the selected key,
{accuracy = 0.5}as the first-step gate, and{"accuracy": 0.0, "overall": 0.9}, the episode must stop despite the selected score exceeding0.5.
Keep the existing FakeSandbox per-turn scoring and scalar-gate assertions; they complement, but do not replace, this missing rollout.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
|
Re the RISK on the agent-to-verifier path and the multi-step Setup. A 3-task dataset whose
The model made real tool calls in every step that ran, two per step: it wrote the file, then ran The second task rules out a gate on the missing |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
I found one non-blocking documentation NOTE and no runtime correctness findings. I inspected all 16 changed files and traced the affected preparation, verification, and episode-result paths.
The earlier preparation failure is fixed: all three readers use the shared TOML reader, and the regression test covers both previously rejected task-local values.
The reported model rollouts resolve the prior missing-smoke-evidence RISK. They report real agent file writes and inspected verifier behavior at this head, including selected-key scoring and scalar gates that both continue and stop an episode.
This was static review. I did not execute tests or independently reproduce the reported rollouts.
Inline comments (escaped v1):
-
environment_servers/harbor/app.pyline 34: NOTE — Suggestion: qualify the remaining reward-rule overviews forreward_key.With dataset
reward_key = "overall"andreward.json = {"accuracy": 1.0, "overall": 0.5}, this overview still promises score1.5and components. The new scoring branch instead produces0.5, keeps both metrics, and omits components. The mixin overview repeats the old rule. This affects maintainers reading the execution contract, not runtime behavior.Could you qualify both overviews? A suggested replacement for lines 34–37 here, keeping the following
failure_kindsentence, is:Reward rule: when ``test.sh`` ran, the sample is measured. A dataset's ``reward_key``, when set, selects the score in ``reward.json``; otherwise a ``reward`` key is the score, or every key is summed when absent. Only that last case names ``reward_components``; with either score key, the other keys are metrics. A missing or invalid reward file scores 0 with a
Static acceptance: both overviews should agree with the updated result contract. The selected-key example must describe
0.5without components; with the key unset, the same file must still describe1.5with both components. Unset-key{"reward": 0.25, "accuracy": 1.0}must retain score0.25without components. These are source-reasoned checks; I did not execute them.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
The module overview in app.py and the verification mixin's docstring still described only the `reward` key and the summed components. Both now say the dataset's `reward_key`, when set, names the score, and that only the summed case names `reward_components`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
|
Addressed the NOTE in |
|
/ok to test b3fd50f |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
I found two non-blocking NOTEs: a failure-classification edge case and misleading verifier docstrings. I found no scoring or continuation defect after inspecting all 16 changed files and their directly affected paths.
The previous NOTE is fixed in the app overview and mixin docstring. The reported model rollouts cover real agent file writes and selected-key gates that both continue and stop an episode.
This was static review. I did not execute tests or reproduce the reported rollouts. One delegated report was unavailable, so I covered its episode and documentation paths directly.
Inline comments (escaped v1):
-
environment_servers/harbor/verifier.pyline 1276: NOTE — Suggestion: classify absent files independently of diagnostic text.With
reward_key = "overall"andreward.json = {"written": 1.0}, the new missing-key diagnostic includeswritten, so_scorereportsharbor:missing_rewardalthough the file exists.reward_key = "written"with{"accuracy": 1.0}has the same result. Both should beharbor:invalid_reward. Scores and masks are unaffected, but the failure kind misdirects diagnosis for these opted-in tasks.Could you replace the substring check at this line with:
kind = ( MISSING_REWARD_KIND if problem == "no reward.json or reward.txt was written" else INVALID_REWARD_KIND )
Only absent files should receive
harbor:missing_reward; interpolated key names must not determine the kind. This replacement matches the parser's absence result and leaves timeout and separate-mode exit-75 precedence unchanged.Proposed acceptance checks: Verify both examples through FakeSandbox and expect reward
0.0,mask_sample is False, andfailure_kind == "harbor:invalid_reward". Retain the missing-file assertion, the ordinary missing-selected-key assertion, and the exit-75 and timeout outcomes. A valid{"written": 0.5}with that key selected must still score0.5without a failure or components. I traced the suggested change through these branches but did not execute it; please verify it before adopting it. -
environment_servers/harbor/verifier.pyline 7: NOTE — Suggestion: clarify the remaining verifier-module contract descriptions.This module overview still describes only
reward, then summation. Withreward_key = "overall"and{"accuracy": 1.0, "overall": 0.5}, it describes1.5with components, while _score returns0.5without components. The parser docstring has the opposite ambiguity: it says an unset key meansrewardand a file lacking it is invalid, although unset-key component files remain valid. This affects maintainers reading the contract, not execution.Could you replace module lines 7–8 with the corrected mixin's rule:
reads reward.json or reward.txt, applies the reward rule (the dataset's ``reward_key`` when set, else the ``reward`` key, else the keys summed as ``reward_components``) and turns the exit into a measured or masked sample. Module level are the
Replace the parser explanation at lines 107–109 with:
``score_key`` is the dataset's ``reward_key``. When set, a file without that key holds no valid reward and reads as ``None`` with the problem named; reward.txt names ``reward``. When unset, a ``reward`` key is the score and otherwise every key is a component whose values are summed.
The descriptions must retain the distinction between selected-key metrics and unset-key components. Static acceptance: the selected-key example describes
0.5without components; the same file with the key unset describes1.5with both components. Unset-key{"reward": 0.25, "accuracy": 1.0}still describes0.25without components.reward.txtstays valid with the key unset or set to"reward", and invalid with"overall". These replacements are source-reasoned suggestions, not executed changes; please verify them before adopting them.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
The failure kind came from a substring check for "written" in the parser's problem text. Under `reward_key`, that text quotes the file's keys, so a file whose keys include "written", or a `reward_key` named "written", was reported as `harbor:missing_reward` although the file exists. The parser now names the absent-file problem by a constant, and only that problem is a missing reward; every other one is `harbor:invalid_reward`. The module and parser docstrings now describe the `reward_key` rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
|
Addressed both NOTEs in
|
|
/ok to test 89e5b3a |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no concerns found. I inspected all 16 changed files and traced configuration through preparation, verification, and episode results.
Both prior NOTEs are fixed. Failure classification now distinguishes absent files from invalid rewards independently of key names, with regression assertions. The verifier overview and parser docstring now distinguish selected-key scoring from unset-key summation.
The reported model rollouts cover real agent file writes and selected-key gates that both continue and stop an episode. This was static review; I did not execute tests or independently reproduce those rollouts. Two delegated reports were unavailable, so I inspected their configuration, episode, and documentation paths directly.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
What does this PR do?
Closes #5641 (a multi-metric
reward.jsonwithout arewardkey scores the sum of its fields).A Harbor verifier that writes several metrics and no
rewardkey is scored as the sum of every field. On the issue's repro ({"accuracy": 1.0, "overall": 0.5}) the result isreward: 1.5with both keys asreward_components, and SkillEvaluator's six metrics plusoverallscore 6.15 instead of 0.85.reward_keyindataset.tomlnames the field that is the score: under[gym.defaults]for every task, or[gym.tasks."<id>".verifier]for one task.rewardis. Its number is the score, and the other keys are metrics, kept inverifier_rewards. A non-numeric field beside it (adetailsblock, areason) is left out instead of failing the file, and the row names noreward_components.harbor:invalid_reward, or, like any file without a valid reward, masked asharbor:verifier_unavailableafter a separately runtest.shexits 75, andharbor:verifier_timeoutpast the deadline.reward.txtreads as arewardkey.min_rewardgatesreward_keywhen it is set. Without that, every step would read negative infinity forrewardand stop the episode.task.tomlcannot set it: the loader ignores it there with the unknown-key warning, as Harbor does, so a task folder scores the same under both. The base-image scan and the build plan drop it too.tests/verifier.pyor the dataset'sverifier.py) takesreward_key = "reward"or none, since the runner writes a returned number asreward. Any other value is refused at load with the grading file named, and the task goes toskipped.json. In a dataset that mixes the two,[gym.tasks."<id>".verifier] reward_key = "reward"keeps each Python-graded task.rewardkey, else the sum) and existing task digests.task_digestdumps the defaults withexclude_none=True, and a test pins the digest main computes for a dataset that only setstimeout_multiplier.Setting
reward_keychanges every task's digest, because the score changes.gym dataset publish --harborleaves out the[gym]table, so Harbor scores a published copy by its own reading ofreward.json.Maintainer updates
Main was merged in at
eff187554(no rebase; the two original commits are unchanged), and four commits were added on top. A second merge of main, atd806a9c68, had no conflicts and changed no file this PR touches. The first merge and the four commits:_scorekeeps main's exit-75 message and scores with the key. Main trimmed the server README and moveddata/harbor-tasks.mdxtodata/task-folders/(the docs move, docs(harbor): document task folders in the job-based nav and trim the harbor server README #5131), so the merge took main's versions and the docs were rewritten into the current pages.[gym.defaults], which lacked theverifier_timeout_multiplieradded on main (the verifier-only budget change, feat(agentic-vbench): training entry, verifier-only budget and compaction continuity on the Harbor path #6225). The code was right. The test now pins the literal digest main computes.reward. Merged as it was, areward_keyfile with adetailsblock was refused whole. The parser now takes the score key. That also makes a file without the key count as "no valid reward" for the exit-75 and timeout rules.tests/verifier.pyandreward_key = "reward"too, while the message named only a datasetverifier.py, so a mixed dataset had no way to keep its Python-graded tasks."reward"is now accepted and the message names the file and the fix.task-folder-format.mdx(the[gym.defaults]row, the per-task override, the digest note, publishing),task-folder-verifiers.mdx(how the key is read, the outcomes, Python verifiers),multi-step-tasks.mdx(min_reward), the tutorial and troubleshooting pages, and a commentedreward_keyline in thegym dataset inittemplate.Validation
Head
308e8e9fb6c49800a514d43e302e8d552818d52d, which merges main atd806a9c68. Python 3.13.18; rootuv.lockwith thedevextra, and the harbor server's ownuv.lockfor its tests.pytest tests/unit_tests/data/tasks/harbor tests/unit_tests/cli tests/unit_tests/verification tests/unit_tests/test_fern_docs_links.py -n 8: 2164 passed, 7 skipped, 5 failed. The 5 failures are theprompt_templatetests, which need jinja2 and fail the same way on main in the root environment. With aCPUSvariable set in the shell, a sixth test,test_document_is_resolved_and_env_file_inlined, also fails: its compose file reads${CPUS:-1}. It fails the same way on main, so these counts are withCPUSunset.pytest tests/unit_tests/data/tasks/harbor -n 8in the harbor server environment, which has jinja2: 1349 passed, 7 skipped.pytest environment_servers/harbor/tests -n 8(harbor server environment): 737 passed.29b89db20, before the second merge, which changed none of these files). With the head's tests on the first merge commit's code, the parser and exit-75 tests (3) and the Python verifier test (1) fail. With the key check removed fromreward_components, the two "names no components" cases fail. Withexclude_none=Trueremoved fromtask_digest, the pinned digest test fails. The digest test as it was fails on the first merge commit.pre-commit run --fileson every file the PR changes passes (the guardwords hook skipped its check: private catalog unavailable).Rollouts
gym dataset validate(oracle) on OpenSandbox at the head above, on two datasets of one-step tasks whosetest.shwrites a fixedreward.json(or whosetests/verifier.pyreturns a fixed number). No model is needed: the oracle'ssolve.shruns, then the verifier.With
[gym.defaults] reward_key = "overall":reward.json(or verifier)failure_kindverifier_rewardsreward_components{"accuracy": 1.0, "overall": 0.5}"overall": 0.85{"overall": 0.5, "accuracy": 1.0, "details": {...}, "reason": "ok"}overall,accuracy{"accuracy": 1.0, "overall": 0.5}, task overridereward_key = "accuracy"{"accuracy": 1.0, "speed": 0.5}harbor:invalid_reward: "reward.json has no 'overall' key (the dataset's reward_key): ['accuracy', 'speed']"{"accuracy": 1.0}, exit 75, separate modeharbor:verifier_unavailabletests/verifier.pyreturning 1.0, task overridereward_key = "reward"{"reward": 1.0}tests/verifier.py, no overrideskipped.json: "reward_key 'overall' applies to test.sh verifiers, and tests/verifier.py grades this task ..."With
reward_keyunset (the current rule, unchanged):reward.jsonreward_components{"accuracy": 1.0, "overall": 0.5}"overall": 0.85No row went to
rollouts_failures.jsonlin either run.With a real model
At
308e8e9fb,gym eval run <dataset> --agent codex_agent(Codex 0.144.4) on OpenSandbox with a hosted GPT modelbehind an OpenAI-compatible endpoint, one rollout per task. The dataset sets
reward_key = "overall"; each step asksthe agent to write a string to a file, and each
test.shwritesreward.jsonwithout arewardkey.reward.json{"overall": 1.0, "accuracy": 1.0, "speed": 0.75}reward1.0, not the sum 2.75min_reward = 0.5{"overall": 1.0, "accuracy": 0.0}overall; step 2 ran;reward1.0min_reward = 0.5{"overall": 0.0, "accuracy": 1.0, "speed": 1.0}reward0.0The model made two tool calls in every step that ran. No row failed, was masked or retried.
Compatibility and benchmark impact
None for existing datasets: with
reward_keyunset, scoring and task digests are unchanged. The key is opt-in, and setting it changes the digests of that dataset's tasks.Checklist
🤖 Generated with Claude Code