Skip to content

feat(harbor): let a dataset name the reward.json key that is the score - #5733

Open
zuraiz-anjum wants to merge 10 commits into
NVIDIA-NeMo:mainfrom
zuraiz-anjum:fix/harbor-reward-key
Open

zuraiz-anjum wants to merge 10 commits into
NVIDIA-NeMo:mainfrom
zuraiz-anjum:fix/harbor-reward-key

Conversation

@zuraiz-anjum

@zuraiz-anjum zuraiz-anjum commented Nov 6, 2026 •

Copy link
Copy Markdown

What does this PR do?

Closes #5641 (a multi-metric reward.json without a reward key scores the sum of its fields).

A Harbor verifier that writes several metrics and no reward key is scored as the sum of every field. On the issue's repro ({"accuracy": 1.0, "overall": 0.5}) the result is reward: 1.5 with both keys as reward_components, and SkillEvaluator's six metrics plus overall score 6.15 instead of 0.85.

reward_key in dataset.toml names the field that is the score: under [gym.defaults] for every task, or [gym.tasks."<id>".verifier] for one task.

  • That key is read as reward is. Its number is the score, and the other keys are metrics, kept in verifier_rewards. A non-numeric field beside it (a details block, a reason) is left out instead of failing the file, and the row names no reward_components.
  • A file without the key holds no valid reward. It is a measured 0 with harbor:invalid_reward, or, like any file without a valid reward, masked as harbor:verifier_unavailable after a separately run test.sh exits 75, and harbor:verifier_timeout past the deadline. reward.txt reads as a reward key.
  • A step's scalar min_reward gates reward_key when it is set. Without that, every step would read negative infinity for reward and stop the episode.
  • A task.toml cannot set it: the loader ignores it there with the unknown-key warning, as Harbor does, so a task folder scores the same under both. The base-image scan and the build plan drop it too.
  • A task graded by a Python verifier (its own tests/verifier.py or the dataset's verifier.py) takes reward_key = "reward" or none, since the runner writes a returned number as reward. Any other value is refused at load with the grading file named, and the task goes to skipped.json. In a dataset that mixes the two, [gym.tasks."<id>".verifier] reward_key = "reward" keeps each Python-graded task.
  • Unset keeps the current rule (a reward key, else the sum) and existing task digests. task_digest dumps the defaults with exclude_none=True, and a test pins the digest main computes for a dataset that only sets timeout_multiplier.

Setting reward_key changes every task's digest, because the score changes. gym dataset publish --harbor leaves out the [gym] table, so Harbor scores a published copy by its own reading of reward.json.

Maintainer updates

Main was merged in at eff187554 (no rebase; the two original commits are unchanged), and four commits were added on top. A second merge of main, at d806a9c68, had no conflicts and changed no file this PR touches. The first merge and the four commits:

  • Merge. _score keeps main's exit-75 message and scores with the key. Main trimmed the server README and moved data/harbor-tasks.mdx to data/task-folders/ (the docs move, docs(harbor): document task folders in the job-based nav and trim the harbor server README #5131), so the merge took main's versions and the docs were rewritten into the current pages.
  • Pinned digest test. After the merge, the digest test failed: it rebuilt the digest from a hand-written copy of [gym.defaults], which lacked the verifier_timeout_multiplier added on main (the verifier-only budget change, feat(agentic-vbench): training entry, verifier-only budget and compaction continuity on the Harbor path #6225). The code was right. The test now pins the literal digest main computes.
  • Reward parsing by the key. Since the Agentic-VBench benchmark entry (feat(agentic-vbench): add the Agentic-VBench Harbor benchmark entry #4049), main's parser skips non-numeric keys only beside a numeric reward. Merged as it was, a reward_key file with a details block was refused whole. The parser now takes the score key. That also makes a file without the key count as "no valid reward" for the exit-75 and timeout rules.
  • Python verifiers. The refusal covered task-local tests/verifier.py and reward_key = "reward" too, while the message named only a dataset verifier.py, so a mixed dataset had no way to keep its Python-graded tasks. "reward" is now accepted and the message names the file and the fix.
  • Docs. task-folder-format.mdx (the [gym.defaults] row, the per-task override, the digest note, publishing), task-folder-verifiers.mdx (how the key is read, the outcomes, Python verifiers), multi-step-tasks.mdx (min_reward), the tutorial and troubleshooting pages, and a commented reward_key line in the gym dataset init template.

Validation

Head 308e8e9fb6c49800a514d43e302e8d552818d52d, which merges main at d806a9c68. Python 3.13.18; root uv.lock with the dev extra, and the harbor server's own uv.lock for its tests.

  • pytest tests/unit_tests/data/tasks/harbor tests/unit_tests/cli tests/unit_tests/verification tests/unit_tests/test_fern_docs_links.py -n 8: 2164 passed, 7 skipped, 5 failed. The 5 failures are the prompt_template tests, which need jinja2 and fail the same way on main in the root environment. With a CPUS variable set in the shell, a sixth test, test_document_is_resolved_and_env_file_inlined, also fails: its compose file reads ${CPUS:-1}. It fails the same way on main, so these counts are with CPUS unset.
  • pytest tests/unit_tests/data/tasks/harbor -n 8 in the harbor server environment, which has jinja2: 1349 passed, 7 skipped.
  • pytest environment_servers/harbor/tests -n 8 (harbor server environment): 737 passed.
  • Each new test fails without its fix (checked at 29b89db20, before the second merge, which changed none of these files). With the head's tests on the first merge commit's code, the parser and exit-75 tests (3) and the Python verifier test (1) fail. With the key check removed from reward_components, the two "names no components" cases fail. With exclude_none=True removed from task_digest, the pinned digest test fails. The digest test as it was fails on the first merge commit.
  • pre-commit run --files on every file the PR changes passes (the guardwords hook skipped its check: private catalog unavailable).

Rollouts

gym dataset validate (oracle) on OpenSandbox at the head above, on two datasets of one-step tasks whose test.sh writes a fixed reward.json (or whose tests/verifier.py returns a fixed number). No model is needed: the oracle's solve.sh runs, then the verifier.

gym dataset validate <dataset> --sandbox opensandbox

With [gym.defaults] reward_key = "overall":

Task reward.json (or verifier) reward failure_kind verifier_rewards reward_components
overall {"accuracy": 1.0, "overall": 0.5} 0.5 none both keys absent
skilleval SkillEvaluator's six metrics plus "overall": 0.85 0.85 none all seven keys absent
details {"overall": 0.5, "accuracy": 1.0, "details": {...}, "reason": "ok"} 0.5 none overall, accuracy absent
acc_override {"accuracy": 1.0, "overall": 0.5}, task override reward_key = "accuracy" 1.0 none both keys absent
missing {"accuracy": 1.0, "speed": 0.5} 0 (measured) harbor:invalid_reward: "reward.json has no 'overall' key (the dataset's reward_key): ['accuracy', 'speed']" none absent
exit75 {"accuracy": 1.0}, exit 75, separate mode 0, masked harbor:verifier_unavailable none absent
pyverify tests/verifier.py returning 1.0, task override reward_key = "reward" 1.0 none {"reward": 1.0} absent
pyrefused tests/verifier.py, no override not run in skipped.json: "reward_key 'overall' applies to test.sh verifiers, and tests/verifier.py grades this task ..."

With reward_key unset (the current rule, unchanged):

Task reward.json reward reward_components
sum {"accuracy": 1.0, "overall": 0.5} 1.5 both keys
skilleval0 SkillEvaluator's six metrics plus "overall": 0.85 6.15 all seven keys

No row went to rollouts_failures.jsonl in either run.

With a real model

At 308e8e9fb, gym eval run <dataset> --agent codex_agent (Codex 0.144.4) on OpenSandbox with a hosted GPT model
behind an OpenAI-compatible endpoint, one rollout per task. The dataset sets reward_key = "overall"; each step asks
the agent to write a string to a file, and each test.sh writes reward.json without a reward key.

Task reward.json Result
One step {"overall": 1.0, "accuracy": 1.0, "speed": 0.75} reward 1.0, not the sum 2.75
Two steps, step 1 min_reward = 0.5 step 1 {"overall": 1.0, "accuracy": 0.0} Gate passed on overall; step 2 ran; reward 1.0
Two steps, step 1 min_reward = 0.5 step 1 {"overall": 0.0, "accuracy": 1.0, "speed": 1.0} Stopped after step 1 (the sum would have passed); step 2's agent never ran; reward 0.0

The model made two tool calls in every step that ran. No row failed, was masked or retried.

Compatibility and benchmark impact

None for existing datasets: with reward_key unset, scoring and task digests are unchanged. The key is opt-in, and setting it changes the digests of that dataset's tasks.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally on the changed files (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header. (No new files.)
  • All commits have DCO sign-off (so the DCO check passes).

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Nov 6, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added feature New capabilities, enhancements, or enablement work area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Nov 7, 2026
A Harbor verifier that writes several metrics and no `reward` key is
scored as the sum of every field, so a verifier that writes six 0 to 1
metrics plus `overall` scores up to about 7.

`reward_key` in dataset.toml (`[gym.defaults]`, or
`[gym.tasks."<id>".verifier]` for one task) names the field that is the
score. The other keys stay in `verifier_rewards` as metrics, a file
without the key is measured 0 with `harbor:invalid_reward`, and a step's
scalar `min_reward` gates that key. A task.toml cannot set it (the loader
ignores it there with the unknown-key warning, as Harbor does), and a
task graded by a Python verifier refuses it, since the runner only
writes a `reward` key or a single metric. Unset keeps the current rule
(a `reward` key, else the sum) and existing task digests.

Closes NVIDIA-NeMo#5641

Signed-off-by: Zuraiz Anjum <zuraizwork@gmail.com>
@zuraiz-anjum
zuraiz-anjum force-pushed the fix/harbor-reward-key branch from c8706e8 to 000034d Compare November 8, 2026 07:24
@zuraiz-anjum
zuraiz-anjum marked this pull request as ready for review November 8, 2026 11:18
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Nov 8, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 9, 2026
@ananthsub
ananthsub requested a review from adil-a November 10, 2026 21:29
@github-actions github-actions Bot added sla:review-overdue Review response is over the one-business-day SLA and removed sla:triage-overdue Review assignment is over the one-business-day SLA labels Nov 10, 2026
@adil-a

adil-a commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

/ok to test 000034d

@adil-a

adil-a commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

/review

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

I found two RISK findings and one documentation NOTE. I inspected all ten changed files and their directly affected configuration, preparation, scoring, and multi-turn paths. This was a static review; I didn't execute tests.

RISK — Ignore task-local reward_key consistently during preparation

Failure: An otherwise valid task with [verifier] reward_key = "" and no dataset-level setting is no longer prepared consistently. load_task deliberately removes that unknown task-local field, but the earlier dockerfile_base_images and plan_builds readers validate raw TOML against the newly constrained field. They catch the validation error and omit the task. With no existing records, a Dockerfile containing FROM ubuntu:24.04\nWORKDIR src\n gets no base-image record; one containing FROM ubuntu:24.04\nCOPY . /app\n gets no build entry. Discovery then rejects the task for missing preparation outputs. Before this PR, these readers ignored the unknown key. This affects task folders without opting into the feature.

Correction: In both preparation readers linked above, replace the raw-TOML validation statement with the following, keeping apply_dataset_config afterward:

data = tomllib.loads((child / TASK_FILE).read_text())
if isinstance(verifier_table := data.get("verifier"), dict):
    data["verifier"] = {key: value for key, value in verifier_table.items() if key != "reward_key"}
config = HarborTaskConfig.model_validate(data)

This restores the invariant that raw task-local reward_key is ignored before validation, while effective dataset settings remain validated. I traced this suggested replacement through both callers, but it wasn't executed; please verify it before adopting it.

Preservation and acceptance — proposed tests: With no image/build records, parameterize task-local values as "" and 1. The relative-WORKDIR fixture should yield {"ubuntu:24.04": ["a"]} from dockerfile_base_images; the COPY fixture should produce a build entry containing task a. Retain the existing configuration assertions: dataset defaults select overall, a task override selects score, an empty dataset key is rejected, a task-local key is ignored with a warning, opting in changes the digest, and leaving the key unset preserves the previous digest. Other verifier fields and environment overrides must retain their validation and application.

RISK — Provide the required real smoke-rollout evidence

The PR changes environment behavior when enabled, including the score-selection branch and the gate controlling whether another agent turn runs. The description reports FakeSandbox validation and explicitly no real model rollouts. The trusted AGENTS.md and CONTRIBUTING.md require a representative model rollout for behavior-changing environment work; opt-in behavior has no stated exemption. This is a validation gap, not a demonstrated runtime defect.

Could you run a small real two-step Harbor task with a model, reward_key = "overall", and a scalar first-step min_reward, then record the command, model/harness/provider, inspected agent work, verifier files, and resulting scores? For files containing {"accuracy": 1.0, "overall": 0.5} on both turns with min_reward = 0.5, acceptance is two turns scoring 0.5, episode mean 0.5, both metrics retained, and no components. An unset-key control without the scalar gate should still score 1.5 with both fields as components. Retain the added below-threshold case, where overall = 0.1 stops the episode, and keep mapping thresholds tied to their explicitly named keys. No scoring rewrite is requested; the correction is to supply and inspect the required integration evidence.


Inline comments (escaped v1):

  • fern/versions/latest/pages/data/harbor-tasks.mdx line 107: NOTE — Suggestion: qualify the existing reward-contract descriptions for reward_key.

    With reward_key = "overall" and reward.json = {"accuracy": 1.0, "overall": 0.5}, this paragraph still promises components because there is no reward key. The implementation instead scores 0.5 and names no components. The adjacent digest and fixed-objective guidance, result docstring, and README gate description also describe only the previous contract.

    Could you replace the components condition here with “when reward_key is unset and the verifier's reward.json named no reward key”? Apply that qualifier to the result docstring and fixed-objective/GDPO guidance, include default reward_key alongside scaling in the digest explanation, and describe the scalar gate as “a number gates reward_key when set, else reward; a table gates each named key.” These are documentation-only corrections; retain the valid unset-key sum/components rule.

    Static acceptance: selected overall means score 0.5, no components, and a scalar gate reading overall; unset means score 1.5, both components, and a scalar gate reading reward. In both cases {accuracy = 0.5} still gates accuracy. Check the revised wording against these cases; these checks weren't executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Nov 13, 2026
The base-image scan and the build plan validated task.toml with a
task-local `[verifier] reward_key` still in place, while the loader
drops it with a warning. A value the field refuses (an empty string,
a number) left the task out of both scans, and discovery then refused
it for the missing records. The loader and both scans now read
task.toml through `read_task_toml`, which drops the key before
validation.

The docs and docstrings now qualify the reward contract for
`reward_key`: components are named only when it is unset, a scalar
`min_reward` gates it when set, and it enters the task digest.

Signed-off-by: Zuraiz Anjum <zuraizwork@gmail.com>
@zuraiz-anjum

Copy link
Copy Markdown
Author

Pushed 35e5def for the review.

Task-local reward_key in preparation. Right: I dropped the key in load_task but missed the two preparation readers, dockerfile_base_images and plan_builds, which validate task.toml on their own. A task-local reward_key = "" or = 1 failed their validation, so the task got no base-image record or build entry and discovery then refused it.

Instead of copying the strip into each reader, I added one helper, read_task_toml in nemo_gym/data/tasks/harbor/task.py, that parses task.toml and drops [verifier].reward_key before validation. load_task and both readers use it, so the rule lives in one place. The loader's warning is unchanged, and dataset-level values are still validated where they were.

The new test is parameterized over "" and 1 with no image or build records. The relative WORKDIR fixture gives {"ubuntu:24.04": ["a"]} from dockerfile_base_images, the COPY fixture gives one build entry for a, and the loader still warns and ignores the key. Each case runs with no dataset.toml and with [gym.defaults] reward_key = "overall". The existing assertions are unchanged and pass: dataset default selects overall, task override selects score, empty dataset key rejected, task-local key ignored with a warning, opting in changes the digest, and the unset key keeps the previous digest.

  • pytest tests/unit_tests/data/tasks/harbor/test_harbor_tasks.py -k task_local_reward_key with the source of 000034d: 2 failed (the base-image scan returns {}). With only image_configs.py fixed, both still fail on plan_builds ([] == [("a",)]), so each reader is covered on its own. At 35e5def: 2 passed.
  • pytest environment_servers/harbor/tests/test_app.py environment_servers/harbor/tests/test_multi_turn.py tests/unit_tests/data/tasks/harbor/test_harbor_tasks.py -k "reward_key or min_reward": 19 passed (17 before).
  • The three touched modules plus the reader modules (test_harbor_image_configs.py, test_harbor_build_plan.py, test_harbor_build_inputs.py, test_harbor_build_compose.py, test_harbor_compose_prepare.py) and tests/unit_tests/test_moved_paths.py, with -m "not sandbox" -n 4: 1095 passed at 000034d, 1097 passed at 35e5def.
  • Ruff 0.9.9 (the pre-commit pin): check, import sort and format clean on the changed files; pre-commit run on the PR's files passes.

Docs. Applied your wording in harbor-tasks.mdx, the HarborEpisodeResult docstring and the README:

  • components are named "when reward_key is unset and the verifier's reward.json named no reward key";
  • the fixed-objective/GDPO guidance says to leave reward_key unset;
  • the digest sentence lists reward_key next to the scaling defaults;
  • the gate reads "a number gates reward_key when the dataset sets it, else reward; a table gates each named key".

The same qualifier also went into the HarborTurnResult.reward_components comment, the HarborTaskStep docstring (which still said a number gates reward) and the README's general reward.json paragraph. Checked against your cases: with overall selected the score is 0.5, there are no components and the scalar gate reads overall. Unset gives 1.5 with both components and the gate on reward. In both, {accuracy = 0.5} gates accuracy.

Smoke rollout. I can't run a real model rollout myself because I don't have model access. The scoring and gating branches are covered with FakeSandbox through the server: the per-turn score under reward_key, the scalar gate on overall passing and stopping, the unset-key sum, and the invalid reward when the key is missing. Could a maintainer run the smoke task below? It's four two-step tasks on ubuntu:24.04 and one gym eval run with the local Docker sandbox. I'll fix anything it turns up.

Smoke dataset, command and expected results

Run in an empty folder; it writes ./reward-key-smoke.

#!/bin/bash
set -eu
root=reward-key-smoke

grader() {  # grader <test.sh path> <file under /app> <word> <reward.json when the agent did the step>
  cat > "$1" <<EOF
#!/bin/bash
if grep -qx $3 /app/$2 2>/dev/null; then
  echo '$4' > /logs/verifier/reward.json
else
  echo '{"accuracy": 0.0, "overall": 0.0}' > /logs/verifier/reward.json
fi
EOF
  chmod +x "$1"
}

task() {  # task <id> <plan step reward.json> <plan step min_reward line, or empty>
  t=$root/$1
  mkdir -p "$t/steps/plan/tests" "$t/steps/build/tests"
  cat > "$t/task.toml" <<EOF
schema_version = "1.4"

[environment]
docker_image = "ubuntu:24.04"
workdir = "/app"

[agent]
timeout_sec = 180

[[steps]]
name = "plan"
$3

[[steps]]
name = "build"
EOF
  echo 'Create the file /app/plan.txt whose only line is the word ready.' > "$t/steps/plan/instruction.md"
  echo 'Create the file /app/build.txt whose only line is the word done.' > "$t/steps/build/instruction.md"
  grader "$t/steps/plan/tests/test.sh" plan.txt ready "$2"
  grader "$t/steps/build/tests/test.sh" build.txt done '{"accuracy": 1.0, "overall": 0.5}'
}

task keyed   '{"accuracy": 1.0, "overall": 0.5}' 'min_reward = 0.5'
task control '{"accuracy": 1.0, "overall": 0.5}' ''
task below   '{"accuracy": 1.0, "overall": 0.1}' 'min_reward = 0.5'
task mapped  '{"accuracy": 0.0, "overall": 0.9}' 'min_reward = { accuracy = 0.5 }'

cat > "$root/dataset.toml" <<EOF
[gym.tasks."keyed".verifier]
reward_key = "overall"

[gym.tasks."below".verifier]
reward_key = "overall"

[gym.tasks."mapped".verifier]
reward_key = "overall"
EOF
gym eval run ./reward-key-smoke --agent <harness> --sandbox docker \
  --model-type openai_model --model <model> --model-url <url> --model-api-key <key>

Expected episode result when the agent writes the file each step asks for (field names of HarborEpisodeResult and its turns):

task turns[].name turns[].reward reward reward_components (episode and each turn) stopped_after_turn
keyed plan, build 0.5, 0.5 0.5 absent null
control plan, build 1.5, 1.5 1.5 {"accuracy": 1.0, "overall": 0.5} null
below plan 0.1 0.1 absent "plan"
mapped plan 0.9 0.9 absent "plan"

These values come from running these four tasks through the server with FakeSandbox, feeding each step the reward.json its grader writes; no model was involved.

Every turn's verifier_rewards holds both keys as written. If the agent misses the plan step, its grader writes zeros: keyed, below and mapped then stop after plan with reward 0.0, and control, which has no gate, runs both turns for 0.75. That is a model miss, not a scoring fault.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added needs-review PR is ready for code review and waiting on a reviewer waiting-on-maintainers Waiting on maintainers to respond labels Nov 16, 2026
adil-a and others added 6 commits November 17, 2026 20:33
Conflicts:
- environment_servers/harbor/verifier.py: `_score` keeps main's exit-75
  `ignored` message and scores with `select_reward(rewards,
  key=settings.reward_key)`.
- environment_servers/harbor/README.md: main's trimmed README; the
  reward_key notes move to the reference pages in a later commit.
- fern/versions/latest/pages/data/harbor-tasks.mdx: deleted, as on main
  (the page moved to data/task-folders/); its reward_key notes move
  there in a later commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
After the merge of main, `test_an_unset_reward_key_leaves_the_digest_of_a_
shaping_dataset_unchanged` failed. It rebuilt the digest from a hand-written
copy of `[gym.defaults]`, and the verifier-only budget change on main (NVIDIA-NeMo#6225) added
`verifier_timeout_multiplier` to those defaults. The code was right: the
merged branch and main give the same digest for this dataset.

The test now pins that digest (6895b51f..., what main computes today), so
a change to the digest's inputs fails it instead of being copied into it.
Dropping `exclude_none=True` from `task_digest` fails it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
Since the Agentic-VBench benchmark entry (NVIDIA-NeMo#4049), main's reward parser
skips non-numeric keys (a `details` block, a `reason`) only beside a
numeric `reward`; without one every key must be a number. The parser did
not know the dataset's `reward_key`, so after the merge a file such as
`{"overall": 0.5, "accuracy": 1.0, "details": {...}}` under `reward_key
= "overall"` was refused whole: a measured 0 with
`harbor:invalid_reward`. The same file with a `reward` key keeps its
score.

`parse_reward_file` now takes the score key, and `_collect_reward` passes
the turn's `reward_key` on both verification paths:
- A numeric value under that key makes the other keys metrics, as
  `reward` does, so non-numeric keys beside it are left out.
- A file without the key holds no valid reward. A separately run
  test.sh that exits 75 is then masked as `harbor:verifier_unavailable`,
  and a run past its deadline is `harbor:verifier_timeout`, as for any
  file without a valid reward. Before, such a file parsed as valid and
  `_score` measured it as `harbor:invalid_reward`. `_score` no longer
  needs its own missing-key branch.

Tests: a `details` block beside the key keeps the score, a file without
the key is masked on exit 75 and a timeout past the deadline, and a file
whose other keys sum to the score (`{"overall": 0.5}` or with a 0.0
penalty) names no `reward_components`. That last test fails if
`reward_components` stops checking the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
`load_task` refused any `reward_key` on a task graded by a Python
verifier. That covered more than the message and README said: a task's
own `tests/verifier.py` (graded when there is no test.sh) as well as the
dataset's `verifier.py`, and even `reward_key = "reward"`, the key the
runner writes for a returned number. A dataset that mixes test.sh and
Python-graded tasks and sets `[gym.defaults] reward_key` lost every
Python-graded task to skipped.json, and no per-task setting could keep
them.

A Python-graded task now takes `reward_key = "reward"` (or none). Any
other value is refused with the grading file named (`tests/verifier.py`
or the dataset's `verifier.py`) and the fix spelled out: set
`reward_key = "reward"` for that task under `[gym.tasks."<id>".verifier]`.
verify() must then return a number or a dict with a `reward` key.

The test covers both grading files, the message, and the per-task
`"reward"` override beside a test.sh task that keeps the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
The PR's notes lived in the server README and in data/harbor-tasks.mdx;
main trimmed the README and moved the page to data/task-folders/, so the
merge took main's versions. Main's reference pages then contradicted the
key: task-folder-format.mdx said Gym refuses any `[gym]` key it does not
list, and multi-step-tasks.mdx said a scalar `min_reward` gates `reward`.

- task-folder-format.mdx: a `reward_key` row in `[gym.defaults]`, the
  per-task override in `[gym.tasks]`, the key in the digest note, and
  that `gym dataset publish --harbor` drops it with the `[gym]` table.
- task-folder-verifiers.mdx: how the key is read, the missing-key
  outcome (and exit 75 or a timeout), the reward.txt caveat, the
  task.toml rule, and the "reward"-only rule for Python verifiers.
- multi-step-tasks.mdx: a scalar `min_reward` gates `reward_key` when set.
- harbor-tasks.mdx and troubleshooting/task-folders.mdx: a pointer, and
  the missing key under `harbor:invalid_reward`.
- `gym dataset init`: a commented `reward_key` line in `[gym.defaults]`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
No conflicts. Nothing main changed since the last merge touches the
Harbor server, nemo_gym/data/tasks/harbor, nemo_gym/verifiers or the
task folder docs pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thanks @zuraiz-anjum, this is a well-scoped fix and the tests cover the main paths.

The branch had fallen behind main and no longer merged, so I merged main into it (your two commits are unchanged) and added four commits on top:

  • Digest test. After the merge, your digest test failed: main added verifier_timeout_multiplier to [gym.defaults] (the verifier-only budget change, feat(agentic-vbench): training entry, verifier-only budget and compaction continuity on the Harbor path #6225), and the test rebuilt the digest by hand. Your code was right, and the merged digest matches main's. The test now pins the literal value instead.
  • Reward parsing. Main's parser now skips a details block or a reason string only beside a numeric reward (the Agentic-VBench entry, feat(agentic-vbench): add the Agentic-VBench Harbor benchmark entry #4049). Merged as it was, a reward_key = "overall" file with such a field scored 0 as harbor:invalid_reward. The parser now reads the dataset's key. That also means a file without the key is masked as harbor:verifier_unavailable on a separate-mode exit 75, and is a harbor:verifier_timeout past the deadline, as for any file without a valid reward.
  • Python verifiers. The load check also refused a task's own tests/verifier.py and reward_key = "reward", so a dataset that mixes test.sh and Python-graded tasks could not keep the Python ones. "reward" is now accepted, and the message names the grading file and the per-task fix.
  • Docs. Main trimmed the server README and moved harbor-tasks.mdx (the docs move, docs(harbor): document task folders in the job-based nav and trim the harbor server README #5131), so I moved your notes into the current reference pages, task-folder-format.mdx and task-folder-verifiers.mdx among them, and added a commented reward_key line to the gym dataset init template.

I then merged main once more. That merge had no conflicts and changed none of the files this PR touches. At the resulting head, 308e8e9fb6c49800a514d43e302e8d552818d52d, I ran gym dataset validate on OpenSandbox:

reward.json reward_key Reward
{"accuracy": 1.0, "overall": 0.5} unset 1.5 (sum, unchanged)
same overall 0.5, no reward_components
same overall, task override accuracy 1.0
{"accuracy": 1.0, "speed": 0.5} (no overall) overall measured 0, harbor:invalid_reward
SkillEvaluator-shaped (six metrics plus overall 0.85) overall 0.85
same unset 6.15 (sum, unchanged)
with a details object and a reason string overall 0.5, details and reason dropped from the metrics

The same run covered three more cases. A separately run test.sh that exits 75 without the key is masked as harbor:verifier_unavailable. A task graded by its own tests/verifier.py scores 1.0 with a per-task reward_key = "reward". Without that override, the task is refused at load and listed in skipped.json.

Nothing more is needed from you.

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/ok to test 308e8e9

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the risk:high Autobot PR risk: high label Nov 18, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

The preparation failure and documentation NOTE from the previous review are fixed. Both preparation readers now use the loader's shared TOML reader, and the regression test covers empty-string and integer task-local values in both readers. The current references explain selected-key scoring without components, digest changes, and scalar versus named-key gates.

I checked the prior findings and subsequent changes against the previous-review tree, then traced their directly affected preparation and verification paths. I found no new finding eligible under the follow-up impact filter. This was static review; I did not execute tests or rollouts.

RISK: Supply the still-required representative model rollout

Unresolved prior finding: Review 5895799665 requested a representative model smoke rollout. The updated evidence reports real OpenSandbox execution, but explicitly uses one-step oracle tasks without a model. The two-step results still come from FakeSandbox. This does not satisfy the model-driven agent/verifier inspection required by trusted AGENTS.md and CONTRIBUTING.md. The gap covers the real agent-to-verifier path, including the selected-key gate deciding whether another turn runs. This is missing validation, not a demonstrated runtime defect.

Correction: Could a maintainer run the proposed two-step smoke dataset with a real model and supported harness? Ask the model to write /app/plan.txt containing ready, then /app/build.txt containing done. Each grader should check its requested file before writing the corresponding reward, rather than always emitting fixed metrics. Record the tested SHA, exact command, model, harness, sandbox provider, inspected agent work, verifier files and result fields. For example:

gym eval run <dataset> --agent <harness> --sandbox opensandbox \
  --model-type openai_model --model <model> --model-url <url> --model-api-key <key>

No scoring rewrite is requested. The invariant to validate is that real model work reaches verification and that selected-key scoring controls continuation without changing unset-key scoring or explicitly named thresholds.

Proposed acceptance checks: Inspect the model's file writes and corresponding verifier outputs. When the requested files are correct:

  • With reward_key = "overall", first-step min_reward = 0.5, and both graders writing {"accuracy": 1.0, "overall": 0.5}, both turns must run and score 0.5; the episode mean must be 0.5, both metrics must be retained, and components must be absent.
  • With the key unset and no scalar gate, the same reward files must score 1.5 per turn and for the episode, with both fields as components.
  • With the selected key and first-step overall = 0.1, the episode must stop after the first turn.
  • With the selected key, {accuracy = 0.5} as the first-step gate, and {"accuracy": 0.0, "overall": 0.9}, the episode must stop despite the selected score exceeding 0.5.

Keep the existing FakeSandbox per-turn scoring and scalar-gate assertions; they complement, but do not replace, this missing rollout.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Re the RISK on the agent-to-verifier path and the multi-step min_reward gate: we ran real-model rollouts at head
308e8e9fb6c49800a514d43e302e8d552818d52d.

Setup. A 3-task dataset whose dataset.toml sets [gym.defaults] reward_key = "overall", run with
gym eval run <dataset> --agent codex_agent (Codex 0.144.4) on the opensandbox provider. The model was a hosted GPT
model behind an OpenAI-compatible endpoint, with one rollout per task. Each step asks the agent to write a given
string to a file and cat it. Each test.sh checks the file and writes reward.json without a reward key.
gym dataset validate (oracle) ran first and gave the same rewards.

Task reward.json Result
single step {"overall": 1.0, "accuracy": 1.0, "speed": 0.75} reward 1.0, not the sum 2.75. verifier_rewards keeps all three keys, reward_components is absent, and aggregate metrics carry only reward.
2 steps, step 1 min_reward = 0.5 step 1 {"overall": 1.0, "accuracy": 0.0}, step 2 {"overall": 1.0, "accuracy": 1.0} The gate passed on overall and step 2 ran. Both turns are listed, reward is 1.0, verifier_rewards.accuracy is 0.5 (mean), and stopped_after_turn is null.
2 steps, step 1 min_reward = 0.5 step 1 always {"overall": 0.0, "accuracy": 1.0, "speed": 1.0} The episode stopped after step 1: stopped_after_turn = "01-write-answer", one turn, reward 0.0. Step 2's agent never ran: there is no turn or verifier run for it, and its instruction appears nowhere in the row.

The model made real tool calls in every step that ran, two per step: it wrote the file, then ran cat. Each
verifier's log shows the file contents it read and the transcript of that step.

The second task rules out a gate on the missing reward key, which would have stopped the episode. The third rules
out a gate on the sum (2.0) or on any other key at 1.0, which would have let step 2 run. So a scalar min_reward
gates the dataset's reward_key, as multi-step-tasks.mdx ("Early stop") says. No row failed or was masked, and no
retry was needed.

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

I found one non-blocking documentation NOTE and no runtime correctness findings. I inspected all 16 changed files and traced the affected preparation, verification, and episode-result paths.

The earlier preparation failure is fixed: all three readers use the shared TOML reader, and the regression test covers both previously rejected task-local values.

The reported model rollouts resolve the prior missing-smoke-evidence RISK. They report real agent file writes and inspected verifier behavior at this head, including selected-key scoring and scalar gates that both continue and stop an episode.

This was static review. I did not execute tests or independently reproduce the reported rollouts.


Inline comments (escaped v1):

  • environment_servers/harbor/app.py line 34: NOTE — Suggestion: qualify the remaining reward-rule overviews for reward_key.

    With dataset reward_key = "overall" and reward.json = {"accuracy": 1.0, "overall": 0.5}, this overview still promises score 1.5 and components. The new scoring branch instead produces 0.5, keeps both metrics, and omits components. The mixin overview repeats the old rule. This affects maintainers reading the execution contract, not runtime behavior.

    Could you qualify both overviews? A suggested replacement for lines 34–37 here, keeping the following failure_kind sentence, is:

    Reward rule: when ``test.sh`` ran, the sample is measured. A dataset's ``reward_key``,
    when set, selects the score in ``reward.json``; otherwise a ``reward`` key is the score,
    or every key is summed when absent. Only that last case names ``reward_components``;
    with either score key, the other keys are metrics. A missing or invalid reward file scores 0 with a

    Static acceptance: both overviews should agree with the updated result contract. The selected-key example must describe 0.5 without components; with the key unset, the same file must still describe 1.5 with both components. Unset-key {"reward": 0.25, "accuracy": 1.0} must retain score 0.25 without components. These are source-reasoned checks; I did not execute them.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

The module overview in app.py and the verification mixin's docstring still described only the
`reward` key and the summed components. Both now say the dataset's `reward_key`, when set, names
the score, and that only the summed case names `reward_components`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Addressed the NOTE in b3fd50f26: the module overview in environment_servers/harbor/app.py and the HarborVerificationMixin docstring in verifier.py now say the dataset's reward_key, when set, names the score (the other keys are metrics), and that only the summed case names reward_components. Docstrings only; ruff and pre-commit are clean on both files.

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/ok to test b3fd50f

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

I found two non-blocking NOTEs: a failure-classification edge case and misleading verifier docstrings. I found no scoring or continuation defect after inspecting all 16 changed files and their directly affected paths.

The previous NOTE is fixed in the app overview and mixin docstring. The reported model rollouts cover real agent file writes and selected-key gates that both continue and stop an episode.

This was static review. I did not execute tests or reproduce the reported rollouts. One delegated report was unavailable, so I covered its episode and documentation paths directly.


Inline comments (escaped v1):

  • environment_servers/harbor/verifier.py line 1276: NOTE — Suggestion: classify absent files independently of diagnostic text.

    With reward_key = "overall" and reward.json = {"written": 1.0}, the new missing-key diagnostic includes written, so _score reports harbor:missing_reward although the file exists. reward_key = "written" with {"accuracy": 1.0} has the same result. Both should be harbor:invalid_reward. Scores and masks are unaffected, but the failure kind misdirects diagnosis for these opted-in tasks.

    Could you replace the substring check at this line with:

    kind = (
        MISSING_REWARD_KIND
        if problem == "no reward.json or reward.txt was written"
        else INVALID_REWARD_KIND
    )

    Only absent files should receive harbor:missing_reward; interpolated key names must not determine the kind. This replacement matches the parser's absence result and leaves timeout and separate-mode exit-75 precedence unchanged.

    Proposed acceptance checks: Verify both examples through FakeSandbox and expect reward 0.0, mask_sample is False, and failure_kind == "harbor:invalid_reward". Retain the missing-file assertion, the ordinary missing-selected-key assertion, and the exit-75 and timeout outcomes. A valid {"written": 0.5} with that key selected must still score 0.5 without a failure or components. I traced the suggested change through these branches but did not execute it; please verify it before adopting it.

  • environment_servers/harbor/verifier.py line 7: NOTE — Suggestion: clarify the remaining verifier-module contract descriptions.

    This module overview still describes only reward, then summation. With reward_key = "overall" and {"accuracy": 1.0, "overall": 0.5}, it describes 1.5 with components, while _score returns 0.5 without components. The parser docstring has the opposite ambiguity: it says an unset key means reward and a file lacking it is invalid, although unset-key component files remain valid. This affects maintainers reading the contract, not execution.

    Could you replace module lines 7–8 with the corrected mixin's rule:

    reads reward.json or reward.txt, applies the reward rule (the dataset's ``reward_key`` when set,
    else the ``reward`` key, else the keys summed as ``reward_components``) and turns the exit into a
    measured or masked sample. Module level are the

    Replace the parser explanation at lines 107–109 with:

    ``score_key`` is the dataset's ``reward_key``. When set, a file without that key holds no valid
    reward and reads as ``None`` with the problem named; reward.txt names ``reward``. When unset,
    a ``reward`` key is the score and otherwise every key is a component whose values are summed.

    The descriptions must retain the distinction between selected-key metrics and unset-key components. Static acceptance: the selected-key example describes 0.5 without components; the same file with the key unset describes 1.5 with both components. Unset-key {"reward": 0.25, "accuracy": 1.0} still describes 0.25 without components. reward.txt stays valid with the key unset or set to "reward", and invalid with "overall". These replacements are source-reasoned suggestions, not executed changes; please verify them before adopting them.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

The failure kind came from a substring check for "written" in the parser's problem text. Under
`reward_key`, that text quotes the file's keys, so a file whose keys include "written", or a
`reward_key` named "written", was reported as `harbor:missing_reward` although the file exists.
The parser now names the absent-file problem by a constant, and only that problem is a missing
reward; every other one is `harbor:invalid_reward`. The module and parser docstrings now describe
the `reward_key` rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Addressed both NOTEs in 89e5b3af4:

  • Failure kind: parse_reward_file now returns a NO_REWARD_FILE constant for an absent file, and _score gives harbor:missing_reward only for that problem; every other problem is harbor:invalid_reward. New tests: reward_key = "overall" with {"written": 1.0} and reward_key = "written" with {"accuracy": 1.0} are harbor:invalid_reward (both failed before the fix), and {"written": 0.5} under reward_key = "written" scores 0.5 with no failure or components. The missing-file, exit-75 and timeout tests are unchanged and pass.
  • Docstrings: the module overview and the parse_reward_file docstring now state the reward_key rule, as suggested.

environment_servers/harbor/tests: 740 passed. ruff and pre-commit clean on the touched files.

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/ok to test 89e5b3a

@adil-a

adil-a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found. I inspected all 16 changed files and traced configuration through preparation, verification, and episode results.

Both prior NOTEs are fixed. Failure classification now distinguishes absent files from invalid rewards independently of key names, with regression assertions. The verifier overview and parser docstring now distinguish selected-key scoring from unset-key summation.

The reported model rollouts cover real agent file writes and selected-key gates that both continue and stop an episode. This was static review; I did not execute tests or independently reproduce those rollouts. Two delegated reports were unavailable, so I inspected their configuration, episode, and documentation paths directly.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Nov 18, 2026

This branch was previously deployed

1 inactive (outdated) deployment
public — 000034d8 Deployed Nov 13, 2026 by copy-pr-bot[bot] via release / finalize / notify #12380
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers community-request Issue reported or requested by someone from the community feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer risk:high Autobot PR risk: high sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

harbor: reward.json without a reward key is scored as the sum of its fields

3 participants