Skip to content

fix(reasoning_gym): score the answer the agent ended on; ask Fabric DeepAgents for <answer> tags - #5324

Merged
qiaochuz-nv merged 6 commits into
mainfrom
qiaochuz/nvb-6900506-reasoning-gym-answer-extraction
Nov 13, 2026
Merged

qiaochuz-nv merged 6 commits into
mainfrom
qiaochuz/nvb-6900506-reasoning-gym-answer-extraction

Conversation

@qiaochuz-nv

@qiaochuz-nv qiaochuz-nv commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

The reasoning_gym verifier now scores the answer the agent ended on. Before, two defects in
_extract_answer_from_response gave correct answers reward 0.0 or partial credit:

  • Without <answer> tags it scored the "".join of every assistant message. Multi-message agents (DeepAgents
    through nemo_fabric_agent) emit their planning narration as earlier assistant messages, so a wrong draft
    anywhere in the trajectory zeroed a correct final answer. (Gym's own
    environments/nemo_fabric_deepagents_reasoning_gym/data/example_rollouts.jsonl recorded 0.0 on 5/5 rows whose
    final answers equal gold, but those rows have a different shape: one <think>…</think>answer message with no
    delimiter, which no extractor can fix and the prompt change below does.)
  • The \boxed{} fallback used re.search(r"\\boxed\{([^}]+)\}"): first box, stops at the first }, LaTeX passed
    through. Twelve *_reasoning_gym environments tell the model it may answer in \boxed{...}, so
    \boxed{Olivia\ is\ a\ laggard,\ ...} scored 0.0 and \boxed{\text{...}} scored 0.65 for a correct answer.

nemo_fabric_deepagents_reasoning_gym was also the only agent-harness reasoning_gym environment whose prompt never
asked for a delimiter; its system prompt now asks for <answer> tags, as
reasoning_gym_langchain_deepagents_agent_model_server.yaml already does. Data rows are unchanged. Not changed here:
the model-only environments/reasoning_gym (verified: false, reasoning_gym_simple_agent) has no delimiter
instruction either and its committed rollouts show the same 0.0-on-correct rows; its rows are the published
training set, so a prompt change there is a separate decision.

Fixes NVBug 6900506.

Cause

resources_servers/reasoning_gym/app.py _extract_answer_from_response (unchanged since the server was added,
apart from #3728/#3887 flags):

  • full_text = "".join(assistant_responses) and return full_text.strip() when no delimiter is found: the
    knights_knaves scorer turns the whole text into a set of (name, role) pairs, so any wrong hypothesis mentioned
    anywhere makes the sets differ.
  • re.search(r"\\boxed\{([^}]+)\}", full_text): first match (the <answer> path keeps the last match), no brace
    balancing, no LaTeX normalization. mcqa (fix(mcqa): boxed answer extraction #1844, _extract_boxed_inner) and math_with_judge already take the
    last balanced box; this server never got that change.
  • environments/nemo_fabric_deepagents_reasoning_gym/config.yaml system prompt only said "return only the answer
    in the exact format requested"; every sibling environment appends "Put only your final answer inside
    <answer></answer> tags or \boxed{...}".

Fix

  • Join messages with "\n"; <answer> tags first (last match, unchanged); else the LAST balanced \boxed{...}
    with LaTeX text normalization (\ , \,, \; → space, except \, between digits, a thousands separator, which is
    dropped; \text{}/\textbf{}/\textit{}/\textrm{}/\mathrm{}/\mbox{} unwrapped at any nesting, surrounding
    $ stripped, a tilde between words → space; a common indent removed and each line right-stripped, with line
    breaks and runs of spaces kept); else the last non-empty assistant message instead of the joined trajectory.
  • Fabric DeepAgents environment: system prompt asks for the final answer inside <answer></answer> tags (config
    only; data/example.jsonl untouched).
  • Follow-up commit afa32682f (Autobot review finding): the fallback collected one entry per output_text content
    part, so a final message split over several parts was scored by its last part only; the extractor now builds one
    text per assistant message (parts joined, as before) and the fallback returns the whole last message. Test added.
  • Follow-up commit 30a440a38 (review, @cmunley1): the box normalization collapsed whitespace across lines, so a
    multi-line boxed grid such as rotate_matrix's 1 2\n3 4 became 1 2 3 4 and scored 0 under exact match, where the
    old regex had kept the newline (14 reasoning_gym tasks have newline-separated oracles: arc_agi, binary_matrix,
    futoshiki, kakurasu, manipulate_matrix, mini_sudoku, modulo_grid, n_queens, pool_matrix, rearc, rotate_matrix,
    sudoku, survo, tower_of_hanoi). That commit collapsed whitespace per line instead (superseded below) and dropped
    \, between digits (1\,000 → 1000). Tests: rotate_matrix gold through \boxed{rows}, an indented block and
    \ spaces (0.0 before, 1.0 after), plus the thin-space cases.
  • Follow-up commit 8c20586ed (review, @yaoyu-33): collapsing runs of spaces within a line still lowered correct
    boxed answers whose spacing is part of the oracle: futoshiki 1.0 → 0.9 (space-aligned cells and ∧/∨ markers),
    emoji_mystery 1.0 → 0.01 (a gold with a double space), codeio 1.0 → 0.0 ("L L" string outputs). The normalizer
    now only removes a common indent (textwrap.dedent) and right-strips each line; line breaks and runs of spaces
    stay. Tests: a real futoshiki gold and a double-space emoji_mystery gold must score the same boxed and tagged, plus
    dedent/right-strip asserts. Main merged in (8b5b62f13).
  • Follow-up commit 8ccdd592b (Autobot review via /review): with a \text{} or $ wrapper around an indented
    block (\boxed{\text{\n 1 2\n 3 4\n}}), the dedent ran before the unwrapping, and the wrapper itself sits at
    column 0, so only the first row lost its indent (1 2\n 3 4). The normalizer now strips the $ delimiters,
    unwraps the wrappers, then dedents once. Tests: the indented block inside \text{}, inside $…$, and one
    \text{} per row.
  • Not done here: a single untagged message with reasoning before the answer is still scored as a whole (there is
    no delimiter to use); that is what the prompt change fixes, and the test for it is a strict xfail that
    documents the limit rather than a last-line heuristic.

Alternatives rejected: a "last line of the last message" heuristic (would misread multi-line answers for other
reasoning_gym tasks); changing data/example.jsonl rows (the QA case pins the shipped prompt, and the config-level
prompt is what the langchain DeepAgents config already does).

Before the fix

Main's extractor (git show origin/main:resources_servers/reasoning_gym/app.py at 85193811b, swapped in) with the
current test file (the e2e controls below ran on main f70bf75c3):

.venv/bin/python -m pytest resources_servers/reasoning_gym/tests/test_app.py -q
FAILED ...::TestAnswerExtraction::test_multi_message_trajectory_scores_the_final_message
  AssertionError: assert 'Zoey is a sa...ey is a sage.' == 'Zoey is a fo...ey is a sage.'   # narration scored
FAILED ...::test_boxed_answers_score_like_tagged_answers[boxed_latex_spaces]
  assert 'Zoey\\ is\\ ...s\\ a\\ sage.' == 'Zoey is a fo...ey is a sage.'                   # reward 0.0
FAILED ...::test_boxed_answers_score_like_tagged_answers[boxed_text_wrapper]
  assert '\\text{Zoey ...ey is a sage.' == 'Zoey is a fo...ey is a sage.'                   # reward 0.65
FAILED ...::test_boxed_answers_score_like_tagged_answers[boxed_dollar_inside]
FAILED ...::test_boxed_answers_score_like_tagged_answers[boxed_draft_then_final_same_message]
  assert 'Zoey is a sa...ey is a fool.' == 'Zoey is a fo...ey is a sage.'                   # first box wins
FAILED ...::test_boxed_answers_score_like_tagged_answers[boxed_draft_message_then_final_message]
FAILED ...::test_boxed_grid_answers_keep_their_rows[boxed_rows_indented_block]
FAILED ...::test_boxed_grid_answers_keep_their_rows[boxed_rows_latex_spaces]
FAILED ...::test_last_balanced_box_wins_and_latex_text_is_normalized
9 failed, 11 passed, 1 xfailed in 2.41s

After the fix

Gym 8ccdd592b (this branch head, rebased onto main 69a514999), same command:

23 passed, 1 xfailed in 2.77s

Validation

  • QA testcase hermes_reasoning_gym (GPU lane, one H100, Qwen3-8B on vLLM) with the new boxed-LaTeX gold controls:
    before (main f70bf75c3) 1/2, FAIL: task 4 gold_boxed_latex_spaces control reward: expected 1.0, got 0.0
    (run 20261028T233732535756Z-d5caca8d); after (this branch) 2/2, 30/30 live verifier checks, mean/reward 1.0
    (run 20261028T234209814956Z-38ef76d3)
  • QA testcase nemo_fabric_deepagents_reasoning_gym (hosted model through the inference provider, README flow on
    the 5 example rows) with the new final_tagged control: before (main f70bf75c3) PASS with mean/reward 0.8,
    no row tagged, task 3 scored 0.0 for a correct answer preceded by reasoning (run 20261028T235401859689Z-3898fc04);
    after (this branch) PASS with mean/reward 1.0, 5/5 rows answered inside <answer> tags, 20/20 verifier controls
    (run 20261028T235316064076Z-bfbcfd92)
  • resources_servers/reasoning_gym unit tests: 23 passed, 1 xfailed (24 cases, 21 new; reasoning-gym 0.1.25 in the test venv)
  • Gold-in-delimiter sweep: every task in reasoning_gym.factory.DATASETS (105 scored; composite needs a config), 3
    entries each (seed 7), the gold wrapped in \boxed{} and in <answer> tags, scored through main's extractor and
    this branch's with each dataset's score_answer. Boxed: no task scores lower than on main; codeio 0.0 → 1.0 (2 of 3
    entries, JSON golds with braces) and fraction_simplification 0.01 → 1.0 (2 of 3, $\frac{}{}$ golds). Tagged:
    identical to main on every row. The intermediate per-line collapse (30a440a38) had lowered futoshiki (1.0 → 0.9,
    3 of 3) and emoji_mystery (1.0 → 0.01) boxed golds; the dedent form does not. Rerun for 8ccdd592b with six box
    shapes (plain, \text{} wrapper, indented block, the indented block inside \text{} and inside $…$, one
    \text{} per row; seed 42), previous head vs this head: no row scores lower; the seven grid tasks (binary_matrix,
    futoshiki, mini_sudoku, modulo_grid, n_queens, rotate_matrix, sudoku) go to 1.0 on the two wrapper + indented-block
    shapes, everything else unchanged.
  • pre-commit run --all-files: passed
  • Rollout evidence: environments/nemo_fabric_deepagents_reasoning_gym/data/example_rollouts.jsonl regenerated
    with the README command on this branch (9bcbe65d0): rewards 1.0, 1.0, 1.0, 1.0, 1.0 (was 0.0 on 5/5 correct
    answers), every final message inside <answer> tags. That gain comes from the prompt change, not the extractor:
    the old rows were single <think>…</think>answer messages and still score 0.0 under the new extractor, and the
    regenerated tagged rows score 1.0 under the old extractor too. The other reasoning_gym reference rollouts are
    untouched.

Compatibility

Trajectories whose last box, last balanced box or last message holds the answer now score it. <answer>-tagged
answers are unchanged, and a gold answer placed in \boxed{} never scores lower than on main (sweep above). One
trade-off: with no delimiter, an answer followed by a short trailing message (for example "Hope this helps!") now
scores 0 where main's joined text earned credit, because the last message alone is scored. Open issue #3035 (move
reasoning_gym verification off the event loop) requires extracted answers to match before and after its change;
this PR changes extracted answers on purpose, so whichever lands second must rebase its parity baseline on the
other. #5057 (nemo_fabric_agent on Fabric 0.5.0) does not touch these files; if it lands first, the regenerated
Fabric rollouts were produced by the current agent and may deserve another run.

🤖 Generated with Claude Code

@qiaochuz-nv qiaochuz-nv added bug Something isn't working QA_fix Fix implemented by a QA team member area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers needs-review PR is ready for code review and waiting on a reviewer complexity:low Localized change in one scope with a small, straightforward review surface labels Oct 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@qiaochuz-nv
qiaochuz-nv requested a review from yaoyu-33 October 29, 2026 17:53
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test edc3912

@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test 9b65642

qiaochuz-nv added a commit that referenced this pull request Oct 29, 2026
The no-delimiter fallback collected one entry per output_text content part, so a
final message split over several parts was scored by its last part only. The
extractor now builds one text per assistant message (its parts joined, as the
base implementation did) and the fallback returns the whole last message.
Found by the Autobot review on #5324.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
Comment thread resources_servers/reasoning_gym/app.py Resolved
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test 30a440a

qiaochuz-nv added a commit that referenced this pull request Oct 29, 2026
The no-delimiter fallback collected one entry per output_text content part, so a
final message split over several parts was scored by its last part only. The
extractor now builds one text per assistant message (its parts joined, as the
base implementation did) and the fallback returns the whole last message.
Found by the Autobot review on #5324.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
qiaochuz-nv added a commit that referenced this pull request Oct 29, 2026
… answers

The boxed-answer normalization collapsed all whitespace, so a multi-line
boxed grid such as rotate_matrix's "1 2\n3 4" became "1 2 3 4" and scored 0
under exact-match scoring, where the previous regex had kept the newline.
Fourteen reasoning_gym tasks (arc_agi, binary_matrix, pool_matrix,
rotate_matrix, sudoku, n_queens, ...) have newline-separated oracles.
Collapse whitespace within each line instead, and drop "\," between digits
("1\,000" -> "1000") rather than turning it into a space.

Found in review of #5324 by cmunley1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
@qiaochuz-nv
qiaochuz-nv force-pushed the qiaochuz/nvb-6900506-reasoning-gym-answer-extraction branch from 9b65642 to 30a440a Compare October 29, 2026 23:22
cmunley1
cmunley1 previously approved these changes Oct 29, 2026
@cmunley1
cmunley1 enabled auto-merge October 29, 2026 23:48
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test 3c32970

cmunley1
cmunley1 previously approved these changes Nov 7, 2026
@cmunley1

cmunley1 commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

/review

A box written as an indented block inside a \text{...} or $...$ wrapper
kept the indent of every row but the first: the dedent ran before the
unwrapping, and the wrapper itself sits at column 0, so there was no
common indent to remove. Strip the $ delimiters, unwrap the wrappers,
then dedent once. Tests cover the indented block inside \text{}, inside
$...$, and one \text{} per row. Registry-wide sweep (105 tasks x 3
entries, each task's own scorer, six box shapes): 0 regressions; the
seven grid tasks now score 1.0 for a gold boxed that way.

Found by the Autobot review on #5324.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test 8ccdd59

@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

Addressed the Autobot review finding (the check run from /review; it posted no thread). With a \text{} or $ wrapper around an indented block, as in \boxed{\text{\n 1 2\n 3 4\n}}, the dedent ran before the unwrapping, and the wrapper itself sits at column 0, so only the first row lost its indent: the extracted answer was 1 2\n 3 4.

Fixed in 8ccdd59: strip the $ delimiters, unwrap the \text{}-style wrappers, then dedent once. Three new grid cases in test_boxed_grid_answers_keep_their_rows: the indented block inside \text{}, inside $…$, and one \text{} per row (the last guards the case a plain reorder of dedent and unwrap would have broken). 23 passed / 1 xfailed.

Registry-wide sweep rerun (105 tasks × 3 entries, each task's own score_answer, six box shapes including the three above): 0 regressions against the previous head; the seven grid tasks (binary_matrix, futoshiki, mini_sudoku, modulo_grid, n_queens, rotate_matrix, sudoku) go to 1.0 on the two wrapper + indented-block shapes, everything else unchanged.

The artifact rows the bot could not read are the five Fabric example rollouts; their rewards and the before/after scoring are in the Validation section of the description.

@cmunley1
cmunley1 added this pull request to the merge queue Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43062224913

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

@cmunley1 The merge-queue run (43062224913) dropped this PR on two failures, neither of which is in this PR's code (it touches only resources_servers/reasoning_gym/):

  • Core tests (shard 1) (job): tests/unit_tests/agents/test_process_supervisor.py::test_foreground_tool_cannot_mutate_task_after_confirmed_receipt[cancel-False-True]. The supervised process ignores SIGTERM by design, was SIGKILLed after the 0.3 s cleanup budget (process_supervisor.sh: line 288: ... Killed), and the supervisor exited 1 where the test expects 0. A sibling test in the same file (test_forced_stop_reaches_processes_that_leave_the_harness_during_the_stop[term_handler]) failed the same way on a shared runner on Nov 7 and passed on rerun.
  • Server suite (shard 2) (job): the "Install system test dependencies" step failed in apt, E: Failed to fetch https://ppa.launchpadcontent.net/git-core/ppa/... Some files failed to download. Same git-core PPA outage that timed out the install step on fix(data): collate validates task_data from any cwd; a data-only dir no longer shadows the server dir #5325 on Nov 10.

The PR's own CI on 8ccdd592b is green and the PR is still approved and mergeable. Re-queue whenever convenient.

@qiaochuz-nv
qiaochuz-nv added this pull request to the merge queue Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43476459704

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

Merge queue run 43476459704 failed in Unit tests / Server suite (shard 2), caused by a flaky test unrelated to this PR: responses_api_agents/stirrup_agent/tests/test_sessions.py::test_an_agent_that_runs_out_of_wall_time_answers_with_its_partial_run_as_incomplete (assert 0 == 1 on function_call count, line 231). It sets a 4s sandbox_runner_timeout_seconds, and on a loaded runner the cutoff fired before the first tool call was recorded. This PR only touches reasoning_gym and its Fabric DeepAgents environment config. All PR-branch checks pass and the other 7 shards were green, so re-queuing.

@qiaochuz-nv
qiaochuz-nv added this pull request to the merge queue Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43480603073

Merged via the queue into main with commit 84baba4 Nov 13, 2026
105 of 110 checks passed
@qiaochuz-nv
qiaochuz-nv deleted the qiaochuz/nvb-6900506-reasoning-gym-answer-extraction branch November 13, 2026 18:38

This branch was previously deployed

1 inactive deployment
public — 8ccdd592 Deployed Nov 7, 2026 by copy-pr-bot[bot] via release / finalize / notify #10863
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer QA_fix Fix implemented by a QA team member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants