Repository navigation
fix(reasoning_gym): score the answer the agent ended on; ask Fabric DeepAgents for <answer> tags - #5324
Conversation
|
/ok to test edc3912 |
|
/ok to test 9b65642 |
The no-delimiter fallback collected one entry per output_text content part, so a final message split over several parts was scored by its last part only. The extractor now builds one text per assistant message (its parts joined, as the base implementation did) and the fallback returns the whole last message. Found by the Autobot review on #5324. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
|
/ok to test 30a440a |
The no-delimiter fallback collected one entry per output_text content part, so a final message split over several parts was scored by its last part only. The extractor now builds one text per assistant message (its parts joined, as the base implementation did) and the fallback returns the whole last message. Found by the Autobot review on #5324. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
… answers
The boxed-answer normalization collapsed all whitespace, so a multi-line
boxed grid such as rotate_matrix's "1 2\n3 4" became "1 2 3 4" and scored 0
under exact-match scoring, where the previous regex had kept the newline.
Fourteen reasoning_gym tasks (arc_agi, binary_matrix, pool_matrix,
rotate_matrix, sudoku, n_queens, ...) have newline-separated oracles.
Collapse whitespace within each line instead, and drop "\," between digits
("1\,000" -> "1000") rather than turning it into a space.
Found in review of #5324 by cmunley1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
9b65642 to
30a440a
Compare
|
/ok to test 3c32970 |
|
/review |
A box written as an indented block inside a \text{...} or $...$ wrapper
kept the indent of every row but the first: the dedent ran before the
unwrapping, and the wrapper itself sits at column 0, so there was no
common indent to remove. Strip the $ delimiters, unwrap the wrappers,
then dedent once. Tests cover the indented block inside \text{}, inside
$...$, and one \text{} per row. Registry-wide sweep (105 tasks x 3
entries, each task's own scorer, six box shapes): 0 regressions; the
seven grid tasks now score 1.0 for a gold boxed that way.
Found by the Autobot review on #5324.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
|
/ok to test 8ccdd59 |
|
Addressed the Autobot review finding (the check run from Fixed in 8ccdd59: strip the Registry-wide sweep rerun (105 tasks × 3 entries, each task's own The artifact rows the bot could not read are the five Fabric example rollouts; their rewards and the before/after scoring are in the Validation section of the description. |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43062224913 |
|
@cmunley1 The merge-queue run (43062224913) dropped this PR on two failures, neither of which is in this PR's code (it touches only
The PR's own CI on |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43476459704 |
|
Merge queue run 43476459704 failed in |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43480603073 |
What does this PR do?
The reasoning_gym verifier now scores the answer the agent ended on. Before, two defects in
_extract_answer_from_responsegave correct answers reward 0.0 or partial credit:<answer>tags it scored the"".joinof every assistant message. Multi-message agents (DeepAgentsthrough
nemo_fabric_agent) emit their planning narration as earlier assistant messages, so a wrong draftanywhere in the trajectory zeroed a correct final answer. (Gym's own
environments/nemo_fabric_deepagents_reasoning_gym/data/example_rollouts.jsonlrecorded 0.0 on 5/5 rows whosefinal answers equal gold, but those rows have a different shape: one
<think>…</think>answermessage with nodelimiter, which no extractor can fix and the prompt change below does.)
\boxed{}fallback usedre.search(r"\\boxed\{([^}]+)\}"): first box, stops at the first}, LaTeX passedthrough. Twelve
*_reasoning_gymenvironments tell the model it may answer in\boxed{...}, so\boxed{Olivia\ is\ a\ laggard,\ ...}scored 0.0 and\boxed{\text{...}}scored 0.65 for a correct answer.nemo_fabric_deepagents_reasoning_gymwas also the only agent-harness reasoning_gym environment whose prompt neverasked for a delimiter; its system prompt now asks for
<answer>tags, asreasoning_gym_langchain_deepagents_agent_model_server.yamlalready does. Data rows are unchanged. Not changed here:the model-only
environments/reasoning_gym(verified: false,reasoning_gym_simple_agent) has no delimiterinstruction either and its committed rollouts show the same 0.0-on-correct rows; its rows are the published
training set, so a prompt change there is a separate decision.
Fixes NVBug 6900506.
Cause
resources_servers/reasoning_gym/app.py_extract_answer_from_response(unchanged since the server was added,apart from #3728/#3887 flags):
full_text = "".join(assistant_responses)andreturn full_text.strip()when no delimiter is found: theknights_knaves scorer turns the whole text into a set of (name, role) pairs, so any wrong hypothesis mentioned
anywhere makes the sets differ.
re.search(r"\\boxed\{([^}]+)\}", full_text): first match (the<answer>path keeps the last match), no bracebalancing, no LaTeX normalization.
mcqa(fix(mcqa): boxed answer extraction #1844,_extract_boxed_inner) andmath_with_judgealready take thelast balanced box; this server never got that change.
environments/nemo_fabric_deepagents_reasoning_gym/config.yamlsystem prompt only said "return only the answerin the exact format requested"; every sibling environment appends "Put only your final answer inside
<answer></answer>tags or\boxed{...}".Fix
"\n";<answer>tags first (last match, unchanged); else the LAST balanced\boxed{...}with LaTeX text normalization (
\,\,,\;→ space, except\,between digits, a thousands separator, which isdropped;
\text{}/\textbf{}/\textit{}/\textrm{}/\mathrm{}/\mbox{}unwrapped at any nesting, surrounding$stripped, a tilde between words → space; a common indent removed and each line right-stripped, with linebreaks and runs of spaces kept); else the last non-empty assistant message instead of the joined trajectory.
<answer></answer>tags (configonly;
data/example.jsonluntouched).afa32682f(Autobot review finding): the fallback collected one entry peroutput_textcontentpart, so a final message split over several parts was scored by its last part only; the extractor now builds one
text per assistant message (parts joined, as before) and the fallback returns the whole last message. Test added.
30a440a38(review, @cmunley1): the box normalization collapsed whitespace across lines, so amulti-line boxed grid such as rotate_matrix's
1 2\n3 4became1 2 3 4and scored 0 under exact match, where theold regex had kept the newline (14 reasoning_gym tasks have newline-separated oracles: arc_agi, binary_matrix,
futoshiki, kakurasu, manipulate_matrix, mini_sudoku, modulo_grid, n_queens, pool_matrix, rearc, rotate_matrix,
sudoku, survo, tower_of_hanoi). That commit collapsed whitespace per line instead (superseded below) and dropped
\,between digits (1\,000→1000). Tests: rotate_matrix gold through\boxed{rows}, an indented block and\spaces (0.0 before, 1.0 after), plus the thin-space cases.8c20586ed(review, @yaoyu-33): collapsing runs of spaces within a line still lowered correctboxed answers whose spacing is part of the oracle: futoshiki 1.0 → 0.9 (space-aligned cells and
∧/∨markers),emoji_mystery 1.0 → 0.01 (a gold with a double space), codeio 1.0 → 0.0 (
"L L"string outputs). The normalizernow only removes a common indent (
textwrap.dedent) and right-strips each line; line breaks and runs of spacesstay. Tests: a real futoshiki gold and a double-space emoji_mystery gold must score the same boxed and tagged, plus
dedent/right-strip asserts. Main merged in (
8b5b62f13).8ccdd592b(Autobot review via/review): with a\text{}or$wrapper around an indentedblock (
\boxed{\text{\n 1 2\n 3 4\n}}), the dedent ran before the unwrapping, and the wrapper itself sits atcolumn 0, so only the first row lost its indent (
1 2\n 3 4). The normalizer now strips the$delimiters,unwraps the wrappers, then dedents once. Tests: the indented block inside
\text{}, inside$…$, and one\text{}per row.no delimiter to use); that is what the prompt change fixes, and the test for it is a strict
xfailthatdocuments the limit rather than a last-line heuristic.
Alternatives rejected: a "last line of the last message" heuristic (would misread multi-line answers for other
reasoning_gym tasks); changing
data/example.jsonlrows (the QA case pins the shipped prompt, and the config-levelprompt is what the langchain DeepAgents config already does).
Before the fix
Main's extractor (
git show origin/main:resources_servers/reasoning_gym/app.pyat85193811b, swapped in) with thecurrent test file (the e2e controls below ran on main
f70bf75c3):After the fix
Gym
8ccdd592b(this branch head, rebased onto main69a514999), same command:Validation
hermes_reasoning_gym(GPU lane, one H100, Qwen3-8B on vLLM) with the new boxed-LaTeX gold controls:before (main
f70bf75c3) 1/2,FAIL: task 4 gold_boxed_latex_spaces control reward: expected 1.0, got 0.0(run
20261028T233732535756Z-d5caca8d); after (this branch) 2/2, 30/30 live verifier checks,mean/reward 1.0(run
20261028T234209814956Z-38ef76d3)nemo_fabric_deepagents_reasoning_gym(hosted model through the inference provider, README flow onthe 5 example rows) with the new
final_taggedcontrol: before (mainf70bf75c3) PASS withmean/reward 0.8,no row tagged, task 3 scored 0.0 for a correct answer preceded by reasoning (run
20261028T235401859689Z-3898fc04);after (this branch) PASS with
mean/reward 1.0, 5/5 rows answered inside<answer>tags, 20/20 verifier controls(run
20261028T235316064076Z-bfbcfd92)resources_servers/reasoning_gymunit tests: 23 passed, 1 xfailed (24 cases, 21 new; reasoning-gym 0.1.25 in the test venv)reasoning_gym.factory.DATASETS(105 scored;compositeneeds a config), 3entries each (seed 7), the gold wrapped in
\boxed{}and in<answer>tags, scored through main's extractor andthis branch's with each dataset's
score_answer. Boxed: no task scores lower than on main; codeio 0.0 → 1.0 (2 of 3entries, JSON golds with braces) and fraction_simplification 0.01 → 1.0 (2 of 3,
$\frac{}{}$golds). Tagged:identical to main on every row. The intermediate per-line collapse (
30a440a38) had lowered futoshiki (1.0 → 0.9,3 of 3) and emoji_mystery (1.0 → 0.01) boxed golds; the dedent form does not. Rerun for
8ccdd592bwith six boxshapes (plain,
\text{}wrapper, indented block, the indented block inside\text{}and inside$…$, one\text{}per row; seed 42), previous head vs this head: no row scores lower; the seven grid tasks (binary_matrix,futoshiki, mini_sudoku, modulo_grid, n_queens, rotate_matrix, sudoku) go to 1.0 on the two wrapper + indented-block
shapes, everything else unchanged.
pre-commit run --all-files: passedenvironments/nemo_fabric_deepagents_reasoning_gym/data/example_rollouts.jsonlregeneratedwith the README command on this branch (
9bcbe65d0): rewards1.0, 1.0, 1.0, 1.0, 1.0(was0.0on 5/5 correctanswers), every final message inside
<answer>tags. That gain comes from the prompt change, not the extractor:the old rows were single
<think>…</think>answermessages and still score 0.0 under the new extractor, and theregenerated tagged rows score 1.0 under the old extractor too. The other reasoning_gym reference rollouts are
untouched.
Compatibility
Trajectories whose last box, last balanced box or last message holds the answer now score it.
<answer>-taggedanswers are unchanged, and a gold answer placed in
\boxed{}never scores lower than on main (sweep above). Onetrade-off: with no delimiter, an answer followed by a short trailing message (for example "Hope this helps!") now
scores 0 where main's joined text earned credit, because the last message alone is scored. Open issue #3035 (move
reasoning_gym verification off the event loop) requires extracted answers to match before and after its change;
this PR changes extracted answers on purpose, so whichever lands second must rebase its parity baseline on the
other. #5057 (nemo_fabric_agent on Fabric 0.5.0) does not touch these files; if it lands first, the regenerated
Fabric rollouts were produced by the current agent and may deserve another run.
🤖 Generated with Claude Code