Repository navigation
Reasoning reward #4563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Reasoning reward #4563
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
f14cee5
Add reasoning accuracy reward
lewtun 9737706
Simplify
lewtun 9329af2
f
lewtun f732d52
Fix
lewtun efc68b9
fix col name
lewtun 4c440b3
Fix
lewtun 456e80d
Clean tests
lewtun f3701c2
Add reasoning reward to registries and docs
lewtun f1da297
Merge branch 'main' into reasoning-reward
lewtun d134d22
Fix
lewtun a0dd6eb
nits
qgallouedec 79547af
Merge branch 'main' into reasoning-reward
lewtun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,7 +12,7 @@ | |
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| from trl.rewards import accuracy_reward, get_soft_overlong_punishment, think_format_reward | ||
| from trl.rewards import accuracy_reward, get_soft_overlong_punishment, reasoning_accuracy_reward, think_format_reward | ||
|
|
||
| from .testing_utils import TrlTestCase, require_math_latex | ||
|
|
||
|
|
@@ -117,27 +117,76 @@ def test_accuracy_reward_unparseable_gold(self): | |
| """Test accuracy_reward with an unparseable gold solution.""" | ||
| completion = [ | ||
| [{"content": "Answer is forty two."}], | ||
| [{"content": "Some other content."}], | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed these cases as they weren't testing what was advertised :) |
||
| [{"content": r"Answer is \boxed{42}."}], | ||
| [{"content": r"Answer is \boxed{\mathbf{42}}."}], # Make response bold | ||
| [{"content": r"Answer is \boxed{\textbf{42}}."}], # Different latex command for bold | ||
| [{"content": r"Answer is \boxed{42}."}], | ||
| [{"content": r"Answer is \boxed{42.3456}."}], | ||
| [{"content": r"Some other content. \boxed{43}."}], | ||
| ] | ||
| solution = [ | ||
| "Answer is forty two.", | ||
| "Answer is forty three.", | ||
| "Answer is 42.0", # Decimal point | ||
| "Answer is 42 43 okay?", # Extra space | ||
| "Answer is 42", | ||
| r"Answer is \n\boxed{42}", # Newline in gold solution | ||
| "Answer is 42.34560", # Extra trailing zero | ||
| ] | ||
| rewards = accuracy_reward(completion, solution) | ||
| assert rewards[0] == 1.0 # Should revert to exact text match | ||
| assert rewards[0] is None | ||
| assert rewards[1] is None | ||
|
|
||
|
|
||
| class TestReasoningAccuracyReward: | ||
| @require_math_latex | ||
| def test_correct_answer_yields_unit_reward(self): | ||
| completions = [ | ||
| [{"content": r"<think> Reasoning content </think> \boxed{\frac{63}{400}}"}], | ||
| [{"content": r"Reasoning content </think> \boxed{\frac{63}{400}}"}], | ||
| ] | ||
| solutions = [r"\frac{63}{400}", r"\frac{63}{400}"] | ||
| rewards = reasoning_accuracy_reward(completions, solutions) | ||
| assert rewards[0] == 1.0 | ||
| assert rewards[1] == 1.0 | ||
|
|
||
| @require_math_latex | ||
| def test_correct_answer_with_custom_tags_yields_unit_reward(self): | ||
| completions = [ | ||
| [{"content": r"<REASONING_START> Reasoning content </REASONING_END> \boxed{\frac{63}{400}}"}], | ||
| ] | ||
| solutions = [ | ||
| r"\frac{63}{400}", | ||
| ] | ||
| rewards = reasoning_accuracy_reward(completions, solutions, reasoning_delimiters=["</REASONING_END>"]) | ||
| assert rewards[0] == 1.0 | ||
|
|
||
| @require_math_latex | ||
| def test_incorrect_answer_yields_zero_reward(self): | ||
| completion = [[{"content": r"<think> Reasoning content </think> \boxed{\frac{64}{400}}"}]] | ||
| solution = [r"\frac{63}{400}"] | ||
| rewards = reasoning_accuracy_reward(completion, solution) | ||
| assert rewards[0] == 0.0 | ||
|
|
||
| @require_math_latex | ||
| def test_correct_answer_in_reasoning_yields_zero_reward(self): | ||
| completions = [ | ||
| [{"content": r"<think> My answer is \boxed{42} </think> Some other text."}], | ||
| [{"content": r"<think> The answer is \boxed{42} </think> Here's a wrong answer: \boxed{43}."}], | ||
| ] | ||
| solutions = [r"\boxed{42}", r"\boxed{42}"] | ||
| rewards = reasoning_accuracy_reward(completions, solutions) | ||
| assert rewards[0] == 0.0 | ||
| assert rewards[1] == 0.0 | ||
|
|
||
| @require_math_latex | ||
| def test_incomplete_reasoning_yields_zero_reward(self): | ||
| completions = [ | ||
| [{"content": r"<think> Incomplete reasoning without closing tag"}], | ||
| [{"content": r"Correct answer \frac{63}{400} but completely missing reasoning content"}], | ||
| ] | ||
| solutions = [r"\frac{63}{400}", r"\frac{63}{400}"] | ||
| rewards = reasoning_accuracy_reward(completions, solutions) | ||
| assert rewards[0] == 0.0 | ||
| assert rewards[1] == 0.0 | ||
| assert rewards[2] == 1.0 | ||
| assert rewards[3] == 1.0 | ||
| assert rewards[4] == 1.0 | ||
| assert rewards[5] == 1.0 | ||
| assert rewards[6] == 1.0 # Should ignore trailing zeros | ||
|
|
||
| @require_math_latex | ||
| def test_unparseable_gold_solution_yields_none_reward(self): | ||
| completions = [ | ||
| [{"content": r"<think> Reasoning content </think> \boxed{42}"}], | ||
| ] | ||
| solutions = [ | ||
| "forty two", | ||
| ] | ||
| rewards = reasoning_accuracy_reward(completions, solutions) | ||
| assert rewards[0] is None | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For viz @burtenshaw a close variant of this reward function is now native in TRL as
reasoning_accuracy_reward()