Repository navigation
Description
Describe the bug
The instruction_following resources server scores count:count_unique ("Only use unique words in your response, no word should be repeated!") with the upstream checker from verifiable-instructions. Its check_following compares nltk.word_tokenize(value) with its set, and punctuation marks are tokens, so a response with two sentences fails on the second . and a response with two commas fails on the second ,. A reply passes only if no punctuation mark appears twice in it (Cats purr. Really? Yes! passes, Cats purr. Dogs bark. does not), however well it follows the instruction.
PR #4624 fixed the sibling instruction count:counting_composition, which had the same punctuation problem, with a local checker and notes that count:count_unique can follow separately. This issue tracks that follow-up.
Steps/Code to reproduce bug
Pure Python, no model or GPU needed (verifiable-instructions at its unpinned head f46a5ac, which resources_servers/instruction_following/requirements.txt installs; nltk 3.10.4):
import nltk
nltk.download("punkt_tab", quiet=True) # word_tokenize needs it; the server fetches it at startup
from verifiable_instructions import instructions_registry
checker = instructions_registry.INSTRUCTION_DICT["count:count_unique"]("count:count_unique")
checker.build_description()
checker.check_following("Cats purr. Dogs bark.") # False: tokens ['Cats', 'purr', '.', 'Dogs', 'bark', '.']
checker.check_following("Cats purr, dogs bark, birds sing.") # False: the second ','
checker.check_following("Hello world.") # True
checker.check_following("Cats purr. Cats bark.") # False (correct: 'Cats' repeats)Expected behavior
Punctuation-only tokens are not words. "Cats purr. Dogs bark." follows the instruction and should pass; "Cats purr. Cats bark." should still fail.
Configs
Default resources_servers/instruction_following/configs/instruction_following.yaml, dataset nvidia/Nemotron-RL-instruction_following. No config changes are involved; the behaviour is in the checker.
Environment details
- NeMo Gym
mainata4f263e32 - verifiable-instructions
f46a5ac(the headrequirements.txtresolves to), nltk 3.10.4, Python 3.12 - Reproduced on macOS; nothing in the repro is platform-specific
Additional context
Scope: in nvidia/Nemotron-RL-instruction_following (46,391 rows), 2,223 rows (4.8%) carry this instruction: 34 on their own and 2,189 together with one or two other instructions. None of them sets grading_mode, so they grade binary and a compliant answer scores 0 on all 2,223; with grading_mode: fraction the reward is capped at (n-1)/n instead.
Suggested fix, same shape as #4624: add a CountUniqueChecker subclass to the override map in app.py that keeps only tokens containing an alphanumeric character before comparing, as allenai/open-instruct#1718 did for both checkers (case and contraction splitting unchanged), plus tests and a README line. Local override rather than an upstream fix for the reason #4624 gave: verifiable-instructions has had no commits since 2025-08-20 and its only PR (abukharin-nv/verifiable-instructions#1, January) is unanswered. Rewards on these 2,223 rows shift once compliant answers can pass; other instructions and benchmarks/ifeval (no count: rows) are unchanged. Fix: PR #5068.
Related: #4613 / #4624 (sibling checker), #215 (a different error in count:count_increment_word when a keyword is a list; not addressed here).
Activity
[-]`count:count_unique` rejects every multi-sentence response because punctuation tokens count as repeated words[/-][+]`count:count_unique` fails almost every multi-sentence response because punctuation tokens count as repeated words[/+]marcromeyn commentedon Oct 26, 2026
Thanks for the report. Confirmed on current
main(6a76e717) with verifiable-instructionsf46a5ac:Cats purr. Dogs bark.Cats purr, dogs bark, birds sing.Hello world.Cats purr. Cats bark.Same root cause as
count:counting_compositionin #4624, so the fix can follow that PR: a local checker inresources_servers/instruction_following/that keeps the upstream instruction text and only overridescheck_following, ignoring tokens that contain no letters or digits, registered next tocount:counting_compositioninINSTRUCTION_DICT. Tests should cover multi-sentence and comma-separated responses that pass, a genuine repeat that still fails, and a check that only this checker is replaced. The PR should also note that rewards for rows with this constraint change and how many dataset rows are affected.Would you like to send the PR? I'm happy to review it. If not, I'll pick it up.
fix(instruction_following): stop count:count_unique from counting pun…