Skip to content

count:count_unique fails almost every multi-sentence response because punctuation tokens count as repeated words #5067

Closed

Description

@sven-rankov

Describe the bug

The instruction_following resources server scores count:count_unique ("Only use unique words in your response, no word should be repeated!") with the upstream checker from verifiable-instructions. Its check_following compares nltk.word_tokenize(value) with its set, and punctuation marks are tokens, so a response with two sentences fails on the second . and a response with two commas fails on the second ,. A reply passes only if no punctuation mark appears twice in it (Cats purr. Really? Yes! passes, Cats purr. Dogs bark. does not), however well it follows the instruction.

PR #4624 fixed the sibling instruction count:counting_composition, which had the same punctuation problem, with a local checker and notes that count:count_unique can follow separately. This issue tracks that follow-up.

Steps/Code to reproduce bug

Pure Python, no model or GPU needed (verifiable-instructions at its unpinned head f46a5ac, which resources_servers/instruction_following/requirements.txt installs; nltk 3.10.4):

import nltk
nltk.download("punkt_tab", quiet=True)  # word_tokenize needs it; the server fetches it at startup
from verifiable_instructions import instructions_registry

checker = instructions_registry.INSTRUCTION_DICT["count:count_unique"]("count:count_unique")
checker.build_description()
checker.check_following("Cats purr. Dogs bark.")               # False: tokens ['Cats', 'purr', '.', 'Dogs', 'bark', '.']
checker.check_following("Cats purr, dogs bark, birds sing.")   # False: the second ','
checker.check_following("Hello world.")                        # True
checker.check_following("Cats purr. Cats bark.")               # False (correct: 'Cats' repeats)

Expected behavior

Punctuation-only tokens are not words. "Cats purr. Dogs bark." follows the instruction and should pass; "Cats purr. Cats bark." should still fail.

Configs

Default resources_servers/instruction_following/configs/instruction_following.yaml, dataset nvidia/Nemotron-RL-instruction_following. No config changes are involved; the behaviour is in the checker.

Environment details

  • NeMo Gym main at a4f263e32
  • verifiable-instructions f46a5ac (the head requirements.txt resolves to), nltk 3.10.4, Python 3.12
  • Reproduced on macOS; nothing in the repro is platform-specific

Additional context

Scope: in nvidia/Nemotron-RL-instruction_following (46,391 rows), 2,223 rows (4.8%) carry this instruction: 34 on their own and 2,189 together with one or two other instructions. None of them sets grading_mode, so they grade binary and a compliant answer scores 0 on all 2,223; with grading_mode: fraction the reward is capped at (n-1)/n instead.

Suggested fix, same shape as #4624: add a CountUniqueChecker subclass to the override map in app.py that keeps only tokens containing an alphanumeric character before comparing, as allenai/open-instruct#1718 did for both checkers (case and contraction splitting unchanged), plus tests and a README line. Local override rather than an upstream fix for the reason #4624 gave: verifiable-instructions has had no commits since 2025-08-20 and its only PR (abukharin-nv/verifiable-instructions#1, January) is unanswered. Rewards on these 2,223 rows shift once compliant answers can pass; other instructions and benchmarks/ifeval (no count: rows) are unchanged. Fix: PR #5068.

Related: #4613 / #4624 (sibling checker), #215 (a different error in count:count_increment_word when a keyword is a list; not addressed here).

@cmunley1 @Thanatos9404, cc @abukharin-nv

Activity

  1. changed the title [-]`count:count_unique` rejects every multi-sentence response because punctuation tokens count as repeated words[/-] [+]`count:count_unique` fails almost every multi-sentence response because punctuation tokens count as repeated words[/+] on Oct 26, 2026
  2. marcromeyn commented on Oct 26, 2026

    @marcromeyn
    Contributor

    Thanks for the report. Confirmed on current main (6a76e717) with verifiable-instructions f46a5ac:

    Response Result Expected
    Cats purr. Dogs bark. False pass
    Cats purr, dogs bark, birds sing. False pass
    Hello world. True pass
    Cats purr. Cats bark. False fail ("Cats" repeats)

    Same root cause as count:counting_composition in #4624, so the fix can follow that PR: a local checker in resources_servers/instruction_following/ that keeps the upstream instruction text and only overrides check_following, ignoring tokens that contain no letters or digits, registered next to count:counting_composition in INSTRUCTION_DICT. Tests should cover multi-sentence and comma-separated responses that pass, a genuine repeat that still fails, and a check that only this checker is replaced. The PR should also note that rewards for rows with this constraint change and how many dataset rows are affected.

    Would you like to send the PR? I'm happy to review it. If not, I'll pick it up.

  3. added a commit that references this issue on Oct 27, 2026
    ecc223f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    community-requestIssue reported or requested by someone from the community

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions