Skip to content

Producers do not declare the outcome of the failures they emit, so each one re-decides retry vs mask vs score #6344

Open

Description

@wprazuch

Child of #2750. Related: #4883 (audit of reward-zero and fake-success producers), #4882 (metadata on already-masked results).

The gap

#2750 puts the decision in the right place: "the framework reports policy-neutral facts; callers decide whether to retry, replace, mask, omit, or stop." nemo_gym/core/failure_kinds.py states the same rule and explains why keeping outcome off the name is deliberate.

What is missing is the other half: nothing makes a producer declare the outcome it chose, and nothing tests it. So each benchmark, agent and environment server re-decides retry-vs-mask-vs-score privately, and nothing notices when it decides wrong.

Why this needs its own issue

The decisions drift in both directions at once, which is what makes it a contract problem rather than a list of bugs:

Since 25 Oct, 34 of 665 commits on main are fixes of this shape, each teaching one more producer what its own failures mean. That rate is the symptom: the knowledge is not written down anywhere a new producer has to meet it.

Proposed shape

  1. A producer declares, per failure kind it can emit, which outcome applies (retry, mask, score, omit) and the attribution (policy or evaluation).
  2. A contract test at onboarding asserts the declaration against what the producer actually writes for each kind — a test a new benchmark cannot skip.
  3. Recovery reads declarations rather than inferring from the kind, so an undeclared kind is a loud failure instead of a silent mask.

Point 3 is the part that converts the whole class from "found in a run weeks later" to "found in CI". Happy to be told the shape is wrong — the gap is the thing I am confident about.

Out of scope

Reclassifying existing producers. #4883 owns that audit, and the per-producer fixes above are already landing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions