You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Producers do not declare the outcome of the failures they emit, so each one re-decides retry vs mask vs score #6344
Child of #2750. Related: #4883 (audit of reward-zero and fake-success producers), #4882 (metadata on already-masked results).
The gap
#2750 puts the decision in the right place: "the framework reports policy-neutral facts; callers decide whether to retry, replace, mask, omit, or stop." nemo_gym/core/failure_kinds.py states the same rule and explains why keeping outcome off the name is deliberate.
What is missing is the other half: nothing makes a producer declare the outcome it chose, and nothing tests it. So each benchmark, agent and environment server re-decides retry-vs-mask-vs-score privately, and nothing notices when it decides wrong.
Why this needs its own issue
The decisions drift in both directions at once, which is what makes it a contract problem rather than a list of bugs:
Since 25 Oct, 34 of 665 commits on main are fixes of this shape, each teaching one more producer what its own failures mean. That rate is the symptom: the knowledge is not written down anywhere a new producer has to meet it.
Proposed shape
A producer declares, per failure kind it can emit, which outcome applies (retry, mask, score, omit) and the attribution (policy or evaluation).
A contract test at onboarding asserts the declaration against what the producer actually writes for each kind — a test a new benchmark cannot skip.
Recovery reads declarations rather than inferring from the kind, so an undeclared kind is a loud failure instead of a silent mask.
Point 3 is the part that converts the whole class from "found in a run weeks later" to "found in CI". Happy to be told the shape is wrong — the gap is the thing I am confident about.
Out of scope
Reclassifying existing producers. #4883 owns that audit, and the per-producer fixes above are already landing.
Activity