Repository navigation
fix(envs): declare retry for environment-side faults they mask - #6072
Conversation
mimo_music scorer timeouts and errors, agentic_turn_simulation simulator failures and wall-clock time_cap, conversational_tool_use_simulation simulator call failures and vcqa_agent artifact materialization failures are faults of the evaluation side, not of the policy. They were masked with a namespaced failure kind and no outcome action, so recovery never reran them and the sample silently left the denominator. Each now declares failure_outcome_action=retry with evaluation attribution. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-6072-a2cc00144e2c.docs.buildwithfern.com/nemo/gym |
…fits them mimo_music: only a fault of the machine (timeout, killed or unstartable scorer, failed renderer process) is a failure to grade this time. It keeps the model's answer, so it is masked as judge_failed and recovery grades the saved answer again. A missing renderer, an unparseable or out-of-range scorer result or a scorer exception fails every rollout the same way: it is now mimo_music:scorer_invalid with failure_outcome_action=fail. agentic_turn_simulation (time_cap, simulator_failed) and conversational_tool_use_simulation (simulator_call_failed) fault inside the episode, whose answer is unusable. Their retry declaration now records the agent stage, so the answer is not kept and recovery reruns the rollout; with the verification stage implied, a retry declaration alone left them masked without a rerun. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…e PR Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
giuliolovisotto
left a comment
There was a problem hiding this comment.
The new failure decisions need to be configurable before they are persisted as applied policy; one finding inline.
| if retry: | ||
| masked.update( | ||
| failure_outcome_action=OutcomeAction.RETRY, | ||
| failure_outcome_attribution=FailureClass.EVALUATION, |
There was a problem hiding this comment.
[P2] Apply configurable policy before saving these outcome decisions
These fields are consumed as an already-applied decision, but this path never consults the shared classifier or a configured outcome policy. For example, I reproduced a time-cap row with classification.by_stage.agent.by_failure_kind["agentic_turn_simulation:time_cap"] = "model" and recovery rules by_failure_class = {"model": "fail", "evaluation": "retry"}: the shared classifier returns model, but the row hardcodes evaluation, so _recovery_action() still selects retry. Consumers correctly preserve the saved attribution, and none of these producer configs provides a way to select a different decision before it is saved. The same hardcoded pair is added in conversational_tool_use_simulation and vcqa_agent.
Please keep the proposed defaults, but obtain the new action and attribution through the shared classification/outcome-policy mechanism with boundary configuration overrides, then persist the selected decision. Preserve the observed failure kind and originating stage independently of that action. Add a regression showing that the same fault can select retry or fail through configuration and that changing the action alone preserves attribution. This can stay scoped to the newly added decisions.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
Fixed for all three producers. Each now has an outcome_policy that classifies the new kinds as evaluation and defaults the agent stage to retry, so the defaults you accepted are unchanged; the action and attribution come from the shared classifier and stage policy and are saved with the stage agent, while the observed failure_kind is left as it was. A configuration that would select count_as_zero for one of these kinds is refused at load, since these rows are masked and cannot be counted zeros. Tests run the same fault with retry and with fail, as evaluation and as model, and your time-cap configuration (model with fail, evaluation with retry) is one of them. A small separate commit makes StageOutcomePolicyConfig.overlay keep a boundary's default classification under a partial override.
…olicy override StageOutcomePolicyConfig.overlay merged only the stage policies over a boundary's defaults, so an override that touched the classification dropped the boundary's own classification rules. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
The retry decisions this PR saves for a time_cap, a failed simulator and an unavailable task artifact no longer hardcode retry and evaluation. Each boundary has an outcome_policy whose defaults classify those kinds as the evaluation's and set the agent stage default to retry; the action and attribution come from the shared classifier and stage policy, so a deployment can reclassify a kind or select fail as the default. The observed failure kind and the agent stage are saved independently of the action. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…leave the row ungraded Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
oyilmaz-nvidia
left a comment
There was a problem hiding this comment.
Review notes. Items 1-4 look blocking; 5-7 are follow-ups.
Likely bugs
-
retryis a no-op formodelattribution (nemo_gym/core/outcome_policy.py:91).require_retry_or_failaccepts amodelclassification combined withretry, but_masked_evaluation_side_kindinrollout_recovery.pysetsnamespaced_retry = retry_declared and attribution != MODEL. The row is saved asretrybut never retried, and the masked row silently stays final. The new tests only assert saved fields for this case, not what recovery does. Either reject the pairing in the validator or make recovery honor it. -
Other masked kinds get no decision (
resources_servers/agentic_turn_simulation/app.py:701). Onlytime_capgetsfailure_outcome_actionand attribution. Any other entry inmasked_stop_reasonsis masked without them, sooutcome_policydoes not govern it. Thefailure_kind == TIME_CAP_KINDcheck is a fragile way to tie the decision to the stop reason. -
Same gap in
conversational_tool_use_simulation(app.py:1436). The decision is applied only forSIMULATOR_CALL_FAILED; other kinds from_masked_failureare masked with no decision, never retried, and fall back to an unknown classification. -
Time-cap rows are now retried by default (
agentic_turn_simulation/app.py:201). The default policy is evaluation + retry, so every time-capped row is rerun up to its attempt cap. The cap tracks serving speed, so a slow endpoint will hit it again and burn repeated full-episode attempts. This is a behavior change the PR doesn't flag.
Design and cleanup
-
Duplicated boilerplate.
agentic_turn_simulation,conversational_tool_use_simulationandvcqa_agenteach copy theoutcome_policyfield,keep_untouched_default_rulesandrequire_retry_or_fail_decisions, plus the literal stage"agent"and its kind tuple. A shared field factory or mixin taking(stage, kinds)inoutcome_policy.pywould prevent drift. -
Fragile overlay merge (
outcome_policy.py:53). It relies onisinstance(classification, dict), so a non-dict classification skips the merge and silently drops the boundary's default kind classification. It also re-dumps every shared stage default on each overlay. A named merge step would be clearer and safer. -
Untyped helper contract (
outcome_policy.py:106).applied_outcome_fieldsreturnsdict[str, Any]from the public API and hard-codes the private_ng_failure_stagekey, which conflicts with the repo'sAnyrule. It also passeshttp_status=None, so anyby_status_coderule in a policy is ignored.
Smaller, in vcqa_agent (app.py:626): decision fields are built with an inline conditional dict-splat in a large constructor call; computing outcome_fields first would be clearer. Only materialize failures get a decision; sandbox_start and model stage kinds get none.
| (``fail``). ``count_as_zero`` would save a masked row that is not a counted zero, so it is refused at load. | ||
| """ | ||
| for kind in failure_kinds: | ||
| action = choose_stage_outcome(failure_kind=kind, http_status=None, stage=stage, config=policy).action |
There was a problem hiding this comment.
retry can be a no-op for model attribution. This validator accepts any kind whose action is retry, including one reclassified as model. But _masked_evaluation_side_kind in rollout_recovery.py (line 469) sets namespaced_retry = retry_declared and attribution != FailureClass.MODEL, so that row is saved as retry and never retried; the masked row silently stays final. The new tests only assert the saved fields for this pairing, not what recovery does. Either reject model + retry here or make recovery honor it.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
Fixed. require_retry_or_fail now refuses retry for a kind that is classified as model, at configuration load, because recovery does not rerun a row attributed to the model. A test covers the refused pairing and the same kind left ungraded (fail), which is accepted.
| dict_fields=("by_failure_kind",), | ||
| ), | ||
| } | ||
| if isinstance(classification, dict) |
There was a problem hiding this comment.
Fragile overlay merge. The classification merge depends on isinstance(classification, dict), so a non-dict override skips the merge and silently drops the boundary's default kind classification. It also re-dumps every shared stage default via model_dump() on each overlay, and the inline conditional inside the returned dict literal is hard to read. A named merge step would be clearer and safer.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
I did not change the overlay in this PR. A classification that is neither a mapping nor a classification model is passed through unchanged and the field validation rejects it, so it fails at load rather than dropping the boundary defaults. Restructuring the merge step is outside this change.
| return policy | ||
|
|
||
|
|
||
| def applied_outcome_fields( |
There was a problem hiding this comment.
Untyped public contract. applied_outcome_fields is exported through nemo_gym.core but returns dict[str, Any] with the private _ng_failure_stage key hard-coded; recovery consumes these keys by name, so a rename is not caught by the type checker. AGENTS.md asks to avoid Any at public boundaries; a TypedDict or small model would document it. Also note http_status=None (line 305), so any by_status_code rule in a configured policy is ignored for these boundaries.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
I left the return type as it is; a typed result for applied_outcome_fields is outside this PR. On http_status=None: these boundaries choose the decision by failure kind and class, and I did not wire status codes into them.
| masked_stop_reasons: list[str] = Field(default_factory=lambda: ["time_cap"]) | ||
| # The decision saved with a row masked for a time_cap or a failed simulator. By default both kinds are classified as | ||
| # the evaluation's and that class is retried; a deployment can reclassify a kind or select fail for a class. | ||
| outcome_policy: StageOutcomePolicyConfig = Field( |
There was a problem hiding this comment.
Default policy changes behavior, and the PR doesn't flag it. With environment_fault_policy(...) the default for time_cap is evaluation + retry, so every time-capped row is now rerun by recovery up to its attempt cap. The time cap tracks serving speed, so a slow endpoint will hit it again and burn repeated full-episode attempts on the slowest rows; previously the row just stayed masked. Please call this out, and consider defaulting time_cap to fail.
Also: this field + keep_untouched_default_rules + require_retry_or_fail_decisions trio (and the literal "agent" stage and kind tuple, repeated three times here) is copied into conversational_tool_use_simulation and vcqa_agent. A shared helper in outcome_policy.py taking (stage, kinds) would stop the copies drifting.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
The retry default for time_cap is intended. A time-capped row says how fast the endpoint served, not what the model answered, so it is retried within the attempt cap instead of staying masked. A deployment that does not want the reruns sets by_stage.agent.by_failure_kind for that kind to fail, which leaves the row ungraded. The shared helper for the repeated field is outside this PR.
| return _masked(f"{FAILURE_KIND_PREFIX}:{stop_reason}", f"the turn was cut off ({stop_reason})") | ||
| masked = _masked(f"{FAILURE_KIND_PREFIX}:{stop_reason}", f"the turn was cut off ({stop_reason})") | ||
| # The wall-clock cap tracks serving speed: the policy decides what recovery does with it. | ||
| return masked | self._decision(TIME_CAP_KIND) if masked["failure_kind"] == TIME_CAP_KIND else masked |
There was a problem hiding this comment.
Only time_cap gets a decision. masked_stop_reasons is configurable, but any other masked kind falls to the else masked branch with no failure_outcome_action or attribution, so outcome_policy does not govern it (same for the context-overflow mask). Comparing failure_kind == TIME_CAP_KIND is a stringly-typed special case; consider deciding for every masked kind the policy covers.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
This PR declares the kinds each boundary reports as environment faults; for agentic_turn_simulation that is the time cap. Other masked stop reasons and the context-overflow mask keep their previous behavior: masked, without a decision. Extending the decision to them is outside this PR.
| failure_kind, failure_reason = self._masked_failure(state) | ||
| if failure_kind is not None: | ||
| response_payload.update(failure_kind=failure_kind, failure_reason=failure_reason) | ||
| if failure_kind == SIMULATOR_CALL_FAILED: |
There was a problem hiding this comment.
Same gap as in agentic_turn_simulation. The decision is applied only when failure_kind == SIMULATOR_CALL_FAILED. Any other kind _masked_failure returns is masked with no decision, is never retried, and falls back to the shared default (unknown) classification. That makes it inconsistent with the simulator-failed kind, which is retried by default.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
Same scope as in agentic_turn_simulation: this PR decides the kind the boundary declares, the simulator call failure. The other kinds from _masked_failure keep their previous behavior, and extending the decision to them is outside this PR.
| # The task's artifacts could not be materialized before the model ran: the shared policy decides what | ||
| # recovery does with the row. | ||
| **( | ||
| applied_outcome_fields(failure_kind=failure_kind, stage="agent", config=self.config.outcome_policy) |
There was a problem hiding this comment.
Inline conditional dict-splat in a large constructor call. Compute outcome_fields before the call instead. Also, only materialize failures (ARTIFACT_UNAVAILABLE) get a decision here; other stage kinds in _STAGE_FAILURE_KINDS such as sandbox_start and model get none, so the policy covers less than its description implies.
There was a problem hiding this comment.
🤖 Adam’s AI assistant
This PR covers the materialize failures of vcqa_agent; the other stage kinds keep their previous behavior. Extending the decision to them, and the cleanup of the constructor call, are outside this PR.
Recovery never reruns a row attributed to the model, so retry for such a kind saved a decision that never happened and left the masked row final. Refuse the pairing at configuration load, like count_as_zero. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…as the model The three environment boundaries refuse that pairing at configuration load, so the saved-decision cases cover evaluation retry, evaluation fail and model fail, and a separate case in each server asserts the refusal. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
giuliolovisotto
left a comment
There was a problem hiding this comment.
The earlier hardcoded-decision finding is resolved, including the model+retry validation. Re-review found one remaining configuration blocker: per-kind action overrides reject the environment kinds introduced here. Focused checks: 349 passed, 1 skipped.
| masked_stop_reasons: list[str] = Field(default_factory=lambda: ["time_cap"]) | ||
| # The decision saved with a row masked for a time_cap or a failed simulator. By default both kinds are classified as | ||
| # the evaluation's and that class is retried; a deployment can reclassify a kind or select fail for a class. | ||
| outcome_policy: StageOutcomePolicyConfig = Field( |
There was a problem hiding this comment.
[P2] Accept the declared environment kinds in per-kind action overrides
The new policy uses namespaced failure kinds, but its inherited action-rule validator (_require_registered_kinds) only accepts the shared registered vocabulary. I reproduced outcome_policy={"by_stage": {"agent": {"by_failure_kind": {"agentic_turn_simulation:time_cap": "fail"}}}} with make_server(): config loading raises Unregistered failure kinds: ['agentic_turn_simulation:time_cap'].
This prevents the selective override described in the discussion: disable retries for time caps while retaining simulator retries, with both faults still attributed to evaluation. Changing the stage default or class rule affects both kinds; changing classification to distinguish them would change attribution merely to select an action. The new conversational simulator and VCQA kinds are rejected by the same validator.
Please make per-kind action rules accept the environment kinds these policies declare (consistent with namespaced classification rules), and add a regression that changes only the time-cap action to fail while the simulator keeps its retry default and both retain evaluation attribution.
…n overrides Action rules validated kind names against the shared registry only, so a boundary's own <server>:<kind> names, which classification already accepts, could not be given their own action. They now pass the same form check; a misspelled registered name or a malformed namespaced name is still refused. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Register each server's own failure kinds through declare_action_kind, so an action rule can select them by name and a misspelled kind is still refused at configuration load. This replaces the form-only check of the previous commit. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
giuliolovisotto
left a comment
There was a problem hiding this comment.
Re-reviewed the current head. Both previously reported blockers are resolved: the producers use configurable shared policy, and declared namespaced kinds now work in per-kind action overrides while undeclared names are rejected. Verified that a time-cap fail override leaves simulator retry enabled, preserves evaluation attribution for both, and gives the expected recovery eligibility after serialization. Focused policy and three producer test modules: 352 passed, 1 skipped. No remaining blocking findings in this scope.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/44080143664 |
The conflict was in resources_servers/agentic_turn_simulation/app.py: this branch moved the server's settings and scoring into schemas.py and turns.py, and main (#6072) added an outcome policy to them. The policy (TIME_CAP_KIND, SIMULATOR_FAILED_KIND, the outcome_policy setting and its validators) now lives in AgenticTurnSimulationSettings, and the saved decision of a masked row in TurnSimulation._mask_reason, so the agent_arena environment server saves the same decision as the resources server. agent_arena's tests check it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
Several environments mask a sample for a fault on the evaluation side, but never say whether recovery should run it again, so such rows are never retried. The time limit and simulator failures of agentic_turn_simulation, the simulator call failure of conversational_tool_use_simulation, and an unavailable artifact in vcqa_agent now record
failure_outcome_action="retry"with an evaluation attribution (and the agent stage for the first three). Rerunning these rows depends on the masked-row handling in #6070. Unit tests in each environment check the recorded fields. No linked issue: it is the producer side of #6070.