Skip to content

fix(envs): declare retry for environment-side faults they mask - #6072

Merged
AdamRajfer merged 11 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:arajfer/env-declare-retry
Nov 17, 2026
Merged

AdamRajfer merged 11 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:arajfer/env-declare-retry

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Several environments mask a sample for a fault on the evaluation side, but never say whether recovery should run it again, so such rows are never retried. The time limit and simulator failures of agentic_turn_simulation, the simulator call failure of conversational_tool_use_simulation, and an unavailable artifact in vcqa_agent now record failure_outcome_action="retry" with an evaluation attribution (and the agent stage for the first three). Rerunning these rows depends on the masked-row handling in #6070. Unit tests in each environment check the recorded fields. No linked issue: it is the producer side of #6070.

mimo_music scorer timeouts and errors, agentic_turn_simulation simulator
failures and wall-clock time_cap, conversational_tool_use_simulation
simulator call failures and vcqa_agent artifact materialization failures
are faults of the evaluation side, not of the policy. They were masked with
a namespaced failure kind and no outcome action, so recovery never reran
them and the sample silently left the denominator. Each now declares
failure_outcome_action=retry with evaluation attribution.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 12, 2026 18:43
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

…fits them

mimo_music: only a fault of the machine (timeout, killed or unstartable scorer,
failed renderer process) is a failure to grade this time. It keeps the model's
answer, so it is masked as judge_failed and recovery grades the saved answer
again. A missing renderer, an unparseable or out-of-range scorer result or a
scorer exception fails every rollout the same way: it is now
mimo_music:scorer_invalid with failure_outcome_action=fail.

agentic_turn_simulation (time_cap, simulator_failed) and
conversational_tool_use_simulation (simulator_call_failed) fault inside the
episode, whose answer is unusable. Their retry declaration now records the
agent stage, so the answer is not kept and recovery reruns the rollout; with
the verification stage implied, a retry declaration alone left them masked
without a rerun.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…e PR

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

@giuliolovisotto giuliolovisotto left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new failure decisions need to be configurable before they are persisted as applied policy; one finding inline.

if retry:
masked.update(
failure_outcome_action=OutcomeAction.RETRY,
failure_outcome_attribution=FailureClass.EVALUATION,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply configurable policy before saving these outcome decisions

These fields are consumed as an already-applied decision, but this path never consults the shared classifier or a configured outcome policy. For example, I reproduced a time-cap row with classification.by_stage.agent.by_failure_kind["agentic_turn_simulation:time_cap"] = "model" and recovery rules by_failure_class = {"model": "fail", "evaluation": "retry"}: the shared classifier returns model, but the row hardcodes evaluation, so _recovery_action() still selects retry. Consumers correctly preserve the saved attribution, and none of these producer configs provides a way to select a different decision before it is saved. The same hardcoded pair is added in conversational_tool_use_simulation and vcqa_agent.

Please keep the proposed defaults, but obtain the new action and attribution through the shared classification/outcome-policy mechanism with boundary configuration overrides, then persist the selected decision. Preserve the observed failure kind and originating stage independently of that action. Add a regression showing that the same fault can select retry or fail through configuration and that changing the action alone preserves attribution. This can stay scoped to the newly added decisions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

Fixed for all three producers. Each now has an outcome_policy that classifies the new kinds as evaluation and defaults the agent stage to retry, so the defaults you accepted are unchanged; the action and attribution come from the shared classifier and stage policy and are saved with the stage agent, while the observed failure_kind is left as it was. A configuration that would select count_as_zero for one of these kinds is refused at load, since these rows are masked and cannot be counted zeros. Tests run the same fault with retry and with fail, as evaluation and as model, and your time-cap configuration (model with fail, evaluation with retry) is one of them. A small separate commit makes StageOutcomePolicyConfig.overlay keep a boundary's default classification under a partial override.

…olicy override

StageOutcomePolicyConfig.overlay merged only the stage policies over a boundary's defaults, so an override that touched the classification dropped the boundary's own classification rules.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
The retry decisions this PR saves for a time_cap, a failed simulator and an unavailable task artifact no longer hardcode retry and evaluation. Each boundary has an outcome_policy whose defaults classify those kinds as the evaluation's and set the agent stage default to retry; the action and attribution come from the shared classifier and stage policy, so a deployment can reclassify a kind or select fail as the default. The observed failure kind and the agent stage are saved independently of the action.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…leave the row ungraded

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Nov 16, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 16, 2026 17:25
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the blocked Work cannot move forward until an external dependency is cleared label Nov 16, 2026

@oyilmaz-nvidia oyilmaz-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review notes. Items 1-4 look blocking; 5-7 are follow-ups.

Likely bugs

  1. retry is a no-op for model attribution (nemo_gym/core/outcome_policy.py:91). require_retry_or_fail accepts a model classification combined with retry, but _masked_evaluation_side_kind in rollout_recovery.py sets namespaced_retry = retry_declared and attribution != MODEL. The row is saved as retry but never retried, and the masked row silently stays final. The new tests only assert saved fields for this case, not what recovery does. Either reject the pairing in the validator or make recovery honor it.

  2. Other masked kinds get no decision (resources_servers/agentic_turn_simulation/app.py:701). Only time_cap gets failure_outcome_action and attribution. Any other entry in masked_stop_reasons is masked without them, so outcome_policy does not govern it. The failure_kind == TIME_CAP_KIND check is a fragile way to tie the decision to the stop reason.

  3. Same gap in conversational_tool_use_simulation (app.py:1436). The decision is applied only for SIMULATOR_CALL_FAILED; other kinds from _masked_failure are masked with no decision, never retried, and fall back to an unknown classification.

  4. Time-cap rows are now retried by default (agentic_turn_simulation/app.py:201). The default policy is evaluation + retry, so every time-capped row is rerun up to its attempt cap. The cap tracks serving speed, so a slow endpoint will hit it again and burn repeated full-episode attempts. This is a behavior change the PR doesn't flag.

Design and cleanup

  1. Duplicated boilerplate. agentic_turn_simulation, conversational_tool_use_simulation and vcqa_agent each copy the outcome_policy field, keep_untouched_default_rules and require_retry_or_fail_decisions, plus the literal stage "agent" and its kind tuple. A shared field factory or mixin taking (stage, kinds) in outcome_policy.py would prevent drift.

  2. Fragile overlay merge (outcome_policy.py:53). It relies on isinstance(classification, dict), so a non-dict classification skips the merge and silently drops the boundary's default kind classification. It also re-dumps every shared stage default on each overlay. A named merge step would be clearer and safer.

  3. Untyped helper contract (outcome_policy.py:106). applied_outcome_fields returns dict[str, Any] from the public API and hard-codes the private _ng_failure_stage key, which conflicts with the repo's Any rule. It also passes http_status=None, so any by_status_code rule in a policy is ignored.

Smaller, in vcqa_agent (app.py:626): decision fields are built with an inline conditional dict-splat in a large constructor call; computing outcome_fields first would be clearer. Only materialize failures get a decision; sandbox_start and model stage kinds get none.

Comment thread nemo_gym/core/outcome_policy.py Outdated
(``fail``). ``count_as_zero`` would save a masked row that is not a counted zero, so it is refused at load.
"""
for kind in failure_kinds:
action = choose_stage_outcome(failure_kind=kind, http_status=None, stage=stage, config=policy).action

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retry can be a no-op for model attribution. This validator accepts any kind whose action is retry, including one reclassified as model. But _masked_evaluation_side_kind in rollout_recovery.py (line 469) sets namespaced_retry = retry_declared and attribution != FailureClass.MODEL, so that row is saved as retry and never retried; the masked row silently stays final. The new tests only assert the saved fields for this pairing, not what recovery does. Either reject model + retry here or make recovery honor it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

Fixed. require_retry_or_fail now refuses retry for a kind that is classified as model, at configuration load, because recovery does not rerun a row attributed to the model. A test covers the refused pairing and the same kind left ungraded (fail), which is accepted.

dict_fields=("by_failure_kind",),
),
}
if isinstance(classification, dict)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fragile overlay merge. The classification merge depends on isinstance(classification, dict), so a non-dict override skips the merge and silently drops the boundary's default kind classification. It also re-dumps every shared stage default via model_dump() on each overlay, and the inline conditional inside the returned dict literal is hard to read. A named merge step would be clearer and safer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

I did not change the overlay in this PR. A classification that is neither a mapping nor a classification model is passed through unchanged and the field validation rejects it, so it fails at load rather than dropping the boundary defaults. Restructuring the merge step is outside this change.

return policy


def applied_outcome_fields(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Untyped public contract. applied_outcome_fields is exported through nemo_gym.core but returns dict[str, Any] with the private _ng_failure_stage key hard-coded; recovery consumes these keys by name, so a rename is not caught by the type checker. AGENTS.md asks to avoid Any at public boundaries; a TypedDict or small model would document it. Also note http_status=None (line 305), so any by_status_code rule in a configured policy is ignored for these boundaries.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

I left the return type as it is; a typed result for applied_outcome_fields is outside this PR. On http_status=None: these boundaries choose the decision by failure kind and class, and I did not wire status codes into them.

masked_stop_reasons: list[str] = Field(default_factory=lambda: ["time_cap"])
# The decision saved with a row masked for a time_cap or a failed simulator. By default both kinds are classified as
# the evaluation's and that class is retried; a deployment can reclassify a kind or select fail for a class.
outcome_policy: StageOutcomePolicyConfig = Field(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default policy changes behavior, and the PR doesn't flag it. With environment_fault_policy(...) the default for time_cap is evaluation + retry, so every time-capped row is now rerun by recovery up to its attempt cap. The time cap tracks serving speed, so a slow endpoint will hit it again and burn repeated full-episode attempts on the slowest rows; previously the row just stayed masked. Please call this out, and consider defaulting time_cap to fail.

Also: this field + keep_untouched_default_rules + require_retry_or_fail_decisions trio (and the literal "agent" stage and kind tuple, repeated three times here) is copied into conversational_tool_use_simulation and vcqa_agent. A shared helper in outcome_policy.py taking (stage, kinds) would stop the copies drifting.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

The retry default for time_cap is intended. A time-capped row says how fast the endpoint served, not what the model answered, so it is retried within the attempt cap instead of staying masked. A deployment that does not want the reruns sets by_stage.agent.by_failure_kind for that kind to fail, which leaves the row ungraded. The shared helper for the repeated field is outside this PR.

return _masked(f"{FAILURE_KIND_PREFIX}:{stop_reason}", f"the turn was cut off ({stop_reason})")
masked = _masked(f"{FAILURE_KIND_PREFIX}:{stop_reason}", f"the turn was cut off ({stop_reason})")
# The wall-clock cap tracks serving speed: the policy decides what recovery does with it.
return masked | self._decision(TIME_CAP_KIND) if masked["failure_kind"] == TIME_CAP_KIND else masked

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only time_cap gets a decision. masked_stop_reasons is configurable, but any other masked kind falls to the else masked branch with no failure_outcome_action or attribution, so outcome_policy does not govern it (same for the context-overflow mask). Comparing failure_kind == TIME_CAP_KIND is a stringly-typed special case; consider deciding for every masked kind the policy covers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

This PR declares the kinds each boundary reports as environment faults; for agentic_turn_simulation that is the time cap. Other masked stop reasons and the context-overflow mask keep their previous behavior: masked, without a decision. Extending the decision to them is outside this PR.

failure_kind, failure_reason = self._masked_failure(state)
if failure_kind is not None:
response_payload.update(failure_kind=failure_kind, failure_reason=failure_reason)
if failure_kind == SIMULATOR_CALL_FAILED:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same gap as in agentic_turn_simulation. The decision is applied only when failure_kind == SIMULATOR_CALL_FAILED. Any other kind _masked_failure returns is masked with no decision, is never retried, and falls back to the shared default (unknown) classification. That makes it inconsistent with the simulator-failed kind, which is retried by default.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

Same scope as in agentic_turn_simulation: this PR decides the kind the boundary declares, the simulator call failure. The other kinds from _masked_failure keep their previous behavior, and extending the decision to them is outside this PR.

# The task's artifacts could not be materialized before the model ran: the shared policy decides what
# recovery does with the row.
**(
applied_outcome_fields(failure_kind=failure_kind, stage="agent", config=self.config.outcome_policy)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline conditional dict-splat in a large constructor call. Compute outcome_fields before the call instead. Also, only materialize failures (ARTIFACT_UNAVAILABLE) get a decision here; other stage kinds in _STAGE_FAILURE_KINDS such as sandbox_start and model get none, so the policy covers less than its description implies.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

This PR covers the materialize failures of vcqa_agent; the other stage kinds keep their previous behavior. Extending the decision to them, and the cleanup of the constructor call, are outside this PR.

Recovery never reruns a row attributed to the model, so retry for such a kind
saved a decision that never happened and left the masked row final. Refuse the
pairing at configuration load, like count_as_zero.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…as the model

The three environment boundaries refuse that pairing at configuration load, so
the saved-decision cases cover evaluation retry, evaluation fail and model fail,
and a separate case in each server asserts the refusal.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

@giuliolovisotto giuliolovisotto left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earlier hardcoded-decision finding is resolved, including the model+retry validation. Re-review found one remaining configuration blocker: per-kind action overrides reject the environment kinds introduced here. Focused checks: 349 passed, 1 skipped.

masked_stop_reasons: list[str] = Field(default_factory=lambda: ["time_cap"])
# The decision saved with a row masked for a time_cap or a failed simulator. By default both kinds are classified as
# the evaluation's and that class is retried; a deployment can reclassify a kind or select fail for a class.
outcome_policy: StageOutcomePolicyConfig = Field(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Accept the declared environment kinds in per-kind action overrides

The new policy uses namespaced failure kinds, but its inherited action-rule validator (_require_registered_kinds) only accepts the shared registered vocabulary. I reproduced outcome_policy={"by_stage": {"agent": {"by_failure_kind": {"agentic_turn_simulation:time_cap": "fail"}}}} with make_server(): config loading raises Unregistered failure kinds: ['agentic_turn_simulation:time_cap'].

This prevents the selective override described in the discussion: disable retries for time caps while retaining simulator retries, with both faults still attributed to evaluation. Changing the stage default or class rule affects both kinds; changing classification to distinguish them would change attribution merely to select an action. The new conversational simulator and VCQA kinds are rejected by the same validator.

Please make per-kind action rules accept the environment kinds these policies declare (consistent with namespaced classification rules), and add a regression that changes only the time-cap action to fail while the simulator keeps its retry default and both retain evaluation attribution.

…n overrides

Action rules validated kind names against the shared registry only, so a
boundary's own <server>:<kind> names, which classification already accepts,
could not be given their own action. They now pass the same form check; a
misspelled registered name or a malformed namespaced name is still refused.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 17, 2026 11:38
Register each server's own failure kinds through declare_action_kind, so an
action rule can select them by name and a misspelled kind is still refused at
configuration load. This replaces the form-only check of the previous commit.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

@giuliolovisotto giuliolovisotto left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head. Both previously reported blockers are resolved: the producers use configurable shared policy, and declared namespaced kinds now work in per-kind action overrides while undeclared names are rejected. Verified that a time-cap fail override leaves simulator retry enabled, preserves evaluation attribution for both, and gives the expected recovery eligibility after serialization. Focused policy and three producer test modules: 352 passed, 1 skipped. No remaining blocking findings in this scope.

@AdamRajfer
AdamRajfer added this pull request to the merge queue Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/44080143664

Merged via the queue into NVIDIA-NeMo:main with commit d00063a Nov 17, 2026
76 of 78 checks passed
@AdamRajfer
AdamRajfer deleted the arajfer/env-declare-retry branch November 17, 2026 12:25
weijiac0619 added a commit that referenced this pull request Nov 17, 2026
The conflict was in resources_servers/agentic_turn_simulation/app.py: this branch moved the server's settings and
scoring into schemas.py and turns.py, and main (#6072) added an outcome policy to them. The policy (TIME_CAP_KIND,
SIMULATOR_FAILED_KIND, the outcome_policy setting and its validators) now lives in AgenticTurnSimulationSettings, and
the saved decision of a masked row in TurnSimulation._mask_reason, so the agent_arena environment server saves the
same decision as the resources server. agent_arena's tests check it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers blocked Work cannot move forward until an external dependency is cleared bug Something isn't working sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants