Skip to content

fix: masked results that still need a failure_kind #5693

Closed

Description

@oyilmaz-nvidia

Masked results that still need a failure_kind (#4882)

Findings were produced by read-only audit agents and have not been confirmed against a running rollout.

Needs a fix

Component Site Proposed fix Status
anyswe_agent Sandbox exception (error_kind="sandbox") masks with no kind provider_unavailable, or the event's kind; reason from repr(exc) Blocked by #4024
anyswe_agent agent_timed_out agent_timeout Blocked
anyswe_agent eval_timeout anyswe_agent:eval_timeout Blocked
anyswe_agent Resolved with max_iteration / context_window agent_limit / model_context_limit Blocked
anyterminal_agent container_timed_out or agent_timed_out or sandbox_failed Carry error_type through first so timeouts and sandbox errors separate; then agent_timeout, anyterminal_agent:eval_timeout and a provider kind Blocked by #2595
osworld_agent Configuration refusals (task_configuration_error, proxy_configuration_error, proxy_required_but_disabled) masked with no kind osworld_agent:configuration_error Blocked by open osworld PRs
rna_design Failed scoring call labelled agent_run_error verifier_error Free to fix
rna_design Missing campaign for the session labelled agent_run_error session_lost Free to fix
deepswe_external1 "No seeded sandbox", "session mismatch" and patch-collection failures all verifier_error session_lost and environment_server_failed, as deepswe does Free to fix
terminal_bench_4 Masked with no kind when termination.failure_kind and the fallback are both None Add a fallback kind Free to fix; confirm the path is reachable first

Needs a decision, not a fix

Activity

  1. oyilmaz-nvidia commented on Nov 11, 2026

    @oyilmaz-nvidia
    ContributorAuthor

    Checked every row against main (1f89875) by reading the code. Every site in "Needs a fix" already gives its masked result a failure_kind, so I'm closing this issue. I did not run rollouts; the kinds are covered by unit tests, not observed in runs.

    Component Kind now set Where
    anyswe_agent, sandbox exception the event's kind, else provider_unavailable; reason from repr(exc) anyswe_agent/app.py
    anyswe_agent, agent_timed_out / eval_timeout / resolved with a limit agent_timeout / anyswe_agent:eval_timeout / agent_limit or model_context_limit _masked_failure_kind
    anyterminal_agent error_type carried through first, then provider_unavailable, agent_timeout or anyterminal_agent:eval_timeout _masked_failure
    osworld_agent, configuration refusals osworld_agent:configuration_error (no failure event, so not retried) #5773
    rna_design, both rows verifier_error and session_lost #5710
    deepswe_external1 session_lost and environment_server_failed #5710
    terminal_bench_4 falls back to infrastructure_error, so a masked result always has a kind #5768

    Decision items:

    The "Blocked by" PRs listed in the table did not need to merge first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions