Skip to content

fix(nooa): preserve complete assistant history across persistence - #4351

Merged
init-nikhil merged 1 commit into
mainfrom
init-nikhil/nooa-durable-history
Nov 5, 2026
Merged

init-nikhil merged 1 commit into
mainfrom
init-nikhil/nooa-durable-history

Conversation

@init-nikhil

@init-nikhil init-nikhil commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Current heads — November 4, 2026

Head 83be41989f40e708158e1c4dd47b327cd720905e; refreshed onto main 653846f949ad08a29df19c7de2ee4166d1c1bab3 before final CI/review. #4357 resolves main's dependency migration by keeping requirements.txt removed, updating all three NOOA pins in the managed environment pyproject.toml, and regenerating its uv.lock. The runtime loader and package layout from main are preserved. #4357 requires automation-team code-owner review for these dependency files.

Validation: 186 focused history, observability, result and sandbox-runtime tests passed; all-files pre-commit passed; uv lock --check --project responses_api_agents/nooa_agent passed. Current-head CI requested with /ok to test and must pass before merge. Below are earlier descriptions/evidence and may reference superseded heads.


Current stack revision — November 3, 2026

Head: 9b9a4f5b9e4e5bf5388193035023ea1500644d10. Stack refreshed onto main 3880fa5024bf97199cba96ade2e0fa6ec9eb6fbf with current import locations and merge conflicts resolved. Each PR remains a separate layer; landing uses native stack merge.

#4088 now packages the launching Gym checkout using its unchanged root pyproject.toml and full core dependencies. The separate responses_api_agents/nooa_agent/runtime/pyproject.toml was removed. Tracked source and package assets are copied with their current contents; untracked files, Git metadata, and caches are excluded. This requires Git on the controller. Sandbox installation stays in its private Python, and source changes invalidate prepared offline wheelhouses. #4813 retains its workflow change and code-owner review requirement.

Validation: 87 focused runtime, offline-bundle, and task-network-policy tests passed after the refresh; all-files pre-commit passed (private guardword catalog unavailable). A real Gym wheel built successfully from the new archive format, and its NOOA entrypoint, resource dispatcher, and core package asset were verified. No model rollout or Linux installation was run for this packaging change. Earlier evidence below is historical and does not establish fresh results for this head.

GitHub dismissed earlier reviews after this update; current-head reviews and required CI are still needed.

Stack refresh — 2026-11-02

Current head: f2910420f5f14795021c5764e2d1d61492890949. Stack refreshed onto main 2b8efc987af03951a80f941f2a278317badfe085. This PR remains a separate layer in the intended stack merge: #4088 → #4820 → #4821 → #4351 → #4357. #4813 is excluded.

Reconstructed each layer from its final reviewed patch instead of replaying foundation commits already merged into main. Resolved current package-import and legacy-environment renames while retaining upstream checkpoint handling and newer harness support. #4088 also exports its networking helpers through the current public API and mechanically updates existing callers; #4351 exports its newly used response types.

Fresh validation on the combined refreshed stack:

  • Full NOOA suite plus conformance runner/probe and harness-behavior tests: 913 passed, 10 Linux-only skips with pinned native NOOA 19caab169b018476ac433d040f6ae3f06aeff101.
  • Public-API facade suite and corrected GDPval recipe fixture: 372 passed.
  • Approved-prefix NOOA/network/recipe/metrics run: 671 passed, 10 skipped, and one GDPval fixture failure caused by the renamed legacy environment config; corrected and passed in the separate recipe run above.
  • pre-commit run --all-files: passed, including package layering, guardwords, lint and format.
  • Main advanced during validation; its intervening paths were disjoint from this stack. The final rebase was conflict-free, with the validated stack files unchanged.

No fresh real-model benchmark was run for this conflict/import refresh. Prior live-rollout evidence below is historical. Fresh reviews and required GitHub CI remain outstanding; this is not a merge-readiness claim.


Earlier implementation and validation (before this refresh)

  • translate authoritative native NOOA history into responses api input
  • preserve ordered assistant metadata through existing json/sqlite serialization when model routes match
  • fall back with explicit evidence gaps without restoring deleted raw-response content

Tests and validation

Head 055e0cfcee5fac6986f2c2dc4be5f833a370bdb7; base bc6a7e7976b29e08af1d9986489c2aabcdeea82c. The existing stack tracks this work; no separate issue is needed.

46 focused history/structured-output/preparation checks passed. Independent native JSON-on-disk and reopened SQLite cases preserved ordered training metadata; edited history and changed model routes produced portable fallback. The same-candidate large-array serialization/replay probe was retained and audited.

Fresh verification made 14 real HTTP history calls across three controlled cases and completed a separate two-call NVIDIA TaskAgent rollout. The synthetic marker task earned reward 1; exact history replay, raw capture, close and cleanup were inspected.

Focused test selection:

python -m pytest -q -p no:cacheprovider --import-mode=importlib \
  responses_api_agents/nooa_agent/tests/test_gym_llm.py \
  responses_api_agents/nooa_agent/tests/test_model_failures.py \
  -k 'history or replay or structured or unsupported or nontext or metadata or preparation or mixed or public or foreign or different_model'

Training-array durability is established by component cases; the live model did not return training token IDs. Public-dictionary provenance and non-model terminal-output gaps remain explicit, so strict ATIF export is not certified. The synthetic rollout used local Linux provisioning and an existing source-bound runtime; it does not qualify cloud infrastructure or cold installation.

pre-commit run --all-files passed. Fresh official Codex review returned no actionable findings; the independent bug hunt passed for this head and base.

CI snapshot (2026-10-24 05:44 UTC) on this head: 6 checks remain pending. Current checks. The all-green readiness gate remains unmet; no final verification-success post or new review request is implied.

@init-nikhil init-nikhil added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior labels Oct 9, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 9, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@init-nikhil
init-nikhil added this pull request to stack #4352 October 9, 2026 20:58
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-durable-history branch from fb70cd2 to ddca469 Compare October 9, 2026 21:58
init-nikhil added a commit that referenced this pull request Oct 9, 2026
… failures

Adapt the useful lifecycle, failure handling, and NVIDIA gateway changes
from Razvan Dinu's PR #4291 (49b8daa)
on top of #4351's durable ordered assistant history.

Keep successful BenchAgent services alive through verification, bound
finish and cleanup, preserve terminal exceptions, isolate optional summary
failures, and retain unrecovered infrastructure failures as ungraded.
Align all active NOOA pins, including controller bootstrap, at 19caab1.
Add regressions for cleanup ordering, cancellation, fresh delegated shells,
recovery, failed-call evidence, and gateway copy-on-write normalization.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-durable-history branch from d511c3d to 4640fd8 Compare October 10, 2026 05:25
init-nikhil added a commit that referenced this pull request Oct 10, 2026
… failures

Adapt the useful lifecycle, failure handling, and NVIDIA gateway changes
from Razvan Dinu's PR #4291 (49b8daa)
on top of #4351's durable ordered assistant history.

Keep successful BenchAgent services alive through verification, bound
finish and cleanup, preserve terminal exceptions, isolate optional summary
failures, and retain unrecovered infrastructure failures as ungraded.
Align all active NOOA pins, including controller bootstrap, at 19caab1.
Add regressions for cleanup ordering, cancellation, fresh delegated shells,
recovery, failed-call evidence, and gateway copy-on-write normalization.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

NOOA-only rebuild of PR #4357 from 4849245.

NOOA-only scope: shared infrastructure remains identical to main.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-durable-history branch from 4640fd8 to 089e0d2 Compare October 10, 2026 05:29
init-nikhil added a commit that referenced this pull request Oct 10, 2026
… failures

Adapt the useful lifecycle, failure handling, and NVIDIA gateway changes
from Razvan Dinu's PR #4291 (49b8daa)
on top of #4351's durable ordered assistant history.

Keep successful BenchAgent services alive through verification, bound
finish and cleanup, preserve terminal exceptions, isolate optional summary
failures, and retain unrecovered infrastructure failures as ungraded.
Align all active NOOA pins, including controller bootstrap, at 19caab1.
Add regressions for cleanup ordering, cancellation, fresh delegated shells,
recovery, failed-call evidence, and gateway copy-on-write normalization.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

NOOA-only rebuild of PR #4357 from 4849245.

NOOA-only scope: shared infrastructure remains identical to main.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
init-nikhil added a commit that referenced this pull request Oct 13, 2026
… failures

Adapt the useful lifecycle, failure handling, and NVIDIA gateway changes
from Razvan Dinu's PR #4291 (49b8daa)
on top of #4351's durable ordered assistant history.

Keep successful BenchAgent services alive through verification, bound
finish and cleanup, preserve terminal exceptions, isolate optional summary
failures, and retain unrecovered infrastructure failures as ungraded.
Align all active NOOA pins, including controller bootstrap, at 19caab1.
Add regressions for cleanup ordering, cancellation, fresh delegated shells,
recovery, failed-call evidence, and gateway copy-on-write normalization.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

NOOA-only rebuild of PR #4357 from 4849245.

NOOA-only scope: shared infrastructure remains identical to main.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-durable-history branch from 089e0d2 to 72072eb Compare October 13, 2026 08:22
@copy-pr-bot

copy-pr-bot Bot commented Oct 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

init-nikhil added a commit that referenced this pull request Oct 14, 2026
… failures

Adapt the useful lifecycle, failure handling, and NVIDIA gateway changes
from Razvan Dinu's PR #4291 (49b8daa)
on top of #4351's durable ordered assistant history.

Keep successful BenchAgent services alive through verification, bound
finish and cleanup, preserve terminal exceptions, isolate optional summary
failures, and retain unrecovered infrastructure failures as ungraded.
Align all active NOOA pins, including controller bootstrap, at 19caab1.
Add regressions for cleanup ordering, cancellation, fresh delegated shells,
recovery, failed-call evidence, and gateway copy-on-write normalization.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

NOOA-only rebuild of PR #4357 from 4849245.

NOOA-only scope: shared infrastructure remains identical to main.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-durable-history branch from 72072eb to 199d650 Compare October 14, 2026 20:28
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-durable-history branch from 0d2ae72 to 71a6ff4 Compare October 20, 2026 07:30

@init-nikhil init-nikhil left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CODEX SELF REVIEW

No actionable findings were returned by the official Codex review engine. This is not a guarantee of correctness.

Reviewed head: 71a6ff48bffdc0c4bb653af3481aa7c9ebb77022
Actual PR base: 4c56c44e964c1c5348d64fa66f3f7623f20c3c8d
Engine: official codex-cli 0.154.0, fresh codex exec --sandbox read-only --ephemeral --json review --base process using the unchanged account configuration. Exit code: 0.

Validation performed by the engine:

  • All 44 adapter tests passed against NOOA 051472343211914222e24ce36d8752f4e86bbe43.
  • 15 selected runner tests passed; 1 runner test was deselected.
  • The source checkout and Codex configuration fingerprints were unchanged after review.

Live-model smoke rollouts were not run by this review engine. Initial pytest setup failures were resolved by using in-memory output capture and binding imports to the reviewed checkout; the passing test invocations and original errors are preserved.

Run evidence: gymcheck-4351-20261020T074700Z/reviewer contains the invocation, stdout/stderr, original final response, test output, and before/after source/configuration fingerprints. Final response SHA-256: 08373c017889d941d00d96625de79dbf3e38269a9aba0bf03f9fb4cd0d1edef8.

@init-nikhil

Copy link
Copy Markdown
Member Author

Verification checks

PASS — evidence date: 2026-10-20.

Head: 71a6ff48bffdc0c4bb653af3481aa7c9ebb77022
Base: 4c56c44e964c1c5348d64fa66f3f7623f20c3c8d

Three component scenarios (14 local HTTP calls) and one real-model rollout (two inference calls) passed. Native metadata survives NOOA’s existing serialization/storage interfaces; the PR adds no disk-writing feature.

  • verify_history_http.py --case history-edits: actual NOOA formatting and HTTP requests retained edited reasoning/text/tools, summaries and omitted turns without restoring stale raw-response content.
  • verify_history_http.py --case restore-scope: JSON and reopened SQLite histories replayed ten native items exactly across changed rollout URLs, including opaque reasoning, refusal, hosted-tool items and training metadata.
  • verify_history_http.py --case portable: changed server/alias and missing replay state retained ordered public history, omitted opaque/training data and recorded explicit evidence gaps.
  • verify_history_http.py --case live-normal: actual TaskAgent worker → Gym adapter → VLLMModel → execute_python → return_result → collector → synthetic resource verifier. Two model calls, exact reasoning/tool replay into the second request, matching Python stdout, reward 1, two model-call joins and successful session close checks.

The live run is fresh. The three component PASS results are reused from the same manual run: their primary harness, source/runtime bindings and executed support paths are unchanged; the subsequent fixture correction affects only the live runtime selector. Earlier blocked fixture attempts remain preserved locally. Production source and cleanup assertions were unchanged.

Tested with local Linux transport, pinned NOOA 051472343211914222e24ce36d8752f4e86bbe43 and OpenAI 2.44.0. The terminal value has the expected non_trainable_terminal_output gap. Scope excludes cloud isolation/provisioning, runtime installation, live training-token validation, benchmark scoring and all-descendant cleanup.

@init-nikhil
init-nikhil marked this pull request as ready for review October 20, 2026 09:13
@init-nikhil
init-nikhil requested a review from furgalep October 20, 2026 09:13
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 20, 2026 09:14
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test 6dc2cc0

@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test da6a5d9

@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test d537a13

@furgalep furgalep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few findings on the new replay/persistence path, roughly most important first: the parsed/content mismatch (L437), missing gap for dict-rendered turns (L208), native replay ignoring public fields (L175), and the permissive scope guard (L172). The rest are lower risk or pre-existing.

@init-nikhil init-nikhil left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CODEX SELF REVIEW

Reviewed head 055e0cfcee5fac6986f2c2dc4be5f833a370bdb7 against actual base bc6a7e7976b29e08af1d9986489c2aabcdeea82c using a fresh official codex exec review process.

No actionable regressions were identified across the five changed files. The focused adapter and model-failure suites passed all 88 tests against the pinned NOOA dependency; real model rollouts were not run.

This is a source-review result, not an approving review or a substitute for the separate validation and CI gates.

furgalep
furgalep previously approved these changes Oct 25, 2026

@furgalep furgalep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test 5db6c04

@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test 60dff88

furgalep
furgalep previously approved these changes Oct 29, 2026

@furgalep furgalep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

furgalep
furgalep previously approved these changes Nov 3, 2026

@furgalep furgalep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test 9b9a4f5

adil-a
adil-a previously approved these changes Nov 4, 2026
@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test 5ddd254

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test 83be419

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41952348655

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41958832710

This branch was previously deployed

1 inactive deployment
public — 83be4198 Deployed Nov 4, 2026 by copy-pr-bot[bot] via release / finalize / notify #10180
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working complexity:medium Single-domain change with interacting parts or a moderate review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants