Skip to content

feat(officeenv): add the OfficeEnv resources server - #5009

Merged
d-molinari merged 13 commits into
mainfrom
dmolinari/officeenv-resources-server
Nov 5, 2026
Merged

d-molinari merged 13 commits into
mainfrom
dmolinari/officeenv-resources-server

Conversation

@d-molinari

@d-molinari d-molinari commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds resources_servers/officeenv, the resources server for OfficeEnv: a deterministic simulator of connected SaaS apps (107 apps at v1.10.0: Slack, Gmail, Google Calendar/Sheets/Drive, HubSpot, Jira, Salesforce, Zendesk, ...). A row's officeenv task spec names the connected apps and the data they hold, the acting user, how the apps reach the policy (tools, api_search, cli or mcp exposure), the faults and data injections that happen during the episode, and the final-state checks that produce the reward.

The environment is the officeenv package (pydantic and the standard library), at v1.10.0, installed as the Dependency section says. The server is a thin wrapper that maps Gym's routes onto OfficeEnv's framework-neutral EpisodeService (seed, call, verify, catalog) and adds Gym's own response fields:

  • Agent seeds (the simple agent's /run, behind the legacy_agent Environment Server in configs/officeenv.yaml): the row goes to /seed_session, each function call to POST /<tool_name> with its arguments as the body, the row and the response to /verify. App-level errors are tool outputs the model sees (200). A body that is not a JSON object (400) and a call without a session (404, ResourcesSessionLostError with failure_kind: session_lost) are integration failures.
  • Environment Server episodes: a ResourcesSeedSessionRequest seeds the OfficeEnv session under its resources_session_id, which the session cookie also carries. A repeated seed for the same episode returns the same session; a seed for another episode is refused with 409 (ResourcesSessionConflictError). /close_session frees a world that verification has not, and a seed of a closed ID is refused while Gym remembers the close. retire_session_state frees the world too. The tests run check_resources_session_contract, including its state-release and masked-verify-after-close probes.
  • MCP exposure (expose_tools_over_mcp: true): every tool of OfficeEnv's tools exposure is served through the catch-all tool route, and each episode's MCP token admits only that episode's tools. MCP clients read tool definitions from the endpoint, not from the row, so a seed whose tools differ from what the endpoint serves (a row in another exposure mode or with another tool-name separator) fails with 400 unsupported_exposure.
  • /verify: OfficeEnv's reward, per-check verdicts, metrics with flat numeric copies for aggregation, scenario and integration logs, and the grading version. A rollout without a session to grade gets reward 0 with mask_sample: true and failure_kind: session_lost (lost_session_verification).
  • Artifacts (artifacts_dirs): a row may name its TaskSpec's large strings by SHA-256 instead of carrying them (OfficeEnv's artifact format, officeenv.artifacts), and a corpus is one directory, <corpus>/<rows>.jsonl beside <corpus>/artifacts/. The server hands the stores (absolute paths, searched in order) to OfficeEnv's episode service, which reads the references back at seed, checks each against its hash and size, and refuses a row whose artifacts cannot be read with 400 (missing_artifact, artifact_mismatch or invalid_artifact) before anything is seeded, for an Agent's seed and an Environment Server's alike. The README describes the corpus layout.
  • Training length penalty (length_penalty.enabled, off by default): AutomationBench's, reported as its own term in reward_components (a BaseMultiRewardVerifyResponse), so a trainer can schedule it separately (NeMo RL grpo.phasic_reward_components).
  • Workers (num_workers): above one, Uvicorn serves the app from each worker (the module-level app), and Gym's session routing sends each episode's requests to the worker that seeded it.

The example rows are five of OfficeEnv's built-in examples, one per exposure mode plus a task with a scripted user. task_data.py declares the row's task fields for Gym's data tooling and the task-data drift gate: the TaskSpec (the only field the wire requires; OfficeEnv validates it at seed), the emitting version, and the task_id.

Dependency

OfficeEnv will be released at a later date. Until then, the server's pyproject.toml and uv.lock hold only its public dependencies, as main's lock rule asks (#4837), and OfficeEnv 1.10.0 installs apart, from NVIDIA's internal GitLab, with a README step that takes the repository's git URL from OFFICEENV_GIT_URL. Without OfficeEnv the server stops at start with an error naming that step, and its tests skip, so CI builds the venv and runs the suite without it. Once the release is out, OfficeEnv moves into pyproject.toml and the lock. Point of contact: Daniele Molinari (dmolinari@nvidia.com).

Validation

Linux aarch64, Python 3.13, OfficeEnv 1.10.0 (the v1.10.0 tag's commit 06e65854), on main 83b5aaf08c (2026-11-04); earlier heads also on macOS arm64. The rebase onto 83b5aaf08c leaves the 12 commits approved at f239b09109 unchanged (git range-diff). One commit follows: it moves the server to locked public dependencies (pyproject.toml and uv.lock) and installs OfficeEnv apart, which also keeps the internal host, which the guardwords check flagged, out of the repository.

  • gym env test --resources-server officeenv +should_validate_data=true: 31 passed with OfficeEnv installed, and the data is validated. Without OfficeEnv the guard's test passes, the other tests skip, and the data still validates.
  • Workers: the server started as Gym starts it (python app.py, the config in NEMO_GYM_CONFIG_DICT) with 1, 2 and 4 workers, and 16 concurrent episodes for each: seed, a Slack post that meets the task's authored fault, a second that succeeds, and verify, each episode on its own session cookie. All 16 reach reward 1.0, unmasked, with each worker count. Every request opens its own connection, so most of an episode's requests reach a worker that did not seed it; the sessions were spread 6/10 over two workers and 2/4/5/5 over four. Before the worker fix, two workers never served: Uvicorn found no app in resources_servers.officeenv.app, and the server exited.
  • The five example rollouts, replayed call by call through this server on 1.9.0 before the rebase: every recorded tool output (61 calls), and each rollout's recorded reward and checks.
  • pre-commit over every file this PR changes: all hooks pass, import-linter included (the guardwords hook skips itself without the private catalog).
  • gym dataset collate "+config_paths=[resources_servers/officeenv/configs/officeenv.yaml]" +mode=example_validation: data/example_metrics.json collated again for the rows 1.10.0 emits; it moves only in the tools' statistics (schedule_kickoff's Calendar tools take send_updates in 1.10.0), and a second collate into a fresh folder finds it up to date.
  • detect-secrets-hook 1.5.0 with .github/workflows/config/.secrets.baseline over every file this PR changes: no findings.
  • This PR changes no file under nemo_gym/. Gym's core unit suite (pytest tests -m "not sandbox", the dev extra only) fails here only as on main 83b5aaf08c itself, in the same venv: the same 28 tests (eval resubmission, Harbor's compose overlay and run summary, reward profiles, outcome policies, the dataset orchestrator, the OpenSandbox scripts, the CI environment, the benchmark listing, failure reports) and 5 test modules that need packages the venv lacks. That includes tests/unit_tests/data/test_task_data.py (the officeenv schema checks and the drift gate over committed rows), tests/unit_tests/test_public_api_facades.py (the server's imports come from the packages' public APIs) and tests/unit_tests/tooling/test_component_dependencies.py (main's lock rule).
  • DCO: every commit is signed off.

Test coverage: Agent seeds through tool calls and verify; every exposure mode reaching the apps through the tool route; integration-failure statuses; lost sessions (404 tool calls, masked verify); artifact stores (a moved corpus seeds from the store beside its rows through both seed kinds, stores are searched in order, a row whose artifacts no store holds is refused and seeds nothing, relative stores are refused); the example rows matching the pinned release (no tool drift); the row schema and the seed request declaring the TaskSpec alike; the Environment Server session contract (check_resources_session_contract), episode binding, freeing on close and retire; MCP grants (tools/list narrowed per episode, tools/call through the catch-all route, verify afterwards), the agent seed's MCP token, and refusal of unserved tools; the catalog matching what rows show the model; the length penalty and turn counting.

Rollout evidence

Real model smoke runs on this branch against OfficeEnv 1.9.0 (the example rows have since been emitted again by 1.10.0, which changes only their version stamp and schedule_kickoff's Calendar tools), before its rebase onto main (the rollouts replay unchanged on it, above), azure/openai/gpt-5.5 through the NVIDIA inference gateway, the five example rows (emitted by 1.9.0), concurrency 2:

task exposure Agent seeds (officeenv_simple_agent) Environment Server (single_agent_turn_legacy, direct_http)
renewal_escalation api_search 1.0 (6/6 checks) 1.0 (6/6)
schedule_kickoff tools 1.0 (6/6) 0.75 (5/6)
vendor_terms_review cli 1.0 (11/11) 1.0 (11/11)
ticket_triage mcp 1.0 (11/11) 1.0 (11/11)
sprint_handover tools (scripted user) 1.0 (5/5) 1.0 (5/5)
  • The Agent-seed run is the committed data/example_rollouts.jsonl; every rollout was graded by 1.9.0.
  • schedule_kickoff turns on where the model books the meeting: the Agent-seed run booked 14:00, after a meeting injected during the episode, and passes every check; the Environment Server run booked 13:30, over that meeting, and fails gcalendar.no_conflicts, as the 1.7.0 and 1.8.0 runs did.
  • Every error the metrics count is an authored fault (a transient HubSpot error, Calendar and HubSpot rate limits) or a tool mistake the model recovered from (an unknown CLI option, a Slack DM before open_dm, an unknown HubSpot op). No rollout met 1.9.0's new refusal of a Gmail attachment sent by file name alone (T50). There were no internal errors, timer failures, tool drift, failed rollouts or masked samples.
  • The Environment Server run used a scratch config (not committed) that wraps the simple agent in single_agent_turn_legacy with resources_tool_transports: [direct_http], so every episode went through typed seed, cookie-scoped tool calls, verify and close.
  • The MCP exposure and the artifact stores are covered by the in-process tests above; no MCP-native harness rollout and no artifact corpus rollout are part of this PR.

Compatibility

A new server: nothing else changes except the generated row in the root README's environment table. configs/officeenv.yaml keeps the internal branch's instance names (officeenv_resources_server, officeenv_simple_agent, officeenv_environment_server), so existing overrides such as ++env.nemo_gym.officeenv_resources_server.resources_servers.officeenv.length_penalty.enabled=true keep working.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@d-molinari d-molinari added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work labels Oct 23, 2026
@d-molinari
d-molinari force-pushed the dmolinari/officeenv-resources-server branch from 664808c to 4f25da6 Compare October 27, 2026 22:57
@cmunley1
cmunley1 added this pull request to stack #5399 October 31, 2026 00:18
@d-molinari
d-molinari force-pushed the dmolinari/officeenv-resources-server branch from b6a008c to aaaca21 Compare October 31, 2026 03:38
@d-molinari
d-molinari marked this pull request as ready for review October 31, 2026 04:07

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — multi-worker startup is broken, and the private dependency prevents credential-free installation.

Reviewed all 12 changed paths and their directly affected startup, agent, session, MCP, installation, and data-contract paths. This was static inspection; no tests were executed. OfficeEnv's external implementation was unavailable, and long rollout JSONL lines were limited to bounded previews, so recorded rewards were not independently verified.

NOTE — Surface OfficeEnv in the published documentation. The integration adds substantial user-facing functionality, but the published environment catalog contains no OfficeEnv entry, and the changes add no Fern instructions. The pinned rubric requires published documentation for notable user-facing features. Please add a discoverable OfficeEnv entry and concise usage instructions, linking the detailed component README. Preserve the distinction between OfficeEnv's simulated mcp exposure and Gym's MCP transport, the absolute artifact-store paths, the opt-in length penalty, and the examples' smoke-test—not benchmark—status. Proposed acceptance: the published catalog reaches the instructions; launch/evaluate examples match the shipped configuration; links and make docs-check pass. These checks were not run.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.



if __name__ == "__main__":
OfficeEnvResourcesServer.run_webserver()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Initialize the module-level application in spawned workers. Setting officeenv_resources_server.resources_servers.officeenv.num_workers=2 takes the inherited multi-worker startup path, which tells Uvicorn to import resources_servers.officeenv.app:app. This module only handles __name__ == "__main__" and never defines that module-level app, so the workers cannot start. The in-process tests call setup_webserver() directly and cannot catch this deployment failure.

Could you add the existing worker-entrypoint hook? Add from nemo_gym.utils import is_nemo_gym_fastapi_entrypoint with the imports, then replace the final block with:

if __name__ == "__main__":
    OfficeEnvResourcesServer.run_webserver()
elif is_nemo_gym_fastapi_entrypoint(__file__):
    app = OfficeEnvResourcesServer.run_webserver()

This matches the existing resources-server entrypoint contract. In a worker, run_webserver() returns the configured application; an ordinary test import must still avoid starting a server, and single-worker script startup must remain unchanged.

Proposed acceptance, not executed: with the dependency installed and valid model configuration, launch OfficeEnv with two workers and confirm readiness, then seed, call a tool, and verify using the returned cookie. Repeat with one worker. Importing this module outside a Gym worker must not invoke run_webserver(). Please verify the suggested code before adopting it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 71edeaeda6 with the hook you suggested, as example_single_tool_call has it. With num_workers above one, Uvicorn imports resources_servers.officeenv.app:app in each worker. The module defined no app, so no worker could start.

Each worker holds the episodes it seeded, and Gym's session routing (on for resources servers) forwards each episode's requests to that worker, by session cookie or MCP token.

I ran your acceptance as written, starting the server as Gym does (python app.py, with the config in NEMO_GYM_CONFIG_DICT):

  • Before the fix, two workers never served. Uvicorn logged Error loading ASGI app. Attribute "app" not found in module "resources_servers.officeenv.app", and the server exited.
  • With the fix, I ran 16 concurrent episodes with 1, 2 and 4 workers. Each episode was seeded, made a Slack post that meets the task's authored fault, made a second post that succeeds, and was verified, all on its own cookie. All 16 reached reward 1.0, unmasked, at every worker count.
  • Every request opened its own connection, so most of an episode's requests reached a worker that had not seeded it. The sessions were spread 6/10 over two workers and 2/4/5/5 over four.
  • Importing the module outside a worker starts no server: the test suite imports it throughout.

I haven't committed a multi-worker test. Like main's other servers with this hook, it relies on core's session-routing e2e for the routing itself. I'm happy to add one if you'd like it here.

@@ -0,0 +1,2 @@
-e nemo-gym[dev] @ ../../
officeenv @ git+ssh://git@gitlab-master.nvidia.com:12051/dmolinari/officeenv.git@v1.9.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Replace the private dependency before merging. A fresh installation requires SSH access to an unpublished internal repository, as the README confirms. The venv installer installs this requirement unconditionally. Users without that access cannot launch OfficeEnv; the changed-server job and scheduled server suite also include it. The draft status appropriately acknowledges this but does not resolve the merge-time failure.

Please replace the pin with an approved public, reproducibly pinned release providing the required 1.9.0 APIs, and remove the internal URL/access instructions from the README. Protected public-repository guidance also prohibits internal URLs. Public replacement coordinates are unavailable here, so an exact dependency patch would be speculative.

Preserve: artifact resolution, row/tool compatibility, and real server-test coverage; do not merely remove the dependency or skip CI. Proposed acceptance, not executed: from a clean checkout without internal credentials or cached OfficeEnv packages, gym env test --resources-server officeenv +should_validate_data=true must install and pass. Retain empty tool_drift for all five version-matched rows, unchanged workbook/text retrieval through both seed shapes, and rejection of tampered or missing artifacts.

class TaskData(BaseModel):
model_config = ConfigDict(extra="allow")

officeenv: dict[str, Any] = Field(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE — Suggestion: enforce the duplicated TaskSpec field contract. officeenv is declared independently here and in OfficeEnvSeedSessionRequest. The repository drift gate and example seed test exercise valid rows, but do not compare these declarations. Making the field optional in only one declaration would leave those checks satisfied. This is a missing drift guard, not a demonstrated current seed failure; the pinned rubric calls for enforcement when declarations are parallel.

A focused addition to the server tests would protect the current invariant:

def test_task_spec_wire_contract_matches_schema() -> None:
    from typing import Any

    from resources_servers.officeenv.app import OfficeEnvSeedSessionRequest
    from resources_servers.officeenv.task_data import TaskData

    for model in (TaskData, OfficeEnvSeedSessionRequest):
        field = model.model_fields["officeenv"]
        assert field.annotation == dict[str, Any]
        assert field.is_required()
        assert model.model_config["extra"] == "allow"

Preserve: optional provenance fields, allowance of extra row fields, and delegation of TaskSpec-content validation to OfficeEnv rather than copying its external schema. Proposed acceptance: this test passes now and fails when either declaration alone changes the field's type, requiredness, or unknown-field policy. Existing committed-row validation and unstamped-row seeding must remain valid. The suggested test was not executed and should be verified before adoption.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in caa6268b0b as test_the_row_schema_and_the_seed_request_declare_the_taskspec_alike in tests/test_app.py, following your sketch.

  • It passes now.
  • It fails when either declaration alone makes the field optional, changes its type, or stops allowing extra fields. I checked each case by editing one declaration at a time.
  • The provenance fields, the allowance of extra row fields, and OfficeEnv's own TaskSpec validation are unchanged.

d-molinari added a commit that referenced this pull request Nov 3, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Nov 3, 2026
@d-molinari
d-molinari force-pushed the dmolinari/officeenv-resources-server branch from 8a118d2 to caa6268 Compare November 3, 2026 17:28
d-molinari added a commit that referenced this pull request Nov 3, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari added a commit that referenced this pull request Nov 3, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari added a commit that referenced this pull request Nov 3, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
@cmunley1

cmunley1 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

/ok to test f239b09

@cmunley1

cmunley1 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

/review

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — the worker-startup and TaskSpec drift-guard findings are fixed; the private dependency and published-documentation findings remain unresolved.

I verified the worker entrypoint against the current runner: workers now receive the module-level application, while ordinary imports and single-worker startup retain their existing behavior. The new TaskSpec contract test checks both declarations for the required dictionary type and allowance of extra fields.

This follow-up checked the prior findings, identifiable fixes, and directly affected contracts. The previously reviewed tree wasn't provided, so I couldn't compare the complete inter-review delta. OfficeEnv's external implementation wasn't available. This was static inspection; I didn't execute installation, tests, or model rollouts. No qualifying new regression was established.

NOTE — Suggestion: surface OfficeEnv in the published documentation. This is the unresolved body finding from review 5721205156. The current Agentic / Tool Use catalog still has no OfficeEnv entry, and the changeset adds no Fern instructions. Readers browsing the published catalog therefore cannot discover the integration or its launch instructions. The pinned rubric requires published documentation for notable user-facing features.

Could you add a catalog row and a short published usage section, linking the component README for advanced configuration? A small suggested addition at the end of that catalog table is:

| [officeenv](#officeenv) | Deterministic connected-app tasks | Rule-based (final-state checks) | |

### OfficeEnv

With your OpenAI model configuration set, start the environment and collect the example rollouts:

```bash
gym env start --resources-server officeenv --model-type openai_model
gym eval run --no-serve --agent officeenv_simple_agent \
  --input resources_servers/officeenv/data/example.jsonl \
  --output results/officeenv_rollouts.jsonl
```

These five examples are integration smoke tests, not a benchmark. OfficeEnv's simulated `mcp` exposure is distinct from Gym's opt-in MCP transport. Artifact-store paths must be absolute. The length penalty is off by default and intended for training only.

This preserves the existing exposure, artifact-path, reward, and smoke-test distinctions. Proposed acceptance, not executed: the published catalog reaches the usage section and detailed README; the commands match the shipped configuration and example dataset; the links and make docs-check pass. Please verify the suggested addition before adopting it.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

-e nemo-gym[dev] @ ../../
# OfficeEnv is not public yet. It will be released at a later date, and this pin will then point to that release;
# until then it installs from NVIDIA's internal GitLab. Point of contact: Daniele Molinari <dmolinari@nvidia.com>
officeenv @ git+ssh://git@gitlab-master.nvidia.com:12051/post-training/officeenv.git@v1.10.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — The credential-free installation failure remains unresolved. This is the same failure as prior comment 4430467912 in review 5721205156. From a clean checkout without internal repository access, gym env test --resources-server officeenv +should_validate_data=true still requires this private SSH dependency. The README confirms that prerequisite. The installer installs the requirement before pytest runs, so users without access cannot install or test the integration. Changing the repository coordinates and adding an explanatory comment doesn't resolve that failure.

The changed-server job and scheduled suite include this server and require the same access. I haven't established whether those runners have credentials. Protected public-content guidance also prohibits internal URLs.

Could you replace this requirement with an approved public, reproducibly pinned release providing the current EpisodeService and ArtifactStore APIs, then remove the internal URL/access instructions from this manifest and the README? The invariant is that a fresh public installation can install and test the actual integration without internal credentials—not by removing the dependency or skipping CI. Public replacement coordinates aren't supplied, so an exact dependency patch would be speculative.

Preserve and check: all five version-matched rows must seed successfully with empty tool_drift, as asserted by the compatibility test. The artifact tests must retain exact workbook/text retrieval through both seed shapes, ordered-store lookup, artifact_mismatch for tampered content, and missing_artifact with no stored episode for unavailable content; rows without references must still seed. The fault-then-success episode must still finish at reward 1.0, unmasked.

Proposed acceptance, not executed: with no internal credentials, cached OfficeEnv package, or prebuilt server venv, the command above installs and passes, including those assertions.

cmunley1
cmunley1 previously approved these changes Nov 4, 2026
d-molinari added a commit that referenced this pull request Nov 4, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
@d-molinari
d-molinari force-pushed the dmolinari/officeenv-resources-server branch from aa1206f to f42e4da Compare November 4, 2026 19:38
@cmunley1

cmunley1 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the update. One request on the lock exemption: rather than adding an exception to scripts/component_locks.py, could the server follow the lock rule with its public dependencies only?

  • Give resources_servers/officeenv a normal pyproject.toml and uv.lock that list only public packages.
  • Until OfficeEnv is published, install it separately (a README step using OFFICEENV_GIT_URL), and import it lazily in the server with a clear error when it's missing.
  • Skip the tests that need OfficeEnv when it isn't installed.

That keeps the lock rule the same for every component, avoids the core change, and nothing internal ends up in the lock. Once OfficeEnv is published it moves into the lock like any other dependency.

…OfficeEnv apart until it is published

Main now requires every server to declare its dependencies in a committed pyproject.toml and uv.lock (#4837). OfficeEnv
is not published yet, and a lock that held it would name NVIDIA's internal repository, which CI cannot reach and the
guardwords check refuses.

The server's pyproject.toml and uv.lock now list only its public dependencies (nemo-gym[dev], declared and locked as the
other servers with that dependency do), and requirements.txt is gone. OfficeEnv installs apart, into the venv Gym
builds, with the step the README gives, from the repository OFFICEENV_GIT_URL names; Gym's later builds only add to that
venv. Without OfficeEnv, app.py stops with an ImportError that names the step, which a test checks with OfficeEnv hidden;
the tests that need OfficeEnv skip, so a venv without it still runs one test rather than none. Once OfficeEnv is
published it moves into pyproject.toml and the lock like any other dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari added a commit that referenced this pull request Nov 4, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
@d-molinari
d-molinari force-pushed the dmolinari/officeenv-resources-server branch from f42e4da to 461815c Compare November 4, 2026 20:12
@d-molinari
d-molinari requested a review from a team as a code owner November 4, 2026 20:12
@cmunley1

cmunley1 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

/ok to test 461815c

@cmunley1
cmunley1 added this pull request to the merge queue Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41920399486

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41920845141

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 4, 2026
@d-molinari
d-molinari added this pull request to the merge queue Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41937686420

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41937735853

Merged via the queue into main with commit d05040d Nov 5, 2026
58 of 63 checks passed
@d-molinari
d-molinari deleted the dmolinari/officeenv-resources-server branch November 5, 2026 00:12
d-molinari added a commit that referenced this pull request Nov 5, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari added a commit that referenced this pull request Nov 6, 2026
#5009 now pins OfficeEnv 1.10.0, whose sandbox examples hold four more
tasks: sandbox_budget_variance, sandbox_statement_aging,
sandbox_stock_reorder and sandbox_timesheet_billing, on files in Box,
SharePoint, OneDrive and a Gmail attachment. sandbox/prepare_rows.py
writes data/sandbox_pilot.jsonl again with it: nine rows, the five it
held changed only in their version stamp.

The test of those rows asked every task for Drive's download tool; it now
asks each for user_complete_task and for a tool that reads a stored file
in any app (CONTENT_DOWNLOADS and ATTACHMENT_DOWNLOADS), as the new tasks
read theirs from the other four. The README no longer counts the tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari added a commit that referenced this pull request Nov 12, 2026
…urces_servers/officeenv_sandbox

The sandbox mode the OfficeEnv server ran when sandbox_provider was set is now a server of its own:
resources_servers/officeenv_sandbox, whose OfficeEnvSandboxResourcesServer subclasses OfficeEnvResourcesServer.
What a sandboxed episode does is unchanged. The OfficeEnv server keeps #5009's routes and options, plus
GET /task_state and close_tools_on_task_end for its Environment Server, and goes back to nemo-gym[dev]: the
sandbox providers' libraries install with the new server only.

- resources_servers/officeenv/app.py goes from 1,860 to 473 lines (371 on main). Its seed, episode call and
  verification keep hook points the subclass extends: _new_episode, _unreachable with a workspace,
  _refuse_after_task_end and _publish_task_state around _episode_call, and _verification.
- resources_servers/officeenv_sandbox holds the sandbox lending and shutdown, the cli and api_search surfaces,
  the workspace-mode host and file API, the guarded routes and per-process secret, and the seed turns. Its
  configs, images, rows, README sections and tests moved from the OfficeEnv server's folder. The configs name
  the server officeenv_sandbox (++oe.resources_servers.officeenv_sandbox.<option>), where sandbox_provider is
  required. An Agent's seed there gets no sandbox, as before.
- The new server has its own lock, task-data schema (sandbox_needs), an install test, and example data: the
  first five pilot rows, Pi's rollouts of them with gpt-5.5 on Docker sandboxes (all at reward 1.0), and
  their metrics. Gym's README lists it.

The OfficeEnv server's tests are 41 and the sandboxed server's 134 (the 133 that moved, and its install test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 461815c9 Deployed Nov 4, 2026 by copy-pr-bot[bot] via release / finalize / notify #10138
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants