Repository navigation
feat(officeenv): an image with document tools, for the sandboxed rows that need them - #5374
Closed
d-molinari wants to merge 26 commits into
Closed
d-molinari wants to merge 26 commits into
d-molinari wants to merge 26 commits into
Conversation
OfficeEnv is a deterministic simulator of connected SaaS apps (Slack, Gmail, Google Calendar, HubSpot, Jira and about a hundred more). This server routes Gym's endpoints to OfficeEnv's framework-neutral episode service and pins OfficeEnv v1.7.0 in requirements.txt. - Agent seeds: the row to /seed_session, each function call to POST /<tool_name>, the row and the response to /verify. - Environment Server episodes: typed seeds keyed by resources_session_id and idempotent per episode; /close_session frees a world verification has not, and a closed ID is refused while Gym remembers the close. - MCP exposure: every tool of OfficeEnv's tools exposure, each episode's token narrowed to its own tools; a seed whose tools differ from what the endpoint serves is refused. - /verify returns OfficeEnv's reward, checks, metrics and logs; a lost session is masked with failure_kind session_lost. An optional training length penalty reports its own reward component. The example rows are five of OfficeEnv's built-in examples; their rollouts were collected on Gym main with azure/openai/gpt-5.5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
Every resources server ships a task_data.py exporting TaskData, which Gym's data tooling and tests/unit_tests/test_task_data.py validate the committed rows against. Declare the three task fields an OfficeEnv row carries: the TaskSpec (the only one the wire requires), the version that emitted the row and its task_id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…sions Gym main now answers a tool call for a session the server no longer holds with ResourcesSessionLostError (404, failure_kind session_lost), builds the masked verify of such a session with lost_session_verification, and refuses conflicting seeds and closes with ResourcesSessionConflictError (still 409). The contract test now also checks that a close releases the session's world and that a verify after the close answers with a masked sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
requirements.txt pins v1.8.0 (minor: per-app id streams, the check semantics and core helper adoption, Sheets formulas, binary Drive files, the katana and dashboards apps). The five example rows are re-emitted by 1.8.0, which changes only their version stamp, and their rollouts are re-collected on this branch with azure/openai/gpt-5.5: four score 1.0 and schedule_kickoff 0.75, as on 1.7.0. example_metrics.json is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…tore OfficeEnv 1.9.0 makes the artifact reference its own contract (officeenv.artifacts): a row may name its TaskSpec's large strings by SHA-256, and a corpus is one directory, <corpus>/<rows>.jsonl beside <corpus>/artifacts/. Its episode service reads the references back at seed from an ArtifactStore, each checked against its hash, and refuses a row whose artifacts cannot be read (400 missing_artifact, artifact_mismatch or invalid_artifact) before anything is seeded. - artifacts_dirs names the stores, searched in order; artifacts_dir is one more, searched last. Both must be absolute paths, since the server runs in its own directory. - The server hands the store to the episode service, so an Agent's seed and an Environment Server's resolve alike. - Tests: a moved corpus seeds from the store beside its rows, for both seed kinds; stores are searched in order and artifacts_dir last; a row whose artifacts no store holds is refused and seeds nothing; relative stores are refused. The README describes the corpus layout. Ported from the internal sandbox and e2e branches (5d7f6aa53, cc7212130) without their copy of the module or its fallback: this imports officeenv.artifacts, which the 1.9.0 pin in the next commit provides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…lected rollouts requirements.txt pins v1.9.0 (9ee4e98d): binary files in Box, SharePoint and OneDrive, Gmail binary attachments, artifact references and the corpus layout, and model-facing text that no longer names the simulator. The five example rows are emitted again by 1.9.0. Their system prompt now reads "It is now <time>" instead of "The session starts at <time>, and time passes as you work"; their TaskSpecs and tool lists are unchanged. The rollouts are re-collected on this branch with azure/openai/gpt-5.5, all five scoring 1.0, and example_metrics.json is regenerated: only the word counts move, by the shorter prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
artifacts_dir was the internal branches' first setting, a single store, which they keep as an alias for older overrides once artifacts_dirs replaced it. Nothing upstream names it, and the launchers that pass stores set artifacts_dirs whenever the server has it, so the server takes the list only: stores are searched in order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
Gym signs session cookies and MCP session tokens with get_session_middleware_key(), a public value derived from the server's class and instance name. A resources server whose routes a sandbox can reach needs signatures that code in the sandbox cannot forge. Add SimpleServer.get_session_secret(): the key for SessionMiddleware, for the session router that reads the same cookie, and for MCP session tokens. It defaults to get_session_middleware_key(), so every existing server signs as before. The cookie name and the key shown in logs, error text and profiling paths stay the public one. Tests: with two workers, a server that returns its own secret routes its cookies and MCP tokens to the worker that holds the session, and a cookie or token signed with the server's public name names no session. Both tests fail when signing uses the public key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
The session cookie is named get_session_middleware_key(), the server's class and instance name. Callers see that name in every answer that sets the cookie, including code in a sandbox a resources server lends, where a model can read it. Add SimpleServer.get_session_cookie_name(): the name SessionMiddleware sets and the session router reads. It defaults to get_session_middleware_key(), so every existing server keeps its cookie; the key in logs, error text and profiling paths is unchanged. The name must stay unique per server, since a client keeps one cookie per name and host. Tests: with two workers, a server that names its cookie sets only that cookie, and the router routes it to the worker that holds the session; run_webserver hands the router the server's cookie name and secret. Both fail when the cookie keeps the middleware key's name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
A resources server's MCP endpoint lists itself under the server's name and
hands each session its token in X-NeMo-Gym-Session-Token, tools/list
answers a caller without a token with every tool, and its errors name the
framework and the tool's route. When the client runs where the model can
read its configuration, as a harness in a sandbox the server lends does,
all of that names the framework, and tools/list shows the model the tools
of every task.
Two opt-in options in BaseResourcesServerConfig, both off by default:
- mcp_server_name: the name /seed_session's MCP metadata gives the
endpoint, which clients name its tools by. Verify-time normalization
strips mcp__<that name>__.
- mcp_bearer_token: the token goes out as a standard "Authorization:
Bearer" header. tools/list then requires it and lists only the session's
tools, a tool outside the session's grant is answered as an unknown
tool, and errors name no framework ("Missing session token
(Authorization header).", "Invalid session token.", "HTTP <status> from
<tool>"). The MCP endpoint, the session router (which sends MCP calls to
the worker that holds the session) and the checkpoint participant's
token claims read the token from there.
Without them, every answer, header and error is as before.
Tests: the endpoint with both options (a bearer token only, tools/list
refused without one or in Gym's header, a session's own tools, a tool
outside the grant answered as unknown, the errors' wording); the defaults
unchanged, errors included; verify stripping the advertised name's
namespace; with two workers, a bearer token routed to the worker that
holds the session; and checkpoint claims read from Authorization. The
routing, claims and verify tests fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
Add opt-in sandbox support to the OfficeEnv resources server. Without sandbox_provider nothing changes: the same routes, signing and behavior as before. - With sandbox_provider set, each Environment Server episode borrows a sandbox from Gym's providers (Docker, OpenSandbox), lent as SandboxAccess. A repeated seed lends the same sandbox; close, retiring the session and shutdown stop it, and a sandbox that fails to start leaves no world behind. Shutdown and retiring a session wait for a seed still starting a sandbox, and a seed after shutdown starts none (503). An episode's lock is kept only while a request holds or awaits it. - A per-episode file API moves Drive files, text or binary, byte for byte between the workspace and the simulated apps (OFFICEENV_API_URL, OFFICEENV_API_TOKEN, OFFICEENV.md in the workspace); uploads are capped at 20 MB while the body streams in. - Sandbox mode guards the server a sandbox can reach: seed, close, verify, the app catalog and the tool routes answer only this node and trusted_networks, and session cookies and MCP tokens are signed with a per-process secret through get_session_secret. - configs/officeenv_sandbox.yaml (Pi, Docker) and configs/officeenv_opensandbox.yaml (Pi, OpenSandbox), with sandbox/Dockerfile, prepare_rows.py, relay.py (a loopback relay for workstations) and probe_opensandbox.py; data/sandbox_pilot.jsonl holds OfficeEnv 1.9.0's five sandbox tasks (CSV and workbook tasks in Drive, with Gmail and Slack). The workbook task seeds a binary Drive file, which 1.7.0 refuses. - The server's venv gains nemo-gym[sandbox]. Tests use a fake sandbox provider: lending, reuse and release of sandboxes, failed starts and stops, shutdown and retiring during a seed, no lock left by a closed session, the session contract in sandbox mode, the file API, the caller check, tokens signed with the server's name refused, plain mode unchanged for any caller, and the sandbox rows seeding on the pinned OfficeEnv. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
/verify graded while the episode's sandbox still ran; the sandbox stopped only at /close_session, after grading. A process the harness left running in the background could still call the episode's tools or its file API, so the reward could depend on timing. The training route has its agent stop the sandbox before verification; here the server owns the sandbox, so /verify does it. In sandbox mode /verify now closes the episode's tools (a tool call is refused with 409) and revokes its file API token, stops its sandbox, and then grades. A failed stop is logged and grading goes on; the episode keeps the sandbox, so /close_session stops it, and close stays idempotent. A file API upload the token admitted before verification began, and still streaming in, is refused (409) once its body has arrived, so it changes nothing. Plain mode is unchanged. Tests: the episode's tools close and its sandbox stops while the world is still ungraded; while the stop is awaited, a tool call is refused with 409, and so is an upload still streaming in; afterwards the file API and the MCP tools refuse; a close after verification stops nothing twice; and a failed stop still grades and leaves the stop to close. The first two fail without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
A sandbox lent to an episode had the internet, and models used it: in a training smoke they searched the web and called the real HubSpot, Calendly and Gmail APIs. sandbox_config.block_internet, first built for the training route, ports here. With block_internet the server sets OpenSandbox's egress policy as provider_options.network_policy, built with Gym's restricted_network_policy: every outbound connection is refused except to this server (the MCP endpoint and the file API) and the model servers, read from Gym's config; model servers on loopback are left out. It fails closed: another provider, a server address a sandbox cannot reach, host_relay, or a network_policy of the config's own are refused. Gym's sandboxed agents restrict only sandboxes they start, so for a sandbox the resources server owns, the server sets the policy. Off by default and in both configs: their sandboxes download at start (Pi's runtime, and pip in the OpenSandbox setup), so turning it on needs an image with those installed. A probe on an OpenSandbox cell (2026-10-27) found the policy enforced on addresses: public names did not resolve, bare public IPs and DNS over HTTPS timed out, and the node answered. Tests: the policy lists this server and the reachable model servers, and Gym's OpenSandbox provider options and the SDK's NetworkPolicy accept it; Docker, an unreachable server, host_relay and a second policy are refused before any sandbox starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
… conversation The file API moves files between the workspace and Drive without the conversation, but Drive's own tools still answered with a file's whole content (a workbook as base64) and took pasted base64, so a model could pull a workbook into its context, where it costs tokens on every later request, or copy file content by hand and get it wrong. In an episode with a sandbox, now: - download_file, export_file and get_revision save the file in the sandbox at /workspace/drive/<file id>/<name> (a revision under revisions/<version>/, an export with its type's extension), mode 644 whatever mode the provider's upload writes, and answer with savedTo and savedBytes. A folder at the file's path, or a file where its folder goes, fails the call (502) rather than taking the copy inside it. A save the provider fails also fails the call, without the provider's error text, which the server logs, and masks the episode's verification (mask_sample, failure_kind provider_unavailable): its reward would measure the sandbox, not the policy. A call holds the episode's lock until its files are written, and /verify stops the sandbox under that lock, so it waits for a call under way. The content stays in the answer only for a text file of at most sandbox_config.inline_file_bytes bytes (8192); a longer text gets its first 2,000 characters as preview; - create_file and update_file refuse binary content, and text over the limit, pasted into the call (upload_from_workspace), with the file API command that sends the file instead; - the five tools are described as they behave there (the app label kept) instead of in OfficeEnv's words (contentBase64, content_base64), and OFFICEENV.md says the same; /verify reports sandbox_files_saved, sandbox_bytes_saved and sandbox_uploads_refused. The file API, short text, calls without content, episodes without a sandbox and plain rows are unchanged. The server wraps the episode's OfficeEnv session exposure (ContentInWorkspace), so OfficeEnv is unchanged; inline_file_bytes: null restores the full answers. Tests (fake provider): downloads saved byte for byte with their paths, previews and modes; pasted uploads refused with the command; descriptions and the guide; no limit; a download the sandbox cannot take (502, without the provider's words, the verification masked), one whose path holds a folder or whose folder cannot be made (502, graded as usual), a mode it cannot set (answered); /verify waiting for a call still saving its download, which without the lock met a released sandbox (502); no sandbox; path segments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…behind it Everything a sandboxed model could read of this server named the environment or Gym: OFFICEENV_* variables, /officeenv/* routes, OFFICEENV.md, 403s saying a route answers "Gym's servers only", the generated docs (/openapi.json lists every route), errors that spoke of episodes and verification, a session cookie named after the server's class, and an MCP endpoint listed under the instance name, with an X-NeMo-Gym-Session-Token header and a tools/list that answered anyone with every tool. Models read them: in training smokes about half the rollouts fetched the server's /openapi.json. Now a sandbox reads a company's API: - COMPANY_API_URL and COMPANY_API_TOKEN, the file API at /v1/files, the workspace's README.md, and the workstation relay at a neutral path; - callers other than this node and trusted_networks get 404 from the routes they may not use, and sandbox mode serves no generated docs; - the file API's and the tool route's errors are neutral, and the session cookie is sid_<hash> in sandbox mode (get_session_cookie_name); - configs/officeenv_sandbox.yaml sets mcp_server_name: company and mcp_bearer_token: true: Pi lists the endpoint as company and names its tools company_<tool> (the longest fits Pi's 64 characters), and the token goes as a standard bearer token, which tools/list requires; - the rows' system prompt names the file API and the tools the same way (data/sandbox_pilot.jsonl emitted again with OfficeEnv 1.9.0; only the system prompt changed); - configs/officeenv_opensandbox.yaml has OpenSandbox's post-create probe print "ready" (sandbox.opensandbox.probe): the exec daemon keeps each command's output in the sandbox, and the provider's default marker names Gym. What a sandbox still reads of Gym comes from outside this server, and the README lists it: the policy's URL, which Gym's agents hand the harness with Gym's path names (/ng-rollout/<id> from rollout correlation, /training-token-capture with token capture) and Pi writes into its models.json; and Pi's own directories, NEMO_GYM_PI_* variables, provider name and process names. Making the two path names configurable is a further core change. Tests: one renders everything a sandbox reads (its variables, files and commands, the MCP endpoint's name, header, tools and answers, and every route it reaches, right and wrong, with the cookies they set) and fails on the environment's names; it fails with the old cookie name or the old 403, finds Gym's header without the MCP settings, and lists the model URL's path names among its findings. Also: the rows' prompts, tool names within Pi's limit under the configured server name, 404s and no docs for a sandbox, the cookie's name, and the OpenSandbox probe the provider runs under the composed settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…f the conversation OfficeEnv 1.9.0 keeps binary files in Box, SharePoint and OneDrive, and their tools answer a file's content to the model (Box as content_base64, the others as contentBase64) and take pasted base64, as Drive's did. In an episode with a sandbox, their content now moves through the workspace as Drive's does: - downloads (Box's get_file_content, SharePoint's get_item_content, OneDrive's get_content and get_version_content) are saved at <app>/<file id>/<name>, a Box or OneDrive version under versions/<version>/, mode 644, and answer with savedTo and savedBytes; text stays inline up to inline_file_bytes, and longer text gets a preview; - uploads (Box's upload_file and upload_file_version, SharePoint's upload_content, OneDrive's upload_file) refuse pasted binary, and text over the limit, with the file API command for the app: PUT for the file the call names, else POST into its folder, which a SharePoint or OneDrive path is resolved to; - the file API serves the four apps through their own ops, with files and folders named by id; the server reads what an op also wants (SharePoint's library, folder and name, a OneDrive item's drive) from the world. An app's error comes back under its own HTTP status, which the episode's exposure records during the file API's calls: Box's and Microsoft Graph's error bodies carry no numeric code. That wrapper is installed for every sandboxed episode, and with inline_file_bytes null it only records the statuses; - the tools are described as they behave there, the workspace guide covers the episode's own file apps, and the metrics count all four. Tests: per app, downloads saved with their paths, previews and modes, workbooks byte for byte, and Box and OneDrive versions (OneDrive's by path too); pasted uploads refused with each app's command; the file API both ways, binary included, and the apps' HTTP statuses (404, Box's 409, SharePoint's missing folder); the descriptions and the guide; and the test of what a sandbox reads covers the three apps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…conversation
OfficeEnv 1.9.0 keeps Gmail's binary attachments: get_attachment answers
contentBase64, and send_message, create_draft and update_draft take
attachments as {filename, content | content_base64} and refuse one that
names a file without its content. Gmail has no file API, so a model
mailing a workbook from a sandbox had to paste its base64 into the call,
and reading a binary attachment put the base64 into the conversation.
The four file apps already keep their content in the workspace; Gmail
now does too, with inline_file_bytes set:
- get_attachment saves the attachment at
gmail/<message id>/<attachment id>/<name> in the workspace, mode 644,
and answers with savedTo and savedBytes, as the file apps' downloads
do (short text inline as well, longer text with a preview);
- an attachment of send_message, create_draft or update_draft may name a
workspace file by its path ({"path": ..., "filename"?, "mimeType"?}; a
relative path is in the workspace, and nothing under /dev, /proc or
/sys is read), and so may a bare string that writes a path or names a
file the workspace holds (--attachments=/workspace/totals.xlsx on a
command line). The tool route reads the file from the sandbox before
the call runs (ContentInWorkspace.attachment_paths parses the call,
without effects, to find the paths), and the call carries it: UTF-8
text as content, other bytes as content_base64. Each path is checked
in the sandbox first (stat, which does not follow a final link): only
a regular file of at most 20 MB is downloaded, and nothing when a
message names more than 20 files or more than 25 MB together. The copy
is checked again, since docker cp copies a link as a link, which the
server would otherwise read on its own host, and a download that fails
attaches nothing. A path that holds no such file, or whose file is
gone since the check, is a 404, as the model left it; any other failed
download, or a check that does not run, is the provider's: the call
fails (502) and the episode's verification is masked
(provider_unavailable). A call holds the episode's lock while it reads
its files and runs, so /verify waits for it, and a call whose
verification begins while its files are read is refused, as any call
from that point is;
- pasted binary content, or text over the limit, is refused
(upload_from_workspace), and so is an attachment that names a file
without its content (attach_from_workspace), each with the attachment
that names the workspace file instead (a path already in the workspace
is named once);
- the attachments parameter is described that way in the MCP schema,
get_attachment's description says where the file goes, the workspace
guide has a paragraph for episodes that connect Gmail, and /verify
adds sandbox_files_attached and sandbox_bytes_attached.
Without inline_file_bytes, attachments move as before (no path form).
The README also drops a stale line that said the file API serves Google
Drive only.
Tests: an attachment saved byte for byte, with its preview and mode; a
message and a draft attaching workspace files by path, sent and read
back byte for byte; a bare path and a bare name of a workspace file
attached; pasted, unnamed and missing attachments refused with the path
to use, a path already in the workspace named once; links, folders,
devices (through a linked folder too) and oversized files refused before
anything is copied, and so are more files or bytes than a message takes;
a copy cut short, or a check that cannot run, failing the call and
masking the verification; a file gone since the check refused as missing
and graded as usual; nothing under /dev or /proc checked or copied; a
call refused when verification begins while it reads its files; the
descriptions and the guide; the behavior without the limit. The test of
what a sandbox reads covers Gmail's attachments and lets Gmail's own
vocabulary pass by name (its sendAs.verify op, "verified" send-as
addresses). 109 server tests pass on OfficeEnv 1.9.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…nt through the workspace, mail by path The sandbox rows' system prompt (sandbox/prepare_rows.py) still told the model to move file content between the workspace and Google Drive with the file API. The file API serves Box, SharePoint and OneDrive too, and Gmail now attaches workspace files by their path, so the prompt says: "Move file content between the workspace and the apps with the file API described in /workspace/README.md (...), and attach workspace files to email by their path, rather than pasting file content into tool calls; do data work with code." It names no app: the workspace guide names the episode's own. data/sandbox_pilot.jsonl is emitted again with the pinned 1.9.0; only the five rows' system message changes. The row-prompt test checks the new sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…file goes by the file API In a sandboxed episode the file apps' uploads refuse binary content pasted into the call, and their descriptions say so, but their content and content_base64 parameters kept OfficeEnv's words: "the file's bytes, base64-encoded, for a binary file such as an XLSX workbook, a PDF or an image". A model reading the schema base64-encoded a Word document in its shell, 50,000 characters into its context, before it sent the file with the file API. Both parameters of the six uploads now say what they take here: content takes text of at most inline_file_bytes, and content_base64 nothing, since any content it carries is refused: a binary file goes from the workspace with the file API, never pasted. Tests: the six uploads' descriptions, and a call pasting base64, even of short text, refused with the file API command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari
force-pushed
the
dmolinari/officeenv-sandbox-documents
branch
from
October 31, 2026 03:38
c4fbfde to
b5324ce
Compare
d-molinari
force-pushed
the
dmolinari/officeenv-sandbox-surfaces
branch
from
October 31, 2026 03:38
3396629 to
96abc0e
Compare
…oints, not only as MCP tools A sandboxed episode reached its apps only through one MCP tool per op, so only a harness that takes MCP tool grants in a sandbox session could run OfficeEnv's tasks. Now the row's exposure decides how the harness reaches the apps (SANDBOX_SURFACES); the world, the checks and the reward are the same for all three: - tools: one MCP tool per op, as before; - cli: one command per app in the sandbox (`<app> <command> [--parameter=value ...]`, `<app> --help`), read as OfficeEnv's CLI exposure reads a command line. The seed writes sandbox/app_command.sh (POSIX sh and curl, its comments stripped) to /usr/local/lib/company-cli/app, makes it executable and links it as /usr/local/bin/<app> for each connected app before the sandbox is lent. A command sends its arguments, NUL-separated, to POST /v1/cli with the episode's token and prints the result on stdout (exit 0) or the app's error on stderr (exit 1). The MCP grant is empty, so any harness with a shell and curl runs these rows with resources_tool_transports: []. A call to /v1/cli without the command's <app>-cli/1.0 User-Agent is refused with a pointer to the command and counted; - api_search: api_search and api_fetch over MCP (the api_search exposure's argument shapes) find and document endpoints, with each endpoint's HTTP path and the curl command that calls it, and the harness calls POST /v1/<app>/<op> with the op's parameters as a JSON object. An endpoint answers with the app's JSON, or its error under the error's HTTP status. The route is a Starlette one: Gym's MCP exposure takes FastAPI's parameterized POST routes for the tool catch-all and allows one. The file API reaches the file apps whatever the surface (FileApiCalls: a cli episode's only tool is run_command and an api_search episode's are the finding tools and api_call, so it reached no app there). File content and Gmail's attachments stay out of the conversation on every surface, and a command's --help and api_fetch's documentation describe the ops that move content as they behave here, Gmail's examples attaching a workspace file by its path (a name already in the workspace named once). Only --help is rewritten, whole lines keyed by app, op and parameter: an op's answer is the app's data. The workspace's README.md is written for the episode's surface, with sandbox_config.mcp_tool_prefix naming the MCP tools as the harness lists them (company_ in the Pi config). The two surface routes take the episode's bearer token, answer other surfaces' episodes as routes that do not exist, and run a call as the tools do: under the episode's lock, so /verify waits for a call under way, refused once verification has begun, and a save or attachment the provider fails answered 502 with the verification masked. An endpoint that fails answers without its exception, which the server logs. /verify adds sandbox_commands, sandbox_commands_refused and sandbox_endpoint_calls. Rows in any other exposure, or with another tool-name separator, are still refused before a sandbox starts. Tests: a cli episode's commands, their setup, answers, exit codes and refusals; an api_search episode's finding tools, endpoints and their statuses; the app command as a sandbox gets it, run by sh against a server on a socket next to the guide's curl commands; the file API on every surface; file content and Gmail's attachments through commands and endpoints, a bare string naming a workspace file included (--attachments=/workspace/out/totals.xlsx), and how --help and api_fetch describe them, help alone rewritten and whole; the examples' attachment paths; the guide on each surface; the 409 once verification begins; /verify waiting for a command or an endpoint saving its download; a save the provider fails answered 502 on both; a failed setup; an endpoint that fails. The test of what a sandbox reads runs on all three surfaces and reads each app's --help too, which names Gmail's own send-as verification; it lets that vocabulary pass by name. 147 server tests pass on OfficeEnv 1.9.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…o, for shells without its environment The apps' commands, the file API's curl commands and the endpoints all take COMPANY_API_URL and COMPANY_API_TOKEN from the sandbox's environment, but a harness need not pass it to its shell: Codex starts its shell with PATH, LANG, LC_ALL, HOME and the XDG folders alone, so on a cli row every command it ran would have called no server. The seed now also writes the two values to /etc/company-api.env (CREDENTIALS_PATH), which a shell sources (`. /etc/company-api.env` sets and exports them); the apps' commands read it when the variables are unset, and the workspace guide says so on every surface. The file names the company's API, as the variables do; the token in it is the one the sandbox's environment already holds. Tests: the file the seed writes; a command run by sh with no API variables in its environment, against a server on a socket; a shell that sources the file; the command and every surface's guide naming it. 147 server tests pass on OfficeEnv 1.9.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
… in every exposure With the apps reachable as commands, any of Gym's harnesses that borrows an episode's sandbox and runs a shell can run OfficeEnv's sandbox tasks: it takes the agent's config, the Environment Server's agent_server.name, and no MCP transports. This adds two, and the rows: - configs/officeenv_sandbox_episodes.yaml holds the resources server for sandboxed episodes alone; configs/officeenv_sandbox.yaml composes it with Pi as before (the merged config is the same, Pi's mcp_tool_prefix aside); - configs/officeenv_sandbox_codex.yaml (Codex CLI) and configs/officeenv_sandbox_miniswe.yaml (mini-SWE) compose it with their harness on rows whose apps are commands, resources_tool_transports: []. Their comments say what each installs in the sandbox at session start, and for Codex how to run tools rows (one tool name is then over 64 characters); - sandbox/prepare_rows.py --exposure cli|api_search writes the sandbox tasks in that exposure, with a prompt that says how the apps are reached and how to complete the task there, to data/sandbox_pilot_<exposure>_prepare.jsonl (gitignored); --tool-prefix names MCP tools as the harness lists them (company_ for Pi, the default; mcp__company__ for Codex). Every surface's prompt says where the file API's URL and token are, now with the file a shell without the API's variables sources first (`. /etc/company-api.env`): Codex's shell has none of them, and its first curl failed before it found the file. data/sandbox_pilot.jsonl is emitted again; only that sentence changes. The README says how to run a commands' row with Codex, mini-SWE or Pi. Tests: each harness config binds its harness to the shared episodes and starts no other; the prompts of every surface name the company's API, the file for a shell without its variables, and nothing of the environment; the prepared cli and api_search rows seed as sandboxed episodes on their surface, with its grant and guide. 151 server tests pass on OfficeEnv 1.9.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
… that need them A task that writes or reads documents in its sandbox needs more than the shell, curl and Python with pandas and openpyxl, and a sandbox without the internet installs nothing at run time. A row may now say what its task needs (sandbox_needs: documents, office), and the server gives its episode the first configured image that provides it all: sandbox_config.image with its provides, then sandbox_config.images. A row that no configured image serves is refused at seed, before its world is built (400 unsupported_needs), as is one that names a tool set no sandbox has. sandbox/Dockerfile.documents adds both sets to any episode image (the pilot image by default, or a training image with its harness staged): python-docx, python-pptx, pypdf, pdfplumber and reportlab, pinned; Poppler's command line tools; LibreOffice without a display, for conversions and formula recalculation; the DejaVu and Liberation fonts. The workspace's README.md lists what the episode's image has, and with block_internet says nothing else can be installed. prepare_rows.py copies an example task's sandbox_needs to its row, and task_data.py documents the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
…its extension A sandboxed episode saves a Docs editors file's export in the workspace with its type's extension (Travel policy.pdf). OfficeEnv 1.9.0 exports as text and PDF; its next release adds Word, Excel and PowerPoint files, which keep their extension here too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
A harness that runs in the sandbox installs its runtime there at session start (Pi and Codex fetch Node and their package), which a sandbox without the internet cannot do, so a training mix needs each harness's runtime in its image. sandbox/Dockerfile.harness builds one image per harness on the documents image: each target runs the adapter's own installer as the adapter runs it (sandbox/harness_runtime.sh), so the runtime is where the adapter looks for it at session start and the adapter downloads nothing. Targets: pi, codex, openclaw, opencode and terminus-2 (tmux; its agent runs on Gym's host). The same script, run in a built image without a network, checks it: the installer must find the runtime and the harness must start. A test fails when an adapter's configured version or runtime folder moves away from the script's. An image keeps only the runtime: once the installer and the harness have run, the script removes what they leave behind, most of it naming the runtime folder: the archives the install unpacked, npm's cache and debug logs (in ~/.npm, or in the runtime folder for an installer that moves HOME, as Codex's and OpenClaw's do), Node's compile cache in /tmp, and the helper links Codex makes in ~/.codex. A second test fails when an adapter's installer puts npm's cache where the script does not remove it. The README lists the names a sandbox still reads of Gym, the system prompts the harnesses send among them: Pi's, Codex's, OpenClaw's and OpenCode's name their documentation, skills or instructions in the adapters' folders, and OpenCode's gives the model's ID as nemo_gym/dummy_model; Terminus 2's names none. Hermes, Stirrup and Claude Code also reuse a runtime they find, but name its folder after their agent server's own install, and mini-SWE and Kilo Code install at every session start, so they have no target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
Each composes OfficeEnv's sandboxed episodes with its harness, as the Codex CLI and mini-SWE configs do, on rows whose apps are commands (cli) with no MCP transport: OpenCode refuses an episode's required MCP tools in a sandbox, and OpenClaw and Terminus 2 take none. OpenCode and OpenClaw run in the episode's sandbox; Terminus 2 runs its agent on Gym's host and types into a tmux session there. On their images of sandbox/Dockerfile.harness, with gpt-5.5 and one repeat, each scored 9 of 9 on OfficeEnv's nine sandbox tasks in the cli exposure: the five released in 1.9.0 and four unreleased ones on files in Box, SharePoint, OneDrive and a Gmail attachment, served and graded from OfficeEnv's source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
d-molinari
force-pushed
the
dmolinari/officeenv-sandbox-surfaces
branch
from
October 31, 2026 04:03
96abc0e to
0352520
Compare
d-molinari
force-pushed
the
dmolinari/officeenv-sandbox-documents
branch
from
October 31, 2026 04:03
b5324ce to
6f6ae28
Compare
Contributor
Author
|
Folded into #5010, which now carries these commits; please review them there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Gives OfficeEnv's sandboxed tasks that write or read documents a sandbox with the tools for it, and lets a row say which tools it needs. Stacked on #5373 (the
cliandapi_searchsurfaces), itself stacked on #5010. Withoutsandbox_needsin the rows and with the new settings unset, the server behaves as before.Four commits, all in
resources_servers/officeenv:feat(officeenv): an image with document tools, for the sandboxed rows that need them:sandbox_needs:documents(python-docx, python-pptx, pypdf, pdfplumber and reportlab; Poppler'spdftotext,pdfinfoandpdftoppm) andoffice(LibreOffice without a display, for conversions and formula recalculation).task_data.pydocuments the field.sandbox_config.providessays whatsandbox_config.imagehas, andsandbox_config.imageslists more images with what each provides. A row's episode gets the first image that provides all it needs,imagefirst, so a training mix can keep the small image for the rows that need nothing more. A row that names an unknown set, or that no configured image serves, is refused at seed before its world is built (400unsupported_needs).sandbox/Dockerfile.documentsadds both sets to an episode image. ItsBASEissandbox/Dockerfile's image by default, or any image with that image's tools, such as a training image with a harness staged in it. The Python packages are pinned (python-docx 1.2.0, python-pptx 1.0.2, pypdf 6.21.4, pdfplumber 0.11.10, reportlab 5.1.0); LibreOffice 7.4 (Writer, Calc and Impress, without a display) and Poppler 22.12 come from Debian 12, with the DejaVu and Liberation fonts (Liberation is metric-compatible with Arial, Times New Roman and Courier New). On the pilot image it is 1.16 GB, about 700 MB more.README.mdlists what the episode's image has, and withblock_internetsays that nothing else can be installed, so a model does not reach forpip installin a sandbox without the internet.sandbox/prepare_rows.pycopies an example task'ssandbox_needsto its row.feat(officeenv): a Drive export to DOCX, XLSX or PPTX is saved under its extension. A sandboxed episode saves a Docs editors file's export in the workspace under its type's extension. OfficeEnv 1.9.0 exports as text and PDF; its next release adds Word, Excel and PowerPoint files.feat(officeenv): one sandbox image per harness, on the documents image. A harness that runs in the sandbox installs its runtime there at session start (Pi and Codex fetch Node and their package), which a sandbox without the internet cannot do, so a training mix needs each harness's runtime in its image.sandbox/Dockerfile.harnessbuilds one image per harness on the documents image: each target runs the adapter's own installer as the adapter runs it (sandbox/harness_runtime.sh), so the runtime is where the adapter looks for it at session start and the adapter downloads nothing. Targets:pi,codex,openclaw,opencodeandterminus-2(tmux; its agent runs on Gym's host). An image keeps only the runtime: once the installer and the harness have run, the script removes what they leave behind, most of it naming the runtime folder: the archives the install unpacked (about 54 MB each), npm's cache and debug logs (in~/.npm, or in the runtime folder for an installer that movesHOME, as Codex's and OpenClaw's do), Node's compile cache in/tmp, and the helper links Codex makes in~/.codex. The same script, run in a built image without a network, is the check. A test fails when an adapter's configured version or runtime folder moves away from the script's, and another when an adapter's installer puts npm's cache where the script does not remove it. Hermes, Stirrup and Claude Code also reuse a runtime they find, but name its folder after their agent server's own install, and mini-SWE and Kilo Code install at every session start, so they have no target.feat(officeenv): sandbox configs for OpenCode, OpenClaw and Terminus 2. Each composes the sandboxed episodes with its harness, as the Codex CLI and mini-SWE configs do, on rows whose apps are commands (cli) with no MCP transport: OpenCode refuses an episode's required MCP tools in a sandbox, and OpenClaw and Terminus 2 take none. OpenCode and OpenClaw run in the episode's sandbox; Terminus 2 runs its agent on Gym's host and types into a tmux session there.Not in this PR
contentBytes,add_attachment), so that waits for it.Known gaps
/tmp/nemo-gym-pi-node-22.19.0-<version>,/tmp/nemo-gym-codex-node-22.19.0-<version>,/tmp/nemo-gym-openclaw-node-22.19.0-<version>,/tmp/nemo-gym-opencode-runtime-<version>and its.lock, and/tmp/nemo-gym-<harness>-sessions/<id>. The images keep the runtimes where the adapters look, and the README lists every one under what a sandbox still reads of Gym. Neutral names need the adapters to change./tmp/nemo-gym-pi-node-22.19.0-<version>), Codex's to five skills in/tmp/nemo-gym-codex-sessions/<id>, OpenClaw's to its documentation in/tmp/nemo-gym-openclaw-node-22.19.0-<version>, and OpenCode's to its instructions in/tmp/nemo-gym-opencode-sessions/<id>. OpenCode's also gives the model's ID asnemo_gym/dummy_model, the provider and model names its adapter configures. Terminus 2's names none. The README lists them; neutral names need the adapters to change.sandbox/Dockerfile) and the Python packages by version; the apt packages and the harnesses' npm installs are not pinned beyond their versions (no lockfiles). OpenCode's installer, the adapter's own on main, downloads its release archive without checking a hash, and the image build runs it as is: a pinned hash needs the adapter.Validation
macOS arm64 and Linux aarch64 (Colossus: the per-commit runs and
gym env test), Python 3.13.14, OfficeEnv 1.9.0 (thev1.9.0tag, as #5009 pins it), on mainfc02bfd7f, #5373's head0352520d5, head6f6ae289e:gym env test --resources-server officeenv +should_validate_data=true: 165 passed; the data is validated. Each commit passes the server's tests: 156, 157, 165 and 165. The branch merges cleanly into main402020a86, and the server's 165 tests pass on the merged tree.docker run --network none): python-docx, python-pptx and reportlab write a memo, a deck and a PDF; LibreOffice converts the DOCX and the PPTX to PDF and recalculates a workbook's formula (openpyxl reads its value only after the round trip);pdftotext, pdfplumber and pypdf read the PDFs back;pdfinfoandpdftoppmrun; both font families are installed.pre-commit run --all-files: all hooks pass, no changes (the guardwords hook skipped: its private catalog is unavailable here).detect-secrets-hook1.5.0 over the changed files: no findings.docker run --network none): each finds its runtime in 0 to 6 s and starts the harness (Pi 0.80.2, codex-cli 0.144.4, OpenClaw 2026.6.11, OpenCode 1.17.11, tmux 3.3a), against 5 to 53 s of downloads at build time. Sizes on amd64: the documents image 1.13 GB, the harness images 1.13 to 2.26 GB.b5324ce02, 2026-10-31: the documents image and the five harness images pass their offline checks, but that head's cleanup ran before the harness's own check and missed the cache folder an installer that movesHOMEuses. Codex's and OpenClaw's images still held npm's cache in their runtime folder (136 MB and 138 MB, with debug logs naming the folder 19 and 27 times), the Pi and Codex images Node's compile cache in/tmp, and Codex's image the helper links its check made in~/.codex.readymarkers and locks, and OpenClaw's empty/tmp/openclaw, which it makes when it runs; no file in them names Gym. The images are 1.85 GB and 1.84 GB (2.12 GB and 2.05 GB before). The Pi, OpenCode and Terminus 2 images and the amd64 images were not rebuilt at this head.Rollouts
Real model smoke, 2026-10-29: macOS with Docker (colima), Pi 0.80.2 on
toolsrows,azure/openai/gpt-5.5through the NVIDIA inference gateway,officeenv-sandbox:pilotasimageand the documents image inimages. The row is OfficeEnv'svendor_spend_xlsxworld with the request changed: a Word memo with a table of each vendor's Q3 paid total, a PDF copy beside it, and a reply naming the top three vendors. It needsdocumentsandoffice. OfficeEnv 1.9.0 reads neither format, so the checks find both files in the Finance folder by name and type.6 of 6 rollouts at 1.0, in two runs: two rollouts before feat(officeenv): run OfficeEnv episodes in a sandbox, for Gym's sandboxed harnesses #5010's last commit, four after. Each episode got the documents image (the models found
sofficeand python-docx, which the pilot image lacks). Each wrote the memo with python-docx; five converted it to PDF with LibreOffice and one wrote the PDF with reportlab; all uploaded both files with the file API and replied with the right three vendors. No file content passed through a tool's call or answer, and no sandbox was left running.Before feat(officeenv): run OfficeEnv episodes in a sandbox, for Gym's sandboxed harnesses #5010's last commit, which describes the uploads' content parameters as they behave in a sandbox, one rollout base64-encoded the 37 KB memo in its shell (49,560 characters into its context) before reading the guide and uploading with the file API, and another computed the base64 first. In the four rollouts after it, none did: median 14.5 calls and 168K input tokens.
On the harness images (before the installs' leftovers were removed), 2026-10-30, the same setup: 39 of 39 rollouts at 1.0, no failed rollout, no sandbox left running. Pi on
documents-pi, the document task, 2 of 2, and onamd64-documents-piunder emulation 1 of 1; Codex CLI, OpenCode, OpenClaw and Terminus 2 on their images, the nineclirows, 9 of 9 each. One OpenClaw rollout ranenvand saw OpenClaw's runtime onPATH(/tmp/nemo-gym-openclaw-node-…), the adapter's folder name, as feat(officeenv): reach a sandboxed episode's apps as commands or endpoints, for any harness with a shell #5373 lists for Pi's and Codex's.At
b5324ce02, 2026-10-31, on images built from scratch (arm64, the same model): 6 of 6 episodes at 1.0, with no command refused. Pi, Codex CLI, OpenClaw, OpenCode and Terminus 2 ran theclirow ofsandbox_timesheet_billing(3 of 3 checks each), and Pi the Word and PDF memo task (4 of 4). Their model calls showed the system prompts naming the adapters' folders (Known gaps). They also showed Codex spending a call on the API's variables, which its shell lacks, before it sourced/etc/company-api.env; feat(officeenv): reach a sandboxed episode's apps as commands or endpoints, for any harness with a shell #5373's row prompt now names that file.Compatibility
sandbox_needs, withprovidesandimagesunset: unchanged; every episode getsimage.block_internetepisodes: the guide adds that nothing else can be installed.sandbox_needsthat no configured image serves is refused at seed (400) rather than run without its tools.Checklist
resources_servers/officeenv/README.md).git commit -s).