Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions fern/versions/latest/pages/infrastructure/sandbox/opensandbox.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,67 @@ Unknown provider option keys raise `ValueError`; values with the wrong type rais

The provider also normalizes metadata values for backend labels by replacing unsupported characters and truncating values to the provider limit.

## Job Attribution

The provider merges `nemo-gym.nvidia.com/team`, `nemo-gym.nvidia.com/user`,
`nemo-gym.nvidia.com/workload`, and `nemo-gym.nvidia.com/run` keys into each sandbox's
metadata when they can be resolved. OpenSandbox propagates sandbox metadata as Kubernetes
labels on the sandbox resources, so running sandboxes are attributable both through the
OpenSandbox list API metadata filter and directly at the cluster level:

```bash
kubectl get pods -l nemo-gym.nvidia.com/team=my-team
```

Each field resolves in order: the provider's `attribution` config, then `NEMO_GYM_TEAM` /
`NEMO_GYM_USER` / `NEMO_GYM_WORKLOAD` environment variables, then Slurm job environment
variables (`SLURM_JOB_ACCOUNT` / `SLURM_JOB_USER` / `SLURM_JOB_NAME`). `user` additionally
falls back to the OS login name (`root` is ignored — containers run as root by default, so
it would attribute the image, not a person), and `workload` to `NEMO_GYM_CONFIG_PATH`, the
server instance name the gym CLI sets on every server process it spawns. Fields that cannot
be resolved are omitted. Explicit `sandbox_spec.metadata` and `default_metadata` keys always
take precedence over attribution keys.

`run` identifies one launch of the creating process: `NEMO_GYM_RUN_ID` if set, else an id
generated per process. The resolved attribution is logged once at the first sandbox create —
note the `run` value from the logs to list or clean up exactly that run's sandboxes (for
example after killing a run partway through), which `team` / `user` / `workload` cannot do
when the same user launches the same workload twice:

```bash
kubectl delete batchsandboxes -l nemo-gym.nvidia.com/run=<run-id>
```

```yaml
sandbox:
opensandbox:
attribution:
enabled: true # set false to disable automatic attribution
key_prefix: nemo-gym.nvidia.com/ # label-key namespace; "" for bare team/user/workload/run keys
# team: my-team # override auto-detection
# user: my-user
# workload: my-benchmark
# run: my-run-id
```

### Attribution in Kubernetes deployments

When the gym servers themselves run in Kubernetes (for example a driver pod launching an
eval), none of the automatic sources resolve: there are no Slurm variables, and the
container's OS login is typically `root`, which is ignored. Set the `NEMO_GYM_*` variables
on the pod spec — either directly or via the downward API from labels the pod already
carries:

```yaml
env:
- name: NEMO_GYM_TEAM
value: my-team
- name: NEMO_GYM_USER
valueFrom:
fieldRef:
fieldPath: metadata.labels['owner']
```

## Lifecycle

The provider creates one OpenSandbox sandbox per Gym sandbox:
Expand Down
150 changes: 150 additions & 0 deletions nemo_gym/sandbox/attribution.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Automatic job attribution (team / user / workload / run) for sandbox metadata.

Sandbox providers merge these keys into every sandbox's metadata so cluster
operators can attribute running sandboxes to the team, user, and workload that
created them (on OpenSandbox, metadata becomes queryable Kubernetes labels on
the sandbox). Each field resolves from explicit configuration first, then
``NEMO_GYM_*`` environment variables, then Slurm job environment variables,
then (for ``user`` only) the OS login name and (for ``workload``) the server
instance's config path. Fields that cannot be resolved are omitted rather than
guessed.

``run`` identifies one launch of the creating process so a run's sandboxes can
be listed or garbage-collected exactly, even when the same user runs the same
workload twice. It resolves from explicit configuration, then ``NEMO_GYM_RUN_ID``,
then a per-process generated id.
"""

import getpass
import logging
import os
import uuid
from collections.abc import Mapping


LOGGER = logging.getLogger(__name__)

TEAM_KEY = "team"
USER_KEY = "user"
WORKLOAD_KEY = "workload"
RUN_KEY = "run"

TEAM_ENV_VARS = ("NEMO_GYM_TEAM", "SLURM_JOB_ACCOUNT")
USER_ENV_VARS = ("NEMO_GYM_USER", "SLURM_JOB_USER")
# NEMO_GYM_CONFIG_PATH is set by the gym CLI on every server process it spawns
# (see nemo_gym.global_config.NEMO_GYM_CONFIG_PATH_ENV_VAR_NAME, not imported
# here to keep this module dependency-free) and names the server instance,
# which is the best automatic description of the workload.
WORKLOAD_ENV_VARS = ("NEMO_GYM_WORKLOAD", "SLURM_JOB_NAME", "NEMO_GYM_CONFIG_PATH")
RUN_ENV_VARS = ("NEMO_GYM_RUN_ID",)

# Container images default to running as root, so a "root" login attributes the
# image, not a person. Treat it as unresolved instead of emitting a false identity.
IGNORED_LOGIN_NAMES = frozenset({"root"})

_process_run_id: str | None = None
_logged_attribution = False


def _first_env(environ: Mapping[str, str], names: tuple[str, ...]) -> str | None:
for name in names:
value = (environ.get(name) or "").strip()
if value:
return value
return None


def _detect_user(environ: Mapping[str, str]) -> str | None:
value = _first_env(environ, USER_ENV_VARS)
if value:
return value
try:
login = getpass.getuser().strip()
except Exception:
# getpass raises on hosts with neither login env vars nor a passwd entry for the uid.
return None
if not login or login.lower() in IGNORED_LOGIN_NAMES:
return None
return login


def resolve_attribution(
*,
team: str | None = None,
user: str | None = None,
workload: str | None = None,
environ: Mapping[str, str] | None = None,
) -> dict[str, str]:
"""Resolve ``team`` / ``user`` / ``workload`` attribution metadata.

Args:
team: Explicit team; falls back to ``NEMO_GYM_TEAM``, then ``SLURM_JOB_ACCOUNT``.
user: Explicit user; falls back to ``NEMO_GYM_USER``, then ``SLURM_JOB_USER``,
then the OS login name (``root`` is ignored — containers run as root by
default, so it attributes the image, not a person).
workload: Explicit workload; falls back to ``NEMO_GYM_WORKLOAD``, then
``SLURM_JOB_NAME``, then ``NEMO_GYM_CONFIG_PATH`` (the server instance
name the gym CLI sets on every server process it spawns).
environ: Environment mapping override, for testing. Defaults to ``os.environ``.

Returns:
A dict with only the resolved keys among ``team``, ``user``, and ``workload``.
Values are returned raw; providers apply their own metadata sanitization.
"""
environ = os.environ if environ is None else environ
resolved: dict[str, str] = {}
team = team or _first_env(environ, TEAM_ENV_VARS)
if team:
resolved[TEAM_KEY] = str(team)
user = user or _detect_user(environ)
if user:
resolved[USER_KEY] = str(user)
workload = workload or _first_env(environ, WORKLOAD_ENV_VARS)
if workload:
resolved[WORKLOAD_KEY] = str(workload)
return resolved


def resolve_run_id(run: str | None = None, environ: Mapping[str, str] | None = None) -> str:
"""Resolve the ``run`` attribution id: explicit value, then ``NEMO_GYM_RUN_ID``,
then an id generated once per process.

Unlike :func:`resolve_attribution` fields, ``run`` is always resolvable. It scopes
sandboxes to one launch of the creating process, so an interrupted run's sandboxes
can be listed and cleaned up exactly (``team`` / ``user`` / ``workload`` cannot
distinguish two runs of the same workload by the same user).
"""
global _process_run_id
resolved = (run or "").strip() or _first_env(os.environ if environ is None else environ, RUN_ENV_VARS)
if resolved:
return resolved
if _process_run_id is None:
_process_run_id = uuid.uuid4().hex[:12]
return _process_run_id


def log_attribution_once(metadata: Mapping[str, str]) -> None:
"""Log the resolved attribution metadata once per process.

The generated ``run`` id only exists in this process, so surfacing it in the logs
is what lets operators later filter or garbage-collect this run's sandboxes.
"""
global _logged_attribution
if _logged_attribution or not metadata:
return
_logged_attribution = True
LOGGER.info("Sandbox attribution metadata: %s", dict(metadata))
2 changes: 2 additions & 0 deletions nemo_gym/sandbox/providers/opensandbox/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"""OpenSandbox provider package."""

from nemo_gym.sandbox.providers.opensandbox.provider import (
OpenSandboxAttributionConfig,
OpenSandboxConnectionConfig,
OpenSandboxCreateConfig,
OpenSandboxCreateError,
Expand All @@ -27,6 +28,7 @@


__all__ = [
"OpenSandboxAttributionConfig",
"OpenSandboxConnectionConfig",
"OpenSandboxCreateConfig",
"OpenSandboxCreateError",
Expand Down
20 changes: 20 additions & 0 deletions nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,23 @@ sandbox:
retry_max_delay_s: 45.0
command_retries: 0
close_timeout_s: 30
# Job attribution merged into every sandbox's metadata. OpenSandbox propagates
# metadata as Kubernetes labels on the sandbox resources, so sandboxes are
# attributable both via the OpenSandbox list API and at the cluster level, e.g.
# `kubectl get pods -l nemo-gym.nvidia.com/team=my-team`. Unset fields auto-detect
# from NEMO_GYM_TEAM / NEMO_GYM_USER / NEMO_GYM_WORKLOAD, then Slurm job env vars
# (SLURM_JOB_ACCOUNT / SLURM_JOB_USER / SLURM_JOB_NAME), then the OS login name for
# `user` ("root" is ignored) and the gym CLI's NEMO_GYM_CONFIG_PATH server instance
# name for `workload`; unresolvable fields are omitted. `run` scopes sandboxes to one
# launch of the creating process (NEMO_GYM_RUN_ID, else generated per process and
# logged) so an interrupted run's sandboxes can be listed and cleaned up exactly.
# Explicit sandbox_spec.metadata and default_metadata keys always win over
# attribution keys.
attribution:
enabled: true
# Kubernetes prefixed-key namespace for the injected label keys; "" for bare keys.
key_prefix: nemo-gym.nvidia.com/
# team: my-team
# user: my-user
# workload: my-benchmark
# run: my-run-id
69 changes: 68 additions & 1 deletion nemo_gym/sandbox/providers/opensandbox/provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
from pathlib import Path
from typing import Any, Awaitable, Callable

from nemo_gym.sandbox.attribution import RUN_KEY, log_attribution_once, resolve_attribution, resolve_run_id
from nemo_gym.sandbox.providers.base import (
SandboxCreateError,
SandboxCreateVerificationError,
Expand Down Expand Up @@ -90,6 +91,10 @@ class OpenSandboxCreateVerificationError(SandboxCreateVerificationError):
"timeout",
)
METADATA_VALUE_RE = re.compile(r"[^A-Za-z0-9_.-]+")
# Kubernetes prefixed-key namespace for auto-injected attribution labels (team/user/workload/run).
DEFAULT_ATTRIBUTION_KEY_PREFIX = "nemo-gym.nvidia.com/"
# Kubernetes label-key prefixes must be DNS-1123 subdomains (max 253 chars).
ATTRIBUTION_KEY_PREFIX_RE = re.compile(r"(?=.{1,253}$)[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*")
DEFAULT_IMAGE_PULL_POLICY = "IfNotPresent"
IMAGE_PULL_POLICY_EXTENSION_KEY = "imagePullPolicy"
IMAGE_PULL_POLICY_ANNOTATION_EXTENSION_KEY = "opensandbox.extensions.image-pull-policy"
Expand Down Expand Up @@ -352,6 +357,47 @@ class OpenSandboxConnectionConfig:
use_server_proxy: bool = False


@dataclass(frozen=True)
class OpenSandboxAttributionConfig:
"""Job attribution merged into every sandbox's metadata (Kubernetes labels on the sandbox).

OpenSandbox propagates sandbox metadata as Kubernetes labels on the sandbox resources, so
attribution is queryable both through the OpenSandbox list API and at the cluster level
(e.g. ``kubectl get pods -l nemo-gym.nvidia.com/team=my-team``). ``key_prefix`` namespaces
the label keys (Kubernetes prefixed-key convention); set it to ``""`` for bare
``team`` / ``user`` / ``workload`` / ``run`` keys.

Unset fields are auto-detected: ``NEMO_GYM_TEAM`` / ``NEMO_GYM_USER`` / ``NEMO_GYM_WORKLOAD``
environment variables first, then Slurm job env vars (``SLURM_JOB_ACCOUNT`` /
``SLURM_JOB_USER`` / ``SLURM_JOB_NAME``), then the OS login name for ``user`` (``root`` is
ignored) and the gym CLI's ``NEMO_GYM_CONFIG_PATH`` server instance name for ``workload``.
Fields that cannot be resolved are omitted. ``run`` scopes sandboxes to one launch of the
creating process (``NEMO_GYM_RUN_ID``, else generated per process and logged) so a run's
sandboxes can be listed and cleaned up exactly. Explicit ``SandboxSpec.metadata`` keys
always take precedence over attribution keys.
"""

enabled: bool = True
team: str | None = None
user: str | None = None
workload: str | None = None
run: str | None = None
key_prefix: str = DEFAULT_ATTRIBUTION_KEY_PREFIX

def __post_init__(self) -> None:
normalized = self.key_prefix.strip()
if normalized and not normalized.endswith("/"):
normalized += "/"
if normalized and not ATTRIBUTION_KEY_PREFIX_RE.fullmatch(normalized[:-1]):
# Invalid prefixes would otherwise fail server-side at sandbox create with an
# opaque error; values are sanitized by the provider but keys pass through.
raise ValueError(
f"attribution key_prefix {self.key_prefix!r} must be a lowercase DNS subdomain "
"(Kubernetes label-key prefix), e.g. 'nemo-gym.nvidia.com/', or '' for bare keys"
)
object.__setattr__(self, "key_prefix", normalized)


@dataclass(frozen=True)
class OpenSandboxCreateConfig:
"""OpenSandbox create/reconnect retry settings."""
Expand Down Expand Up @@ -499,11 +545,13 @@ def __init__(
create: OpenSandboxCreateConfig | Mapping[str, Any] | None = None,
probe: OpenSandboxProbeConfig | Mapping[str, Any] | None = None,
operations: OpenSandboxOperationConfig | Mapping[str, Any] | None = None,
attribution: OpenSandboxAttributionConfig | Mapping[str, Any] | None = None,
) -> None:
self._connection = _coerce_config(connection, OpenSandboxConnectionConfig)
self._create = _coerce_config(create, OpenSandboxCreateConfig)
self._probe = _coerce_config(probe, OpenSandboxProbeConfig)
self._operations = _coerce_config(operations, OpenSandboxOperationConfig)
self._attribution = _coerce_config(attribution, OpenSandboxAttributionConfig)

def _resolve_extensions(self, extensions: Mapping[str, str]) -> dict[str, str]:
"""Add the configured default image pull policy to SDK create extensions."""
Expand Down Expand Up @@ -842,8 +890,27 @@ async def _create_with_retries(

raise OpenSandboxCreateError("OpenSandbox create retry loop did not run")

def _attribution_metadata(self) -> dict[str, str]:
if not self._attribution.enabled:
return {}
resolved = resolve_attribution(
team=self._attribution.team,
user=self._attribution.user,
workload=self._attribution.workload,
)
resolved[RUN_KEY] = resolve_run_id(self._attribution.run)
prefix = self._attribution.key_prefix
metadata = {f"{prefix}{key}": value for key, value in resolved.items()}
log_attribution_once(metadata)
return metadata

async def create(self, spec: SandboxSpec) -> SandboxHandle:
"""Create one sandbox through the configured OpenSandbox path."""
"""Create one sandbox through the configured OpenSandbox path.

Job attribution keys (``team`` / ``user`` / ``workload`` / ``run``) are merged into the
spec's metadata (explicit spec keys win) so every sandbox is attributable via its labels.
"""
spec = replace(spec, metadata={**self._attribution_metadata(), **spec.metadata})
return await self._create_with_retries(_normalize_spec(spec))

async def status(self, handle: SandboxHandle) -> SandboxStatus:
Expand Down
Loading
Loading