Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions fern/versions/latest/pages/observability/metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,23 @@ How to read them:

A slot costs a few integer updates and no OpenTelemetry call. The exporter's background thread reads the totals, and they are reported whenever metrics are exported.

### Sandbox Metrics

These are recorded whenever metrics are exported, whatever the span groups. `AsyncSandbox` records them for every sandbox provider, except `gym.sandbox.create_retry_total`, which a provider that retries creates records from its own loop (Daytona and OpenSandbox do). Every metric carries `nemo.gym.sandbox.provider`, such as `docker` or `opensandbox`.

| Metric | Type | Unit | Meaning |
| --- | --- | --- | --- |
| `gym.sandbox.active` | Up/down counter | count | Sandboxes this process holds, from the provider handing back a handle until Gym releases it. |
| `gym.sandbox.startup_duration_ms` | Histogram | ms | Time to provision one sandbox. |
| `gym.sandbox.exec_duration_ms` | Histogram | ms | Time to run one command in a sandbox. |
| `gym.sandbox.create_retry_total` | Counter | count | Sandbox-create attempts a provider retried. |
| `gym.sandbox.transfer_duration_ms` | Histogram | ms | Time to move one file into or out of a sandbox, by `nemo.gym.sandbox.transfer.direction` (`upload` or `download`) and `nemo.gym.sandbox.transfer.outcome` (`ok`, `error`, or `cancelled`). |
| `gym.sandbox.transfer_bytes_total` | Counter | bytes | Bytes of regular files moved successfully, by `nemo.gym.sandbox.transfer.direction`. |

Transfers include `upload()`, `download()`, and the seed files in `SandboxSpec.files`, which `start()` uploads after the provider returns the sandbox, so seed uploads are not part of `gym.sandbox.startup_duration_ms`. Sidecar downloads are not counted. Transfer spans record the file size but not the paths, because a path can name task content.

A transfer that raises is an `error`, including a download that probes for a file that may not exist yet. A transfer cut short by cancellation is `cancelled`, including one the caller abandons when its own deadline (for example `asyncio.timeout`) expires; a timeout inside the provider is an `error`.

### Memory Metrics

Memory metrics are emitted only when `telemetry.memory_profiling.enabled` is true. Server points carry `nemo.gym.server.name`, `nemo.gym.server.type`, and the root `process.pid`; server memory gauges also carry `memory.type`, which is `rss` or `pss`. Host points rely on the existing `host.name` resource attribute and carry no per-sample attributes.
Expand Down
2 changes: 1 addition & 1 deletion fern/versions/latest/pages/observability/span-groups.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ gives every root the job's trace id, so both of those still produce one trace pe
| `agent` | `gym.agent.responses` | The agent server's `/v1/responses` endpoint |
| `model_call` | `gym.model.chat_completions`, `gym.model.responses`, `gym.model.messages` | Model servers |
| `verify` | `gym.verify` | Resources servers' `/verify` endpoint |
| `sandbox` | `gym.sandbox.start`, `gym.sandbox.exec` | Any sandbox provider |
| `sandbox` | `gym.sandbox.start`, `gym.sandbox.exec`, `gym.sandbox.upload`, `gym.sandbox.download`, `gym.sandbox.stop`, `gym.sandbox.request`, and the resource spans described in [OpenTelemetry](/main/observability/opentelemetry) | Any sandbox provider |
| `checkpoint` | `gym.checkpoint.<operation>`, `gym.checkpoint.coordinate.<operation>`, and their child spans | Servers taking part in partial-rollout checkpoints and the controller that coordinates them, only when checkpointing is on |

### Groups You Should Not Disable
Expand Down
87 changes: 83 additions & 4 deletions nemo_gym/sandbox/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

import asyncio
import inspect
import os
import stat
import tempfile
import threading
import time
Expand Down Expand Up @@ -58,11 +60,14 @@
)
from nemo_gym.telemetry._fallbacks import is_span_group_enabled, managed_span, safe_set_span_attributes
from nemo_gym.telemetry.gym_metrics import (
SandboxTransferDirection,
SandboxTransferOutcome,
record_sandbox_active,
record_sandbox_exec_duration,
record_sandbox_startup,
record_sandbox_transfer,
)
from nemo_gym.telemetry.setup import is_span_group_or_metrics_active
from nemo_gym.telemetry.setup import is_metrics_exporter_active, is_span_group_or_metrics_active
from nemo_gym.telemetry.span_groups import GymSpanGroup


Expand Down Expand Up @@ -403,6 +408,65 @@ def _sandbox_id(handle: Any) -> str | None:
return getattr(handle, "sandbox_id", None)


def _file_size(path: Path) -> int | None:
"""The size of a regular file, or None when it cannot be read, so telemetry never fails a transfer.

One ``stat`` call: on a shared filesystem such as Lustre, each one is a metadata server round trip.
"""
try:
result = os.stat(path)
except OSError:
return None
return result.st_size if stat.S_ISREG(result.st_mode) else None


async def _instrumented_transfer(
direction: SandboxTransferDirection,
transfer: Callable[[], Awaitable[None]],
*,
local_path: Path,
provider: str,
span_attributes: Mapping[str, Any],
) -> None:
"""Run one file transfer with a span, a duration sample and a byte count when spans or metrics are on.

``local_path`` is the host side of the transfer.
Its size is read once after the transfer succeeds, and only when a recording span or a metrics exporter uses it.
Paths are deliberately not recorded.
A path can name task content, the same way an exec command can.
"""
if not is_span_group_or_metrics_active(GymSpanGroup.SANDBOX):
await transfer()
return
started = time.perf_counter()
outcome: SandboxTransferOutcome = "error"
size_bytes: int | None = None
try:
with managed_span(
GymSpanGroup.SANDBOX,
f"gym.sandbox.{direction}",
**{"nemo.gym.sandbox.provider": provider, **default_span_attributes(), **span_attributes},
) as span:
await transfer()
recording = span is not None and span.is_recording()
if recording or is_metrics_exporter_active():
size_bytes = _file_size(local_path)
if recording and size_bytes is not None:
safe_set_span_attributes(span, {"nemo.gym.sandbox.transfer.size_bytes": size_bytes})
outcome = "ok"
except asyncio.CancelledError:
outcome = "cancelled"
raise
finally:
record_sandbox_transfer(
(time.perf_counter() - started) * 1000.0,
provider=provider,
direction=direction,
outcome=outcome,
size_bytes=size_bytes if outcome == "ok" else None,
)


def _missing_keywords(method: Callable[..., Any], keywords: Mapping[str, Any]) -> list[str]:
"""The names in ``keywords`` that ``method`` does not take, none when it takes them all or ``**kwargs``."""
try:
Expand Down Expand Up @@ -555,7 +619,7 @@ async def start(
for index, (target_path, contents) in enumerate(requested_spec.files.items()):
source_path = tmp_path / f"file-{index}"
source_path.write_text(contents, encoding="utf-8")
await self._provider.upload_file(handle, source_path, target_path)
await self.upload(source_path, target_path)
except BaseException:
await self.stop()
raise
Expand Down Expand Up @@ -701,10 +765,25 @@ def set_default_workdir(self, workdir: str | None) -> None:
self._spec = replace(self._spec, workdir=workdir) if self._spec is not None else SandboxSpec(workdir=workdir)

async def upload(self, local_path: Path | str, remote_path: str) -> None:
await self._provider.upload_file(self._require_handle(), Path(local_path), remote_path)
handle, local_path = self._require_handle(), Path(local_path)
await self._transfer("upload", lambda: self._provider.upload_file(handle, local_path, remote_path), local_path)

async def download(self, remote_path: str, local_path: Path | str) -> None:
await self._provider.download_file(self._require_handle(), remote_path, Path(local_path))
handle, local_path = self._require_handle(), Path(local_path)
await self._transfer(
"download", lambda: self._provider.download_file(handle, remote_path, local_path), local_path
)

async def _transfer(
self, direction: SandboxTransferDirection, transfer: Callable[[], Awaitable[None]], local_path: Path
) -> None:
await _instrumented_transfer(
direction,
transfer,
local_path=local_path,
provider=self._telemetry_provider_name(),
span_attributes={"nemo.gym.sandbox.id": _sandbox_id(self._handle), **self._resource_attributes},
)

def sidecar(self, name: str) -> "AsyncSandboxSidecar":
"""Return a sidecar container declared in ``SandboxSpec.sidecars``, to run commands in and read files from."""
Expand Down
68 changes: 66 additions & 2 deletions nemo_gym/telemetry/gym_metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,16 @@
``gym.sandbox.create_retry_total`` (counter): one per create attempt a provider retried.
Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry ``nemo.gym.sandbox.provider``.
``gym.sandbox.transfer_duration_ms`` (histogram): wall-clock of one file transfer into or out of a sandbox,
including the seed files ``AsyncSandbox.start()`` writes after the provider returns the sandbox.
It carries ``nemo.gym.sandbox.transfer.direction`` (``upload`` or ``download``)
and ``nemo.gym.sandbox.transfer.outcome`` (``ok``, ``error`` or ``cancelled``).
Sidecar downloads are not counted.

``gym.sandbox.transfer_bytes_total`` (counter, bytes): the size of each regular file a transfer moved successfully,
by direction.

All six carry ``nemo.gym.sandbox.provider``.

Server startup
--------------
Expand Down Expand Up @@ -150,7 +159,7 @@ class name). Labels come from the dispatch's original routing fields, before per
import logging
import threading
from collections.abc import Callable, Iterable, Sequence
from typing import Any, Optional
from typing import Any, Literal, Optional


logger = logging.getLogger(__name__)
Expand All @@ -160,6 +169,12 @@ class name). Labels come from the dispatch's original routing fields, before per
SANDBOX_STARTUP_INSTRUMENT = "gym.sandbox.startup_duration_ms"
SANDBOX_EXEC_INSTRUMENT = "gym.sandbox.exec_duration_ms"
SANDBOX_CREATE_RETRY_INSTRUMENT = "gym.sandbox.create_retry_total"
SANDBOX_TRANSFER_DURATION_INSTRUMENT = "gym.sandbox.transfer_duration_ms"
SANDBOX_TRANSFER_BYTES_INSTRUMENT = "gym.sandbox.transfer_bytes_total"
SANDBOX_TRANSFER_DIRECTION_ATTRIBUTE = "nemo.gym.sandbox.transfer.direction"
SANDBOX_TRANSFER_OUTCOME_ATTRIBUTE = "nemo.gym.sandbox.transfer.outcome"
SandboxTransferDirection = Literal["upload", "download"]
SandboxTransferOutcome = Literal["ok", "error", "cancelled"]
SERVER_STARTUP_STAGE_INSTRUMENT = "gym.server.startup_stage_duration_ms"
SERVER_NAME_ATTRIBUTE = "nemo.gym.server.name"
SERVER_TYPE_ATTRIBUTE = "nemo.gym.server.type"
Expand Down Expand Up @@ -210,6 +225,27 @@ class name). Labels come from the dispatch's original routing fields, before per
1_800_000,
)

#: Milliseconds.
#: Copying a small file into a local container takes tens of milliseconds, while a large file to a remote sandbox
#: can take minutes, so the boundaries start lower than the lifecycle ones.
SANDBOX_TRANSFER_DURATION_BOUNDARIES_MS: tuple[float, ...] = (
10,
25,
50,
100,
250,
500,
1_000,
2_500,
5_000,
10_000,
30_000,
60_000,
120_000,
300_000,
600_000,
)

#: Milliseconds.
#: A cold venv install can run for minutes and a local model load for longer, so the boundaries run to an hour.
#: Init stages of a warm server fall in the lowest buckets.
Expand Down Expand Up @@ -372,6 +408,34 @@ def record_sandbox_create_retry(*, provider: str) -> None:
)


def record_sandbox_transfer(
duration_ms: float,
*,
provider: str,
direction: SandboxTransferDirection,
outcome: SandboxTransferOutcome,
size_bytes: int | None,
) -> None:
"""Record one file transfer's wall-clock and, when it succeeded, the bytes it moved."""
attributes = {SANDBOX_PROVIDER_ATTRIBUTE: provider, SANDBOX_TRANSFER_DIRECTION_ATTRIBUTE: direction}
_record_histogram(
SANDBOX_TRANSFER_DURATION_INSTRUMENT,
"ms",
"Wall-clock time to move one file into or out of a sandbox.",
duration_ms,
{**attributes, SANDBOX_TRANSFER_OUTCOME_ATTRIBUTE: outcome},
boundaries=SANDBOX_TRANSFER_DURATION_BOUNDARIES_MS,
)
if size_bytes is not None:
_record_counter(
SANDBOX_TRANSFER_BYTES_INSTRUMENT,
"Bytes of files moved into or out of sandboxes.",
attributes,
size_bytes,
unit="By",
)


def _server_attributes(server_name: str, server_type: Optional[str]) -> dict[str, Any]:
attributes: dict[str, Any] = {SERVER_NAME_ATTRIBUTE: server_name}
if server_type:
Expand Down
39 changes: 39 additions & 0 deletions tests/unit_tests/telemetry/test_call_sites.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,13 @@ async def exec(self, handle, command, **kwargs):

return SandboxExecResult(stdout="out", stderr="", return_code=self._return_code)

async def upload_file(self, handle, source_path, target_path):
if target_path.endswith("/refused"):
raise RuntimeError("upload refused")

async def download_file(self, handle, source_path, target_path):
target_path.write_bytes(b"downloaded")

async def close(self, handle):
self.closed = True

Expand Down Expand Up @@ -132,6 +139,38 @@ async def test_sandbox_stop_emits_a_span_for_the_release(recorded_spans, started
assert stop_span.attributes["nemo.gym.sandbox.id"] == "handle-1"


async def test_sandbox_transfers_emit_spans_with_size_but_no_paths(recorded_spans, started_sandbox, tmp_path):
"""Paths can name task content, so a transfer span records only the provider, the sandbox and the size."""
sandbox = await started_sandbox().start()
local = tmp_path / "task-secret-name.png"
local.write_bytes(b"x" * 64)
await sandbox.upload(local, "/workspace/task-secret-name.png")
await sandbox.download("/workspace/out.txt", tmp_path / "out.txt")

spans = {span.name: span for span in recorded_spans()}
request_id = spans["gym.sandbox.start"].attributes["nemo.gym.sandbox.request.id"]
for name, size in (("gym.sandbox.upload", 64), ("gym.sandbox.download", len(b"downloaded"))):
assert spans[name].attributes["nemo.gym.sandbox.provider"] == "stub"
assert spans[name].attributes["nemo.gym.sandbox.request.id"] == request_id
assert spans[name].attributes["nemo.gym.sandbox.id"] == "handle-1"
assert spans[name].attributes["nemo.gym.sandbox.transfer.size_bytes"] == size
assert not any("task-secret-name" in str(value) for value in spans[name].attributes.values())


async def test_failed_sandbox_upload_marks_its_span_as_an_error(recorded_spans, started_sandbox, tmp_path):
from opentelemetry.trace import StatusCode

sandbox = await started_sandbox().start()
local = tmp_path / "file.bin"
local.write_bytes(b"payload")
with pytest.raises(RuntimeError, match="upload refused"):
await sandbox.upload(local, "/workspace/refused")

upload_span = next(span for span in recorded_spans() if span.name == "gym.sandbox.upload")
assert upload_span.status.status_code == StatusCode.ERROR
assert "nemo.gym.sandbox.transfer.size_bytes" not in upload_span.attributes


async def test_sandbox_spans_carry_the_rollout_id_inside_a_rollout(
recorded_spans, started_sandbox, rollout_attributes_registered
):
Expand Down
Loading
Loading