Skip to content

[feat] CPU performance-counter telemetry for sandboxed workloads #3305

Open

Description

@sephmard

Problem

Users want CPU performance-counter visibility into sandboxed workloads that Gym doesn't provide today. SandboxObservation defines cpu_time_s/peak_memory_mib fields, but there's no perf integration, cgroup metrics, or microarchitectural counters in the codebase yet to build on.

What's being asked for, based on Grace/Vera (and IA) performance work:

  • An initial, reasonable metric set is "Top-down L1" (perf stat -M TopdownL1 on Grace/Vera; IA has an equivalent metric, name may differ).
  • Counter/metric names need to be configurable rather than hardcoded, since they're platform-specific.
  • Collection should be scoped to the cores mapped to a single agent/sandbox (perf's -c arg), not the whole socket.
  • Time-series samples across the run (perf's -I arg) are more valuable than a single sample at the end.

What exists vs. what's needed

Capability Status
Tool-call start/end timestamps, duration ✅ Already collected — enough to detect idle-GPU periods
Sandbox cpu_time_s / peak_memory_mib fields ⚠️ Defined in schema, never populated
cgroup-level CPU utilization ⚠️ In flight, unmerged, single-harness only (#2980)
perf-based microarchitectural counters (Top-down L1) ❌ Does not exist
Per-sandbox core scoping for counter collection ❌ Does not exist
Time-series/interval sampling ❌ Does not exist — every metric today is a single end-of-run number
Platform-specific counter-name abstraction ❌ Does not exist

What this needs to answer

  • How to actually gather Top-down L1 (or platform-equivalent) counters, scoped to the cores a given sandbox owns, as a time series rather than a single end-of-run sample.
  • Counter/metric names are platform-specific (Grace/Vera vs. IA) — the mechanism needs to accommodate that rather than assume one fixed set.
  • Whether perf stat can run with the privileges available inside Gym's sandbox providers (Apptainer, Docker, OpenSandbox) — this needs checking per-provider, since perf_event_paranoid restrictions could rule out approaches before they're worth designing further.
  • How this relates to the core-pinning work already landing in feat(anyterminal): OpenSandbox oracle config, resource knobs, cgroup metrics, TB2.1 prepare #2980 — "which cores does this sandbox own" is a question that work is already answering for one harness; worth understanding before deciding whether to duplicate or share it.
  • Where the resulting data should live — Gym already has an OTel/nemo-lens pipe for other metrics (nemo_gym/telemetry/), though that infrastructure only provides a place to export numbers, not a way to collect CPU counters — nothing in lens touches perf or hardware performance counters today.

Related

Activity

  1. sshiddib commented on Oct 23, 2026

    @sshiddib

    Field data in support of this, from a SWE-agent (swe_agents, OpenHands in Apptainer) study:

    • CPU per rollout changed 2.1× with the core budget (360.7 CPU-seconds unpinned vs 169.7 pinned to 32 cores, at 32 rollouts per node; 509.7 vs 215.7 at 64 with thread caps and pinning), and none of it was visible in Gym's telemetry.
    • Gym's process-tree memory showed 332 MiB while the container used 948 MB, because the containers sit outside the server's process tree.

    One cheap source: the memory watchdog already scans each container's process tree every second, so it could fill SandboxObservation.cpu_time_s and peak_memory_mib from /proc, with cgroups used where delegation exists. Threads and I/O would also help: thread caps halved threads per node in our runs.

    Happy to test on a Slurm + Apptainer cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions