Repository navigation
Open
Labels
Description
Problem
Users want CPU performance-counter visibility into sandboxed workloads that Gym doesn't provide today. SandboxObservation defines cpu_time_s/peak_memory_mib fields, but there's no perf integration, cgroup metrics, or microarchitectural counters in the codebase yet to build on.
What's being asked for, based on Grace/Vera (and IA) performance work:
- An initial, reasonable metric set is "Top-down L1" (
perf stat -M TopdownL1on Grace/Vera; IA has an equivalent metric, name may differ). - Counter/metric names need to be configurable rather than hardcoded, since they're platform-specific.
- Collection should be scoped to the cores mapped to a single agent/sandbox (
perf's-carg), not the whole socket. - Time-series samples across the run (
perf's-Iarg) are more valuable than a single sample at the end.
What exists vs. what's needed
| Capability | Status |
|---|---|
| Tool-call start/end timestamps, duration | ✅ Already collected — enough to detect idle-GPU periods |
Sandbox cpu_time_s / peak_memory_mib fields |
|
| cgroup-level CPU utilization | |
perf-based microarchitectural counters (Top-down L1) |
❌ Does not exist |
| Per-sandbox core scoping for counter collection | ❌ Does not exist |
| Time-series/interval sampling | ❌ Does not exist — every metric today is a single end-of-run number |
| Platform-specific counter-name abstraction | ❌ Does not exist |
What this needs to answer
- How to actually gather Top-down L1 (or platform-equivalent) counters, scoped to the cores a given sandbox owns, as a time series rather than a single end-of-run sample.
- Counter/metric names are platform-specific (Grace/Vera vs. IA) — the mechanism needs to accommodate that rather than assume one fixed set.
- Whether
perf statcan run with the privileges available inside Gym's sandbox providers (Apptainer, Docker, OpenSandbox) — this needs checking per-provider, sinceperf_event_paranoidrestrictions could rule out approaches before they're worth designing further. - How this relates to the core-pinning work already landing in feat(anyterminal): OpenSandbox oracle config, resource knobs, cgroup metrics, TB2.1 prepare #2980 — "which cores does this sandbox own" is a question that work is already answering for one harness; worth understanding before deciding whether to duplicate or share it.
- Where the resulting data should live — Gym already has an OTel/nemo-lens pipe for other metrics (
nemo_gym/telemetry/), though that infrastructure only provides a place to export numbers, not a way to collect CPU counters — nothing in lens touchesperfor hardware performance counters today.
Related
- feat(anyterminal): OpenSandbox oracle config, resource knobs, cgroup metrics, TB2.1 prepare #2980 — cgroup CPU/memory metrics + pinning for
anyterminal_agent(unmerged, narrower) nemo_gym/rollout_observability.py—SandboxObservationschema
Activity
sshiddib commentedon Oct 23, 2026
Field data in support of this, from a SWE-agent (
swe_agents, OpenHands in Apptainer) study:One cheap source: the memory watchdog already scans each container's process tree every second, so it could fill
SandboxObservation.cpu_time_sandpeak_memory_mibfrom/proc, with cgroups used where delegation exists. Threads and I/O would also help: thread caps halved threads per node in our runs.Happy to test on a Slurm + Apptainer cluster.