Skip to content

Add training benchmark suites, telemetry, and CI workload recipes - #305

Draft
Kailash Ranganathan (kailash109) wants to merge 5 commits into
mainfrom
codex/training-benchmarks-review
Draft

Kailash Ranganathan (kailash109) wants to merge 5 commits into
mainfrom
codex/training-benchmarks-review

Conversation

@kailash109

@kailash109 Kailash Ranganathan (kailash109) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a benchmark harness that deploys a selected revision, exercises the stock Tinker SDK, exports numeric backend and sampler telemetry, publishes portable result bundles, and cleans up its own applications. Results can be inspected locally or sent to a separate compatible training dashboard with --collector-app.

CI recipes cover small dense, small MoE, and large MoE models, forward-only eval and SFT, 1,024/8,192-token contexts, and 1/2/4 clients, with pinned model revisions and hardware. Additional suites measure weight synchronization and authenticated control-plane mean/p95 request latency. Nightly recipes exercise the same MoE DAPO workload with replay off, router replay, score centering, and both, plus SWE-Gym. All numeric recipe fields, backend metrics, token counts, reward, and policy lag remain available.

Capture-start and sampler-ready timestamps are correlated by model/publication version per sampler, separately from update-to-first-response latency. Cross-host timing requires synchronized clocks. This PR makes no performance improvement claims.

The complete workflow is in benchmarks/workflows/gpu-benchmarks.yml. GitHub rejected publishing .github/workflows/benchmarks.yml because the available credentials lack workflow-write permission. An account with that permission must install the template under .github/workflows and configure the collector environment before nightly execution is enabled.

Validation

  • Current-main integration at e8d2277: 1,351 tests passed and eight skipped in the full CPU run. The existing --=1 argparse test still fails. An editable-package metadata mismatch after upgrading to 0.1.7 was resolved by reinstalling the locked editable package; version tests pass afterward.
  • Targeted benchmark, executor, sampler, and both Miles actor integration tests: 203 passed, one skipped. Benchmark regressions also pass on the final checkout.
  • Eval tests assert no backward pass or optimizer update. Load tests preserve samples and failure summaries. Replay payload tests cover all four combinations. Weight-sync tests cover out-of-order records, replicas, unmatched versions, negative intervals, and reimports.
  • All 18 workload recipes validate and resolve their deployment configs. The pinned MoE configuration supplies 40 layers and eight experts per token.
  • Full pre-commit was attempted. Repository-wide Ruff reports missing annotations in existing benchmark code; typing retains two existing Torch autograd backward override errors. The new capture modules and tests pass targeted Ruff and typing checks.
  • The new GPU matrix, replay RL variants, and capture-to-ready timing have not run end to end on this head. The workflow template is not active.
  • Reconciled with current main without removing its usage accounting, Miles v0.2 actor support, or sampler retry changes. CUDA observations initialize through the shared actor helper used by both actor versions.

Fine-grained synthetic monitor capture

Synthetic monitors now retain all numeric recipe fields and raw events, plus stock SDK trainer, optimizer, checkpoint, and sampling result metrics. Capture includes token/sequence counts and submit-to-observed-result latency without retaining training inputs or sampled text. Partial results survive workload failure, each run gets a durable portable bundle, and optional authenticated collector uploads preserve collection-health details. The launcher is not deployed by this PR update.

Model creation adds run attribution without losing existing metadata. OTLP points and spans override their shared server resource run ID, so trainer metrics can follow a specific synmon while shared device measurements remain deployment-scoped. Server-only timings, replay statistics, and GPU signals still require an instrumented server exporting those fields; client capture does not reconfigure existing deployments.

Validation for this update: 33 benchmark/synmon tests passed, including real stock SDK calls against a local Spindle API, sync/async sampling and optimizer futures, timeout/retry behavior, partial failed-run import, and shared-server attribution. The full CPU suite passed 1,356 tests and skipped eight; the previously observed --=1 CLI parsing test remains the single failure. The new capture modules and tests pass targeted Ruff and typing checks. Full pre-commit was attempted; existing annotation gaps and the two Torch backward override diagnostics remain. No GPU workload or production synmon was deployed or rerun.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant