Repository navigation
Add training benchmark suites, telemetry, and CI workload recipes - #305
Draft
Kailash Ranganathan (kailash109) wants to merge 5 commits into
Draft
Kailash Ranganathan (kailash109) wants to merge 5 commits into
Kailash Ranganathan (kailash109) wants to merge 5 commits into
Conversation
Keep the dashboard in Frontier and publish portable benchmark bundles.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a benchmark harness that deploys a selected revision, exercises the stock Tinker SDK, exports numeric backend and sampler telemetry, publishes portable result bundles, and cleans up its own applications. Results can be inspected locally or sent to a separate compatible training dashboard with
--collector-app.CI recipes cover small dense, small MoE, and large MoE models, forward-only eval and SFT, 1,024/8,192-token contexts, and 1/2/4 clients, with pinned model revisions and hardware. Additional suites measure weight synchronization and authenticated control-plane mean/p95 request latency. Nightly recipes exercise the same MoE DAPO workload with replay off, router replay, score centering, and both, plus SWE-Gym. All numeric recipe fields, backend metrics, token counts, reward, and policy lag remain available.
Capture-start and sampler-ready timestamps are correlated by model/publication version per sampler, separately from update-to-first-response latency. Cross-host timing requires synchronized clocks. This PR makes no performance improvement claims.
The complete workflow is in
benchmarks/workflows/gpu-benchmarks.yml. GitHub rejected publishing.github/workflows/benchmarks.ymlbecause the available credentials lack workflow-write permission. An account with that permission must install the template under.github/workflowsand configure the collector environment before nightly execution is enabled.Validation
e8d2277: 1,351 tests passed and eight skipped in the full CPU run. The existing--=1argparse test still fails. An editable-package metadata mismatch after upgrading to 0.1.7 was resolved by reinstalling the locked editable package; version tests pass afterward.Fine-grained synthetic monitor capture
Synthetic monitors now retain all numeric recipe fields and raw events, plus stock SDK trainer, optimizer, checkpoint, and sampling result metrics. Capture includes token/sequence counts and submit-to-observed-result latency without retaining training inputs or sampled text. Partial results survive workload failure, each run gets a durable portable bundle, and optional authenticated collector uploads preserve collection-health details. The launcher is not deployed by this PR update.
Model creation adds run attribution without losing existing metadata. OTLP points and spans override their shared server resource run ID, so trainer metrics can follow a specific synmon while shared device measurements remain deployment-scoped. Server-only timings, replay statistics, and GPU signals still require an instrumented server exporting those fields; client capture does not reconfigure existing deployments.
Validation for this update: 33 benchmark/synmon tests passed, including real stock SDK calls against a local Spindle API, sync/async sampling and optimizer futures, timeout/retry behavior, partial failed-run import, and shared-server attribution. The full CPU suite passed 1,356 tests and skipped eight; the previously observed
--=1CLI parsing test remains the single failure. The new capture modules and tests pass targeted Ruff and typing checks. Full pre-commit was attempted; existing annotation gaps and the two Torch backward override diagnostics remain. No GPU workload or production synmon was deployed or rerun.