Skip to content

ROADMAP.md

Latest commit

 

History

History
112 lines (91 loc) · 6.49 KB

ROADMAP.md

File metadata and controls

112 lines (91 loc) · 6.49 KB
Raw

churn — Mission & Roadmap

Mission

Make decentralized training over unreliable, heterogeneous, permissionless hardware robust to the messy middle — nodes that are slow, flaky, or partitioned but not dead — and make that robustness reproducible and testable without a GPU fleet.

Today's decentralized-training stacks (e.g. Prime Intellect's prime-diloco + PCCL) handle dead nodes well. They do not handle slow-but-alive stragglers, and there is no deterministic way to reproduce membership chaos — a partition, a crash mid-collective, a slow node — to develop and prove the policies that would. churn exists to close both gaps: a deterministic membership/fault testbed, and a library of membership policies that demonstrably beat the status quo and are validated against real stacks.

Principles

  • Determinism is an invariant, not a feature. Same seed + scenario ⇒ byte-identical trace. Everything is built so this stays true; it is enforced by property tests and golden traces.
  • Mechanism vs. policy. The engine owns time, membership, network, and events. Policies own only the decisions, behind one trait. New policies are drop-ins.
  • Relative truth before absolute truth. We model time, membership, and coordination — not tensors. Results are meaningful as A/B deltas on identical seeds; we earn the right to claim absolute numbers only through calibration and validation against real code.
  • Zero-GPU by default. The core is a pure simulator anyone can run on a laptop. Real-hardware validation is a deliberate, scoped layer on top — not a prerequisite to contribute.

Where we are today

Area Status
Deterministic engine (churn-core) + BaselinePolicy + CLI + golden traces ✅ shipped
StragglerPolicy (adaptive deadline + quorum/sideline + graduated evict) + A/B harness ✅ shipped
Python wheel (churn) — scenario builders, run()/compare(), typed results ✅ shipped
Joiner-stall modeling (bandwidth-modeled late-join state fetch) ✅ shipped
Calibration against real gloo runs + directional validation on a real torch.distributed DiLoCo loop ✅ shipped
CI quality/determinism gate (GitHub Actions) ✅ shipped
Parameter sweeps, published wheels, plotting, docs site ⏳ not started

Demonstrated in-sim: on a persistent-straggler scenario, StragglerPolicy is ~4.6× faster in wall-clock and lifts utilization 0.62 → 0.89 vs. the PRIME-mirroring baseline.

Validated on real code: a torch.distributed/gloo DiLoCo loop (integration/) runs both policies on real 4-process runs; the straggler win reproduces directionally (slow rank sidelined then evicted, ~1.5× faster), and a calibrated sim predicts the same direction. See integration/results/REPORT.md.

Roadmap

v1.1 — Credibility (the critical path) — ✅ shipped

The single most important thing: turn "compelling in-sim" into "directionally validated on real code."

  • ✅ Calibration. Inner-step time and transfer bandwidth measured from small local gloo runs (CPU, 2–4 processes); the engine's per-inner-step timing model is fit to them (constants documented in integration/, not magic numbers).
  • ✅ Real directional bridge (integration/, out of the core wheel/CI). The StragglerPolicy decisions are ported into a minimal torch.distributed/gloo DiLoCo loop with slowed workers; the direction of the sim's predicted win reproduces (see integration/results/REPORT.md). Scope: directional, not absolute-number matching. (Real prime-diloco/PCCL remains gated on upstream packaging — pccl is a PyPI placeholder today.)
  • ✅ Joiner-stall modeling. Late-join state-fetch latency is modeled via the network model and populates the joiner_stall_time metric.

v1.2 — Rigor

Show where the policy wins and loses, and harden the testbed.

  • ⏳ Parameter sweeps. worker-count × slowdown-factor × quorum/k → a map of where StragglerPolicy helps, breaks even, or hurts (e.g. an over-aggressive deadline sidelining healthy nodes). Publish the map.
  • ⏳ Richer scenarios. Jittery stragglers that actually exercise median + k·MAD (not just the min_deadline floor); end-to-end partition scenarios driven through to eviction; mixed crash+straggler+rejoin runs. Grow the committed golden-trace library alongside.
  • ✅ Determinism + quality gate in CI. GitHub Actions runs cargo test/clippy, the proptest, golden-trace diffs, and the Python smoke tests on every push.

v1.3 — Adoption

Make it trivial to install, trust, and use.

  • Published wheels (PyPI) + cross-platform builds; cargo crate publish for churn-core.
  • churn.plot — optional plotting helpers (arrival/stall timelines, baseline-vs-straggler bars).
  • A docs site / example notebooks; a CHANGELOG. (Dual LICENSE-APACHE + LICENSE-MIT and CONTRIBUTING / CODE_OF_CONDUCT are in place.)

v2 — Research surface

Use the testbed to push past the first policy.

  • More policies: predictive/learned deadlines, bandwidth-aware quorum, partition-aware membership.
  • Adversarial/Byzantine membership and incentive-aware eviction (a natural tie-in to permissionless, staked participation).
  • Pseudo-gradient normalization-by-participant-count correctness, exercised against real tensors via the integration bridge.

Continuous engine health

  • Split sim.rs (round loop / heartbeat / inject handling) before it grows further.
  • Richer, more honest Metrics (true compute accounting for sidelined/crashed workers).
  • A scenario DSL / TOML loader for the chaos library.

How you'll know it's working

  • v1.1: the sim's predicted directional win reproduces on a real 2–4 process prime-diloco/PCCL run with slowed workers.
  • v1.2: a published failure-mode map that says, honestly, when the straggler policy is worth it.
  • Adoption: people use the deterministic testbed for their own membership/fault logic — independent of whether any single upstream adopts the policy. Phase 1 is useful on its own.

Contributing

The core is zero-GPU and runs on a laptop — see the README for quickstart. Good first areas: new scenarios (especially partitions and jittery stragglers), parameter-sweep tooling, and the CI gate. The calibration and real-PCCL work (v1.1) is the highest-impact and needs access to a small local PCCL/gloo setup.

Design specs and implementation plans for shipped work live under docs/superpowers/.