Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ These rules apply to everyone who changes this repo, people and coding agents al
- Make the code explain itself. Before you add a comment, try a clearer name, an extracted variable or a simpler structure. Keep a comment only when it says something the code can't, such as a non-obvious constraint or an upstream bug, and keep it to one line. Put the reasoning behind a change in the commit message or the PR description.
- Keep changes small. Reuse existing helpers instead of adding new ones. Remove dead branches, unused constants and flags left over from a refactor.
- Wrap replacements for pinned upstream functions in `spindle.patching.same_signature` where ty can see the upstream type.
- Type every function you add or move. The per-file `ANN` ignores in `pyproject.toml` only shrink: never add a file to them, including a new, moved or renamed one.
- Don't delete existing comments or docstrings that a change doesn't touch.

## Tests
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ Notable changes to Spindle, following [Keep a Changelog](https://keepachangelog.
- Reference runs automatically index successful executions under `reference_runs/outputs/`, recording server-reported Spindle commit and version. Process locks protect run IDs, output paths and index updates; scratch outputs remain uncatalogued.
- A two-node Qwen3.8-27B DAPO-Math reference definition uses the shared Tinker workload.
- `spindle usage --group-by` takes several label keys separated by commas, such as `team_id,run_id`, and prints one column per key.
- `beam-501b-a23b-lora-5k` trains `reflection/Beam-1.0-501B-A23B` with BF16 LoRA on two eight-H200 trainer nodes and samples on one eight-B300 node. Attention and MLP gates are separate LoRA targets, and training uses routing replay. It trains on `miles-v0.2.0-8c5de26` and samples on the new `sglang-v0.5.23-568ab74` image, which serves Beam LoRA including its attention output gate.
- The reference-run catalog has a Beam definition, `beam_blog_repro.toml`, that trains on DAPO-Math and evaluates on AIME25 to reproduce the published Beam LoRA run. The index holds an earlier 42-update execution and the published curve.
- Every reference-run output follows one format, validated by `reference_runs/catalog.py` and described in `reference_runs/README.md`. Index entries for executions outside Spindle record `source` instead of Spindle provenance.
- Miles LoRA clients accept the stock Tinker SDK’s `create_lora_training_client(seed=...)` to reproduce adapter initialization without changing other clients’ random streams.
- Sampling supports the Tinker SDK's `prompt_logprobs_last_n`. It returns prompt logprobs for only the last N prompt tokens, so the rest of the prompt can come from the prefix cache.
- `nemotron35-super-vl-120b-a12b-lora-64k` trains LoRA adapters on NVIDIA Nemotron 3.5 Super VL 120B-A12B. Training and sampling take text only: image inputs are not accepted, and adapters apply to the language model.
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ ignore = [
]

[tool.ruff.lint.per-file-ignores]
# Files written before ANN was enforced. Remove a file once its functions are typed.
# Files written before ANN was enforced. This list only shrinks: never add a file,
# including a new or moved one; type its functions instead. Remove a file once it is typed.
"examples/codeforces-codegolf/codegolf/app.py" = ["ANN"]
"examples/codeforces-codegolf/codegolf/cli.py" = ["ANN"]
"examples/codeforces-codegolf/codegolf/config.py" = ["ANN"]
Expand Down
1 change: 1 addition & 0 deletions reference_runs/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
runs/
*.png
outputs/*/
59 changes: 45 additions & 14 deletions reference_runs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ references train through the stock Tinker SDK.
reference_runs/
configs/dapo_qwen35_4b.toml # Immutable validation definition
configs/dapo_qwen38_27b_multinode.toml
configs/beam_blog_repro.toml
scripts/dapo_math.py # Dataset, grading and training settings
scripts/beam_blog_repro.py # Beam LoRA training with AIME25 evaluation
run.py # Prepare config or launch the script
index.toml # Execution provenance and output paths
outputs/*.jsonl # Flat, incremental metric files
Expand Down Expand Up @@ -53,6 +55,21 @@ The workload generates at most 2,048 response tokens; it does not validate 256K
sequences. Compare new
measurements against the same validation, rather than the 4B learning curve.

`beam_blog_repro.toml` trains `reflection/Beam-1.0-501B-A23B` with BF16
LoRA on the bundled `beam-501b-a23b-lora-5k` recipe: two eight-H200 trainer
nodes and one eight-B300 sampler. Its script uses routing replay, CISPO with
one update of rollout prefetch and a 400-update cosine learning-rate schedule,
and evaluates on AIME25. It writes local evaluations, run identity, the replay
cache and plots to a work directory next to the output (`outputs/NAME/`), which
Git ignores. The launcher indexes the run when the script finishes all
planned updates. A run that fails or exceeds its update time budget exits with
an error after saving a checkpoint and is not indexed. Run the script directly
with `--resume` or `--stop-after` to continue after a restart, and add that run
to the index yourself; `--plot-only` compares a run with the published Beam
LoRA curve, indexed as `beam-blog-digitized`. Index entries for executions
outside Spindle record `source` instead of Spindle provenance. The indexed
execution predates the recipe's Miles v0.2.0 image and stopped at 42 of 100 planned updates.

Once published, a validation TOML file is immutable. Create a new definition
when changing its script selection, arguments or deployment overrides. Spindle
commits and observed runtime image versions identify executions and belong in
Expand Down Expand Up @@ -92,20 +109,33 @@ the historical deployment revisions in the index, not today's parent config.

## Metrics and index

Each JSONL record has `time` (Unix seconds) and `event`. Records are appended as
the workload progresses so a partial run remains useful.
Every reference script writes the same JSONL format, so `plot.py` can compare any
two executions. `read_output` in `catalog.py` validates each complete line with the
`Record`, `Evaluation` and `Update` models; tests check every indexed output and
each script's output against them. Records are appended as the workload
progresses, so a partial run remains useful; a final partial line is ignored.

| Event | Contents |
| --- | --- |
| `run` | Effective workload settings, Adam parameters, SDK version, selected deployment name |
| `publication` | Accepted update number and elapsed seconds |
| `evaluation` | Update number, accuracy, response count, mean tokens, truncation fraction, seconds |
| `checkpoint` | Update number and save duration; no checkpoint URI |
| `update` | Accepted step, batch accuracy/tokens/truncation, client sampling/backward/optimizer times, server metrics |
Every record has `event` and `time` (Unix seconds). `evaluation` and `update`
records also require:

Update intervals include publication, evaluation and checkpoint overhead. The
first update has no preceding interval. Missing server metrics stay absent.
The script rejects an explicitly unsuccessful optimizer result.
| Field | Meaning |
| --- | --- |
| `step` | Accepted optimizer updates before the evaluation, or the update number |
| `responses` | Sampled responses scored |
| `reward` | Mean reward; for exact-answer grading, the fraction answered correctly |
| `response_tokens_mean` | Mean sampled response length in tokens |
| `truncated_fraction` | Fraction of responses that hit the token limit |

`update` records add `seconds_since_previous_update` from the second accepted
update on. The interval includes publication, evaluation and checkpoint
overhead. Scripts may add any other fields and events, for example
`run`, `publication` and `checkpoint` in `dapo_math.py`, or per-problem accuracy
and replay diagnostics in `beam_blog_repro.py`. Add new fields under their own
names rather than reusing a standard name with a different meaning.

An execution outside Spindle, indexed with `source`, holds only `evaluation`
records with `step` and optional `reward` and `response_tokens_mean`, read from
its published chart.

`index.toml` maps stable run IDs to validation definitions and JSONL files.
`ReferenceRunIndex` and `ReferenceRun` in `catalog.py` validate its fields and unique IDs.
Expand Down Expand Up @@ -135,8 +165,9 @@ Failed runs keep their partial metrics and are not indexed. Keep the persistent
hidden lock files in place. These locks coordinate processes using the same
filesystem; independent checkouts must merge their catalogs through Git.
Do not invent digests or hardware observations
for historical runs where they were not recorded. Existing historical JSONL
files are kept byte-for-byte, including their original hardware metadata.
for historical runs where they were not recorded. The Qwen3.5-4B JSONL files are
kept byte-for-byte, including their original hardware metadata. The Beam file's
field names were converted to this format; its index notes list the changes.

Only completed runs' compact metrics and index entries belong in Git. Keep
partial or in-progress metrics uncommitted. Raw generations, token
Expand Down
99 changes: 93 additions & 6 deletions reference_runs/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
import tomllib
from collections.abc import Generator
from pathlib import Path
from typing import Self
from typing import Literal, Self

from pydantic import BaseModel, ConfigDict, model_validator
from pydantic import BaseModel, ConfigDict, Field, model_validator


class ReferenceRun(BaseModel):
Expand All @@ -22,13 +22,39 @@ class ReferenceRun(BaseModel):
output: str
spindle_commit: str | None = None
spindle_version: str | None = None
trainer_image: str
miles_commit: str
tinker_sdk: str
hardware: str
trainer_image: str | None = None
miles_commit: str | None = None
tinker_sdk: str | None = None
hardware: str | None = None
source: str | None = None
region: str | None = None
notes: str = ""

@model_validator(mode="after")
def spindle_or_external(self) -> Self:
"""A Spindle execution records its runtime; an external one only its source.

spindle_commit and spindle_version stay unset when the server does not report them.
"""
runtime = {
"trainer_image": self.trainer_image,
"miles_commit": self.miles_commit,
"tinker_sdk": self.tinker_sdk,
"hardware": self.hardware,
}
if self.source is None:
missing = [name for name, value in runtime.items() if value is None]
if missing:
raise ValueError(f"Spindle execution is missing {', '.join(missing)}")
elif any(
value is not None
for value in (self.spindle_commit, self.spindle_version, *runtime.values())
):
raise ValueError(
"external execution records source, not Spindle provenance"
)
return self


class ReferenceRunIndex(BaseModel):
model_config = ConfigDict(extra="forbid")
Expand All @@ -42,6 +68,67 @@ def unique_ids(self) -> Self:
return self


class Record(BaseModel):
"""One line of a Spindle execution's output. Scripts may add any other fields."""

model_config = ConfigDict(extra="allow")

event: str
time: float


class Measurement(BaseModel):
"""Fields every evaluation and update line carries."""

model_config = ConfigDict(extra="allow")

time: float
step: int = Field(ge=0)
responses: int = Field(gt=0)
reward: float
response_tokens_mean: float
truncated_fraction: float = Field(ge=0, le=1)


class Evaluation(Measurement):
"""Held-out sampling with the policy after ``step`` accepted updates."""

event: Literal["evaluation"]


class Update(Measurement):
"""The batch that produced accepted optimizer update ``step``."""

event: Literal["update"]
seconds_since_previous_update: float | None = None


class PublishedEvaluation(BaseModel):
"""A point read from an external execution's published chart."""

model_config = ConfigDict(extra="forbid")

event: Literal["evaluation"]
step: int = Field(ge=0)
reward: float | None = None
response_tokens_mean: float | None = None


SCHEMAS: dict[str, type[BaseModel]] = {"evaluation": Evaluation, "update": Update}


def read_output(path: Path, *, published: bool = False) -> list[dict]:
"""Validate an output's complete lines; a run interrupted mid-write ends in a partial one."""
lines = path.read_text().splitlines(keepends=True)
records = [json.loads(line) for line in lines if line.endswith("\n")]
for record in records:
if published:
PublishedEvaluation.model_validate(record)
else:
SCHEMAS.get(record.get("event"), Record).model_validate(record)
return records


@contextlib.contextmanager
def _lock(path: Path, *, blocking: bool = True) -> Generator[None]:
path.parent.mkdir(parents=True, exist_ok=True)
Expand Down
6 changes: 6 additions & 0 deletions reference_runs/configs/beam_blog_repro.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
config = "beam-501b-a23b-lora-5k"
script = "../scripts/beam_blog_repro.py"

[tokenizer]
model = "reflection/Beam-1.0-501B-A23B"
revision = "12950ae640b48cbbe9b5bb7735bf197507d6383f"
51 changes: 51 additions & 0 deletions reference_runs/index.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,54 @@ tinker_sdk = "0.26.0"
hardware = "Two eight-H200 trainer nodes (TP2/CP8); two four-H200 sampler replicas. All GPUs reported 143771 MiB, driver 610.57.04."
region = "Trainers: us-south. Samplers: us-south and eu-south."
notes = "Completed 20-update stock-Tinker execution. Server commit/version were captured from get_server_capabilities before cleanup; four critical runtime file hashes matched that commit on all four GPU containers. Trainer Miles revision and H200 allocation were verified inside the executing containers. Sampler image sglang-v0.5.23-c4182b2 overlays fork c4182b22139a5730c9f18907cc0b178a305bb0f1 onto its upstream base. Cached base-weight snapshot: 37561816e66b5b9153b1b64c10968c18c1e6f4a0. Initial model download and client creation are excluded from recorded time; first-step compilation and subsequent warmup are included. All updates were accepted with finite metrics; evaluations completed at 0/5/10/15/20, publications at 0 through 20, checkpoints saved at 0/10/20. Held-out accuracy: 65/256 to 137/256. Median interval 559.4 seconds; total recorded time 13809.8 seconds. One workload seed; checkpoints saved, not restored. Compiler caches used the shared volume. This execution predates the new tooling PR and does not validate its head end to end."

[[runs]]
id = "beam-blog-repro-miles-v012"
validation = "beam_blog_repro"
output = "outputs/beam-blog-repro-miles-v012.jsonl"
spindle_commit = "2a0a9611bde3966b63b4827a9b5ef6687fb288b4"
trainer_image = "Miles v0.1.2 with Beam support"
miles_commit = "6f5a4a8429e4f57a65a7251facb7d3d9faf0b3ff"
tinker_sdk = "0.26.0"
hardware = "Two eight-H200 trainer nodes; one eight-B300 sampler replica."
notes = '''
Historical execution. Field names were converted to the catalog output format with
values unchanged: accuracy -> reward, mean_tokens -> response_tokens_mean,
completions -> responses, token_limit_fraction -> truncated_fraction, and capped
divided by responses -> truncated_fraction. Update responses (32 groups x 8) and
seconds_since_previous_update (from consecutive update times) were added. It
predates this TOML catalog and the miles-v0.2.0 recipe; it validates only the runtime pins recorded
here. Trainer SGLang 9ac1c65b, sampler SGLang 991fb37566. The server ran at Spindle
2a0a961 and was redeployed at 94edd89 with identical src/. Client source was the
reference-run script at 6bcf5aa through update 10 and 35f365f afterward.
The run stopped at 42 accepted updates of the planned 100. Update 43 finished
backward and was cancelled before optimizer submission. The update-10 cold restart
matched all 1,664 adapter, master and Adam tensors at optimizer step 10.
AIME25 (30 problems, eight samples, 4,096-token cap): 97/240 (40.42%) initially,
154/240 (64.17%) at update 40, the latest complete evaluation. Mean response tokens:
3,180.54 -> 2,492.28. Paired-problem 95% bootstrap interval for the change: +12.92
to +35.83 points; an intermediate comparison, not the planned update-100 test.
Independent strict regrading agreed with all 2,160 local responses.
Median seconds: native forward/backward 449.40, trainer API wall time 513.19,
between accepted updates 621.90 (including the planned cold restart). Wall time
through the stop: 10.90 hours.
The metrics hold aggregates and accuracy per problem; no prompts, answers, labels
or routing payloads. Plot with the published Beam LoRA curve:
uv run --with matplotlib==3.11.2 python reference_runs/scripts/beam_blog_repro.py \
--output reference_runs/outputs/beam-blog-repro-miles-v012.jsonl --plot-only
'''

[[runs]]
id = "beam-blog-digitized"
validation = "beam_blog_repro"
output = "outputs/beam-blog-digitized.jsonl"
source = "https://huggingface.co/blog/reflection-beam#train-with-lora"
notes = '''
Not a Spindle execution and not raw run data: an approximate manual digitization of
the published Beam LoRA chart, kept to overlay on Beam executions. Evaluation
records hold AIME25 accuracy as reward and mean response tokens. The published
evaluation-step axis is not verified to count optimizer updates, and the exact
invocation, hardware and orchestration are unavailable. The overlay is descriptive;
do not infer matched learning speed or parity. The published endpoint is 64.2%
accuracy and 2,197 mean response tokens.
'''
Loading
Loading