Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions docs/models/deepseek/deepseek-v3-2.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Run everything inside the `radixark/miles:latest` container at `/root/miles`. Th
|---|---|---|
| `--model-org` / `--model-name` | `deepseek-ai` / `DeepSeek-V3.2` | The HF repository to download, and the stem of every derived directory name. |
| `--model-dir` | `/root/models` | Holds the HF checkpoint, the `-bf16` cast, and the Megatron `_torch_dist` directory as siblings. |
| `--model-local-dir` | `/root/models` | Node-local destination that `prepare-cp` rsyncs into; worth changing only when `--model-dir` is on shared storage. |
| `--model-local-dir` | `/root/models` | Node-local destination each trainer pod rsyncs into before training; worth changing only when `--model-dir` is on shared storage. |
| `--data-dir` | `/root/datasets` | Where dapo-math-17k and aime-2024 are downloaded. |
| `--megatron-path` | `/root/Megatron-LM` | Added to `PYTHONPATH` for both conversion and training. |
| `--output-dir` | `/root/shared_data` | Training checkpoints land under `{output-dir}/{run-id}/checkpoints`. |
Expand Down Expand Up @@ -93,7 +93,7 @@ python scripts/run_deepseek_v32.py full-train \
--actor-num-nodes 8 --rollout-num-gpus 8
```

The `full-train` subcommand chains download → FP8 → BF16 cast → optional rollout quantization → `torch_dist` conversion → training. It does not run `prepare-cp`; call that separately if you stage checkpoints onto node-local disk.
The `full-train` subcommand chains download → FP8 → BF16 cast → optional rollout quantization → `torch_dist` conversion → training. Staging onto node-local disk is not a separate stage: `train` rsyncs the HF checkpoint and `torch_dist` into `--model-local-dir` on every trainer pod before the job starts.

### 4.2 Individual stages

Expand All @@ -104,9 +104,6 @@ python scripts/run_deepseek_v32.py prepare --actor-num-nodes 8
# re-run just the Megatron conversion
python scripts/run_deepseek_v32.py prepare-megatron-ckpt --actor-num-nodes 8

# rsync the HF checkpoint and torch_dist into --model-local-dir on every node
python scripts/run_deepseek_v32.py prepare-cp --actor-num-nodes 8

# train, assuming the stages above already ran
python scripts/run_deepseek_v32.py train --actor-num-nodes 8 --rollout-num-gpus 8
```
Expand Down
4 changes: 2 additions & 2 deletions docs/models/deepseek/deepseek-v4-flash.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ python scripts/run_deepseek_v4.py full-train \
--num-nodes 8 --num-gpus-per-node 4
```

The `full-train` subcommand chains `prepare-download → [prepare-fp8 →] prepare-single → prepare-spmd → prepare-cp → train` (`prepare-fp8` only for the MXFP4 `0731` variant). Each stage has a sentinel-based skip so you can re-run safely after the first invocation.
The `full-train` subcommand chains `prepare-download → [prepare-fp8 →] prepare-single → prepare-spmd → train` (`prepare-fp8` only for the MXFP4 `0731` variant). Each stage has a sentinel-based skip so you can re-run safely after the first invocation.

### 3.2 Launcher path defaults

Expand All @@ -67,7 +67,7 @@ The Python launcher (`scripts/run_deepseek_v4.py`) takes its path arguments from
|---|---|---|
| `--data-dir` | `/root/datasets` | HF datasets (e.g. dapo-math-17k, …) |
| `--model-dir` | `/root/models` | parent directory holding the HF checkpoint and Megatron `_torch_dist` artifacts as separate sibling sub-directories |
| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; `prepare-cp` rsyncs the HF checkpoint and `_torch_dist` here so the trainer reads from local disk instead of shared storage (only worth setting when `--model-dir` is on shared/remote storage) |
| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; when it differs from `--model-dir`, every trainer rsyncs the HF checkpoint and `_torch_dist` here before training starts, so the trainer reads from local disk instead of shared storage (only worth setting when `--model-dir` is on shared/remote storage) |
| `--save-dir` | `/root/models` | training checkpoints under `{save-dir}/{run-id}/checkpoints/` |

You can override these via the CLI flags above or equivalently via env vars — every launcher option binds to `MILES_SCRIPT_<FIELD_NAME_UPPER>` (e.g. `MILES_SCRIPT_MODEL_DIR`), with precedence CLI flag > env var > built-in default; run `train --help` to see each option's `[env var: …]` name.
Expand Down
4 changes: 2 additions & 2 deletions docs/models/deepseek/deepseek-v4-pro.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,15 @@ python scripts/run_deepseek_v4.py full-train \
--num-nodes 32 --num-gpus-per-node 8
```

The `full-train` subcommand chains `prepare-download → prepare-single → prepare-spmd → prepare-cp → train`. Each stage has a sentinel-based skip so you can re-run safely after the first invocation.
The `full-train` subcommand chains `prepare-download → prepare-single → prepare-spmd → train`. Each stage has a sentinel-based skip so you can re-run safely after the first invocation.

### 3.2 Launcher path defaults

| Flag | Default | Use |
|---|---|---|
| `--data-dir` | `/root/datasets` | HF datasets (e.g. dapo-math-17k, …) |
| `--model-dir` | `/root/models` | parent directory holding the HF checkpoint and Megatron `_torch_dist` artifacts |
| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; `prepare-cp` rsyncs the HF checkpoint and `_torch_dist` here so the trainer reads from local disk (set it when `--model-dir` is on shared/remote storage) |
| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; when it differs from `--model-dir`, every trainer rsyncs the HF checkpoint and `_torch_dist` here before training starts (set it when `--model-dir` is on shared/remote storage) |
| `--save-dir` | `/root/models` | training checkpoints under `{save-dir}/{run-id}/checkpoints/` |

Pro uses the same launcher as V4-Flash, so every option above can also be preconfigured via `MILES_SCRIPT_<FIELD_NAME_UPPER>` env vars (precedence: CLI flag > env var > built-in default) — see [V4-Flash §3.2](/models/deepseek/deepseek-v4-flash#32-launcher-path-defaults) for details.
Expand Down
18 changes: 11 additions & 7 deletions docs/models/deepseek/deepseek.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,21 @@ definition — the single-node smoke-test path.

## 3. Launch

`scripts/run_deepseek.py` drives the whole pipeline: HF download, FP8 → BF16 cast, HF → Megatron
`torch_dist` conversion, an rsync of both directories to node-local storage, and the `train.py`
submission.
`scripts/run_deepseek.py` splits the pipeline across three commands:

- `prepare` — HF download, FP8 → BF16 cast, HF → Megatron `torch_dist` conversion.
- `train` — the `train.py` submission; each trainer pod first rsyncs both model directories to
node-local storage.
- `full-train` — `prepare` followed by `train`, which is the whole pipeline in one invocation.

```bash
python scripts/run_deepseek.py train --num-nodes 16 --num-gpus-per-node 8
python scripts/run_deepseek.py full-train --num-nodes 16 --num-gpus-per-node 8
```

Single-node smoke test on a pruned checkpoint:

```bash
python scripts/run_deepseek.py train --model-name DeepSeek-V3-0324-5layer --num-nodes 1
python scripts/run_deepseek.py full-train --model-name DeepSeek-V3-0324-5layer --num-nodes 1
```

Directories default to `--model-dir /root/models` (shared FS), `--data-dir /root/datasets`, and
Expand All @@ -59,7 +62,7 @@ ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 \
--node-ip-address ${WORKER_IP} --disable-usage-stats

# back on node 0
MILES_SCRIPT_EXTERNAL_RAY=1 python scripts/run_deepseek.py train \
MILES_SCRIPT_EXTERNAL_RAY=1 python scripts/run_deepseek.py full-train \
--num-nodes 16 --num-gpus-per-node 8
```

Expand All @@ -84,7 +87,8 @@ runs on that one node.

## 4. Checkpoint conversion

`train` performs the two conversion steps for you; the equivalent manual commands are below.
`prepare`, and therefore `full-train`, performs the two conversion steps for you; the equivalent
manual commands are below.

The HF checkpoint ships in block-quantized FP8 — first cast it to BF16:

Expand Down
9 changes: 3 additions & 6 deletions docs/models/glm/glm5-2.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ python scripts/run_glm5_2_744b_a40b.py prepare --model-name GLM-5.2 --num-nodes

### 3.2 HF → Megatron `torch_dist` conversion

Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5.2 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`, `num_hidden_layers=78`, no `auto_map`) and fails fast if it does not, then converts it to the `glm5.2-744B-A40B` Megatron model type. The full model converts with PP = 4 (18/20 first/last layer split); the pruned model converts on a single GPU, because DSA's cross-layer index sharing forbids a pipeline stage that starts on a skip layer. Run `prepare-cp` afterwards on every node to copy the converted checkpoint from shared NFS to local disk.
Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5.2 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`, `num_hidden_layers=78`, no `auto_map`) and fails fast if it does not, then converts it to the `glm5.2-744B-A40B` Megatron model type. The full model converts with PP = 4 (18/20 first/last layer split); the pruned model converts on a single GPU, because DSA's cross-layer index sharing forbids a pipeline stage that starts on a skip layer. `train` copies the converted checkpoint from shared NFS to node-local disk on every trainer pod before the job starts.

## 4. Launch

Expand All @@ -54,18 +54,15 @@ Full model (≥ 16 nodes):
python scripts/run_glm5_2_744b_a40b.py full-train --model-name GLM-5.2 --num-nodes 32
```

The Typer app exposes four subcommands:
The Typer app exposes three subcommands:

```bash
python scripts/run_glm5_2_744b_a40b.py full-train --model-name GLM-5.2 --num-nodes <N>

# Just download model + datasets and convert to Megatron
python scripts/run_glm5_2_744b_a40b.py prepare --model-name GLM-5.2 --num-nodes <N>

# Copy converted checkpoint from shared NFS to local disk (run on every node)
python scripts/run_glm5_2_744b_a40b.py prepare-cp --model-name GLM-5.2 --num-nodes <N>

# Train only (assumes prepare/prepare-cp done)
# Train only (assumes prepare done)
python scripts/run_glm5_2_744b_a40b.py train --model-name GLM-5.2 --num-nodes <N>
```

Expand Down
9 changes: 3 additions & 6 deletions docs/models/glm/glm5.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ python scripts/run_glm5_744b_a40b.py prepare --model-name GLM-5 --num-nodes 16

### 3.2 HF → Megatron `torch_dist` conversion

Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`) and fails fast if it does not, then converts it to the `glm5-744B-A40B` Megatron model type. Run `prepare-cp` afterwards on every node to copy the converted checkpoint from shared NFS to local disk.
Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`) and fails fast if it does not, then converts it to the `glm5-744B-A40B` Megatron model type. Training itself copies the converted checkpoint from shared NFS to each node's local disk before it starts.

## 4. Launch

Expand All @@ -42,18 +42,15 @@ Also handled by `prepare`. Before conversion the launcher validates, via `_valid
python scripts/run_glm5_744b_a40b.py full-train --model-name GLM-5 --num-nodes 16
```

The Typer app exposes four subcommands:
The Typer app exposes three subcommands:

```bash
python scripts/run_glm5_744b_a40b.py full-train --model-name GLM-5 --num-nodes <N>

# Just download model + datasets and convert to Megatron
python scripts/run_glm5_744b_a40b.py prepare --model-name GLM-5 --num-nodes <N>

# Copy converted checkpoint from shared NFS to local disk (run on every node)
python scripts/run_glm5_744b_a40b.py prepare-cp --model-name GLM-5 --num-nodes <N>

# Train only (assumes prepare/prepare-cp done)
# Train only (assumes prepare done); copies the checkpoint to each node's local disk first
python scripts/run_glm5_744b_a40b.py train --model-name GLM-5 --num-nodes <N>
```

Expand Down
36 changes: 12 additions & 24 deletions scripts/amd/run_deepseek_v4.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,12 @@
--model-name DeepSeek-V4-Flash-FP8-4layer \
--num-nodes 1 --num-gpus-per-node 8

2. Individual steps (download -> FP8->BF16 -> BF16->torch_dist -> rsync -> train):
2. Individual steps (download -> FP8->BF16 -> BF16->torch_dist -> train):
python scripts/run_deepseek_v4.py prepare-download --model-name DeepSeek-V4-Flash-FP8
python scripts/run_deepseek_v4.py prepare-single --model-name DeepSeek-V4-Flash-FP8 \
--hf-checkpoint /root/models/DeepSeek-V4-Flash-FP8
python scripts/run_deepseek_v4.py prepare-spmd --model-name DeepSeek-V4-Flash-FP8 \
--num-nodes 1 --num-gpus-per-node 8
python scripts/run_deepseek_v4.py prepare-cp --model-name DeepSeek-V4-Flash-FP8
python scripts/run_deepseek_v4.py train --model-name DeepSeek-V4-Flash-FP8 \
--num-nodes 4 --num-gpus-per-node 8 \
--hf-checkpoint /root/models/DeepSeek-V4-Flash-FP8
Expand Down Expand Up @@ -236,23 +235,17 @@ def prepare_spmd(args: ScriptArgs):
_prepare_spmd(args)


@app.command()
@U.dataclass_cli
def prepare_cp(args: ScriptArgs):
_prepare_cp(args)
def _prepare_cmd(args: ScriptArgs) -> dict[str, str]:
if args.model_local_dir == args.model_dir:
return {}


def _prepare_cp(args: ScriptArgs):
U.rsync_simple(
path_src=f"{args.model_dir}/{args.torch_dist_name}",
path_dst=f"{args.model_local_dir}/{args.torch_dist_name}",
num_nodes=args.num_nodes,
)
U.rsync_simple(
path_src=f"{args.model_dir}/{args.model_name}",
path_dst=f"{args.model_local_dir}/{args.model_name}",
num_nodes=args.num_nodes,
)
copies = [
U.rsync_cmd(
f"{args.model_dir}/{args.torch_dist_name}", f"{args.model_local_dir}/{args.torch_dist_name}"
),
U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"),
]
return {"trainer": " && ".join(copies)}


def _get_parallel_config(args: ScriptArgs) -> str:
Expand Down Expand Up @@ -516,11 +509,11 @@ def _train(args: ScriptArgs):

U.execute_train(
train_args=train_args,
config=args,
num_gpus_per_node=args.num_gpus_per_node,
megatron_model_type=args.megatron_model_type,
extra_env_vars={**extra_env_vars},
megatron_path=args.megatron_path,
prepare_cmd=_prepare_cmd(args),
)


Expand Down Expand Up @@ -550,11 +543,6 @@ def full_train(args: ScriptArgs):
else:
print(f"[full_train] Skipping BF16->torch_dist conversion: {torch_dist_sentinel} already exists.")

if args.model_local_dir != args.model_dir:
_prepare_cp(args)
else:
print(f"[full_train] Skipping rsync: model_local_dir == model_dir ({args.model_dir})")

if args.hf_checkpoint is None:
args.hf_checkpoint = f"{args.model_local_dir}/{args.model_name}"

Expand Down
39 changes: 24 additions & 15 deletions scripts/run_deepseek.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@

import re
from dataclasses import dataclass
from functools import partial
from typing import Literal

import typer
Expand Down Expand Up @@ -93,15 +92,15 @@ def _prepare_megatron_ckpt(args: ScriptArgs):
)


def _prepare_cp(args: ScriptArgs):
U.rsync_simple(
path_src=f"{args.model_dir}/{args.model_name}_torch_dist",
path_dst=f"{args.model_local_dir}/{args.model_name}_torch_dist",
)
U.rsync_simple(
path_src=f"{args.model_dir}/{args.model_name}",
path_dst=f"{args.model_local_dir}/{args.model_name}",
)
def _prepare_cmd(args: ScriptArgs) -> dict[str, str]:
copies = [
U.rsync_cmd(
f"{args.model_dir}/{args.model_name}_torch_dist",
f"{args.model_local_dir}/{args.model_name}_torch_dist",
),
U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"),
]
return {"trainer": " && ".join(copies)}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Stage checkpoints on inference workers too

When --cluster-backend kubernetes is used with model_local_dir != model_dir (the default here), this map attaches both rsyncs only to the trainer role. In deliver-2, helm_backend/launcher/values/pool_entry.py::_with_prepare_cmd skips inference-engine pods, while compute_engine_launch_cmd passes the local HF path to SGLang as model_path; disaggregated rollout nodes therefore never receive the model, and colocated inference pods can start before the trainer's rsync completes. Add inference-side staging; the same pattern affects the V4/GLM launchers.



def _execute_train(args: ScriptArgs, before_ray_job_submit=None):
Expand Down Expand Up @@ -299,27 +298,37 @@ def _execute_train(args: ScriptArgs, before_ray_job_submit=None):

U.execute_train(
train_args=train_args,
config=args,
# TODO may get it from `config`
num_gpus_per_node=args.num_gpus_per_node,
megatron_model_type=args.megatron_model_type,
extra_env_vars={**sglang_extra_env_vars},
megatron_path=args.megatron_path,
before_ray_job_submit=before_ray_job_submit,
prepare_cmd=_prepare_cmd(args),
)


@app.command()
@U.dataclass_cli
def train(args: ScriptArgs):
def full_train(args: ScriptArgs) -> None:
_prepare_download(args)
_prepare_bf16_ckpt(args)
_execute_train(args, before_ray_job_submit=partial(_prepare_ray_dependent, args))
_prepare_megatron_ckpt(args)
_execute_train(args)


def _prepare_ray_dependent(args: ScriptArgs):
@app.command()
@U.dataclass_cli
def prepare(args: ScriptArgs) -> None:
_prepare_download(args)
_prepare_bf16_ckpt(args)
_prepare_megatron_ckpt(args)
_prepare_cp(args)


@app.command()
@U.dataclass_cli
def train(args: ScriptArgs) -> None:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Update every caller of the renamed launcher roles

This split makes train submit-only and removes prepare-cp, but deliver-2 still documents the old interfaces in docs/models/deepseek/deepseek.md:27-38, docs/models/glm/glm4-5.md:60-73, docs/models/deepseek/deepseek-v3-2.md:96-111, and docs/models/glm/glm5-2.md:39-69. The first two commands no longer perform the documented full pipeline, while the latter two fail with an unknown subcommand; update all affected pages as required by .claude/rules/launch-and-model-scripts.md:135-140.

_execute_train(args)


@app.callback()
Expand Down
Loading
Loading