diff --git a/docs/models/deepseek/deepseek-v3-2.md b/docs/models/deepseek/deepseek-v3-2.md index a0e2e011d1..6f367a2078 100644 --- a/docs/models/deepseek/deepseek-v3-2.md +++ b/docs/models/deepseek/deepseek-v3-2.md @@ -33,7 +33,7 @@ Run everything inside the `radixark/miles:latest` container at `/root/miles`. Th |---|---|---| | `--model-org` / `--model-name` | `deepseek-ai` / `DeepSeek-V3.2` | The HF repository to download, and the stem of every derived directory name. | | `--model-dir` | `/root/models` | Holds the HF checkpoint, the `-bf16` cast, and the Megatron `_torch_dist` directory as siblings. | -| `--model-local-dir` | `/root/models` | Node-local destination that `prepare-cp` rsyncs into; worth changing only when `--model-dir` is on shared storage. | +| `--model-local-dir` | `/root/models` | Node-local destination each trainer pod rsyncs into before training; worth changing only when `--model-dir` is on shared storage. | | `--data-dir` | `/root/datasets` | Where dapo-math-17k and aime-2024 are downloaded. | | `--megatron-path` | `/root/Megatron-LM` | Added to `PYTHONPATH` for both conversion and training. | | `--output-dir` | `/root/shared_data` | Training checkpoints land under `{output-dir}/{run-id}/checkpoints`. | @@ -93,7 +93,7 @@ python scripts/run_deepseek_v32.py full-train \ --actor-num-nodes 8 --rollout-num-gpus 8 ``` -The `full-train` subcommand chains download → FP8 → BF16 cast → optional rollout quantization → `torch_dist` conversion → training. It does not run `prepare-cp`; call that separately if you stage checkpoints onto node-local disk. +The `full-train` subcommand chains download → FP8 → BF16 cast → optional rollout quantization → `torch_dist` conversion → training. Staging onto node-local disk is not a separate stage: `train` rsyncs the HF checkpoint and `torch_dist` into `--model-local-dir` on every trainer pod before the job starts. ### 4.2 Individual stages @@ -104,9 +104,6 @@ python scripts/run_deepseek_v32.py prepare --actor-num-nodes 8 # re-run just the Megatron conversion python scripts/run_deepseek_v32.py prepare-megatron-ckpt --actor-num-nodes 8 -# rsync the HF checkpoint and torch_dist into --model-local-dir on every node -python scripts/run_deepseek_v32.py prepare-cp --actor-num-nodes 8 - # train, assuming the stages above already ran python scripts/run_deepseek_v32.py train --actor-num-nodes 8 --rollout-num-gpus 8 ``` diff --git a/docs/models/deepseek/deepseek-v4-flash.md b/docs/models/deepseek/deepseek-v4-flash.md index 3e72a8fbe7..47c4d65a0f 100644 --- a/docs/models/deepseek/deepseek-v4-flash.md +++ b/docs/models/deepseek/deepseek-v4-flash.md @@ -57,7 +57,7 @@ python scripts/run_deepseek_v4.py full-train \ --num-nodes 8 --num-gpus-per-node 4 ``` -The `full-train` subcommand chains `prepare-download → [prepare-fp8 →] prepare-single → prepare-spmd → prepare-cp → train` (`prepare-fp8` only for the MXFP4 `0731` variant). Each stage has a sentinel-based skip so you can re-run safely after the first invocation. +The `full-train` subcommand chains `prepare-download → [prepare-fp8 →] prepare-single → prepare-spmd → train` (`prepare-fp8` only for the MXFP4 `0731` variant). Each stage has a sentinel-based skip so you can re-run safely after the first invocation. ### 3.2 Launcher path defaults @@ -67,7 +67,7 @@ The Python launcher (`scripts/run_deepseek_v4.py`) takes its path arguments from |---|---|---| | `--data-dir` | `/root/datasets` | HF datasets (e.g. dapo-math-17k, …) | | `--model-dir` | `/root/models` | parent directory holding the HF checkpoint and Megatron `_torch_dist` artifacts as separate sibling sub-directories | -| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; `prepare-cp` rsyncs the HF checkpoint and `_torch_dist` here so the trainer reads from local disk instead of shared storage (only worth setting when `--model-dir` is on shared/remote storage) | +| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; when it differs from `--model-dir`, every trainer rsyncs the HF checkpoint and `_torch_dist` here before training starts, so the trainer reads from local disk instead of shared storage (only worth setting when `--model-dir` is on shared/remote storage) | | `--save-dir` | `/root/models` | training checkpoints under `{save-dir}/{run-id}/checkpoints/` | You can override these via the CLI flags above or equivalently via env vars — every launcher option binds to `MILES_SCRIPT_` (e.g. `MILES_SCRIPT_MODEL_DIR`), with precedence CLI flag > env var > built-in default; run `train --help` to see each option's `[env var: …]` name. diff --git a/docs/models/deepseek/deepseek-v4-pro.md b/docs/models/deepseek/deepseek-v4-pro.md index 6544a586f0..4e2ffce79a 100644 --- a/docs/models/deepseek/deepseek-v4-pro.md +++ b/docs/models/deepseek/deepseek-v4-pro.md @@ -39,7 +39,7 @@ python scripts/run_deepseek_v4.py full-train \ --num-nodes 32 --num-gpus-per-node 8 ``` -The `full-train` subcommand chains `prepare-download → prepare-single → prepare-spmd → prepare-cp → train`. Each stage has a sentinel-based skip so you can re-run safely after the first invocation. +The `full-train` subcommand chains `prepare-download → prepare-single → prepare-spmd → train`. Each stage has a sentinel-based skip so you can re-run safely after the first invocation. ### 3.2 Launcher path defaults @@ -47,7 +47,7 @@ The `full-train` subcommand chains `prepare-download → prepare-single → prep |---|---|---| | `--data-dir` | `/root/datasets` | HF datasets (e.g. dapo-math-17k, …) | | `--model-dir` | `/root/models` | parent directory holding the HF checkpoint and Megatron `_torch_dist` artifacts | -| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; `prepare-cp` rsyncs the HF checkpoint and `_torch_dist` here so the trainer reads from local disk (set it when `--model-dir` is on shared/remote storage) | +| `--model-local-dir` | unset → same as `--model-dir` | local NVMe path on each node; when it differs from `--model-dir`, every trainer rsyncs the HF checkpoint and `_torch_dist` here before training starts (set it when `--model-dir` is on shared/remote storage) | | `--save-dir` | `/root/models` | training checkpoints under `{save-dir}/{run-id}/checkpoints/` | Pro uses the same launcher as V4-Flash, so every option above can also be preconfigured via `MILES_SCRIPT_` env vars (precedence: CLI flag > env var > built-in default) — see [V4-Flash §3.2](/models/deepseek/deepseek-v4-flash#32-launcher-path-defaults) for details. diff --git a/docs/models/deepseek/deepseek.md b/docs/models/deepseek/deepseek.md index 4fcfcd4f9a..4210f4f404 100644 --- a/docs/models/deepseek/deepseek.md +++ b/docs/models/deepseek/deepseek.md @@ -24,18 +24,21 @@ definition — the single-node smoke-test path. ## 3. Launch -`scripts/run_deepseek.py` drives the whole pipeline: HF download, FP8 → BF16 cast, HF → Megatron -`torch_dist` conversion, an rsync of both directories to node-local storage, and the `train.py` -submission. +`scripts/run_deepseek.py` splits the pipeline across three commands: + +- `prepare` — HF download, FP8 → BF16 cast, HF → Megatron `torch_dist` conversion. +- `train` — the `train.py` submission; each trainer pod first rsyncs both model directories to + node-local storage. +- `full-train` — `prepare` followed by `train`, which is the whole pipeline in one invocation. ```bash -python scripts/run_deepseek.py train --num-nodes 16 --num-gpus-per-node 8 +python scripts/run_deepseek.py full-train --num-nodes 16 --num-gpus-per-node 8 ``` Single-node smoke test on a pruned checkpoint: ```bash -python scripts/run_deepseek.py train --model-name DeepSeek-V3-0324-5layer --num-nodes 1 +python scripts/run_deepseek.py full-train --model-name DeepSeek-V3-0324-5layer --num-nodes 1 ``` Directories default to `--model-dir /root/models` (shared FS), `--data-dir /root/datasets`, and @@ -59,7 +62,7 @@ ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 \ --node-ip-address ${WORKER_IP} --disable-usage-stats # back on node 0 -MILES_SCRIPT_EXTERNAL_RAY=1 python scripts/run_deepseek.py train \ +MILES_SCRIPT_EXTERNAL_RAY=1 python scripts/run_deepseek.py full-train \ --num-nodes 16 --num-gpus-per-node 8 ``` @@ -84,7 +87,8 @@ runs on that one node. ## 4. Checkpoint conversion -`train` performs the two conversion steps for you; the equivalent manual commands are below. +`prepare`, and therefore `full-train`, performs the two conversion steps for you; the equivalent +manual commands are below. The HF checkpoint ships in block-quantized FP8 — first cast it to BF16: diff --git a/docs/models/glm/glm5-2.md b/docs/models/glm/glm5-2.md index 9126e49b60..bae3095c1c 100644 --- a/docs/models/glm/glm5-2.md +++ b/docs/models/glm/glm5-2.md @@ -36,7 +36,7 @@ python scripts/run_glm5_2_744b_a40b.py prepare --model-name GLM-5.2 --num-nodes ### 3.2 HF → Megatron `torch_dist` conversion -Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5.2 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`, `num_hidden_layers=78`, no `auto_map`) and fails fast if it does not, then converts it to the `glm5.2-744B-A40B` Megatron model type. The full model converts with PP = 4 (18/20 first/last layer split); the pruned model converts on a single GPU, because DSA's cross-layer index sharing forbids a pipeline stage that starts on a skip layer. Run `prepare-cp` afterwards on every node to copy the converted checkpoint from shared NFS to local disk. +Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5.2 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`, `num_hidden_layers=78`, no `auto_map`) and fails fast if it does not, then converts it to the `glm5.2-744B-A40B` Megatron model type. The full model converts with PP = 4 (18/20 first/last layer split); the pruned model converts on a single GPU, because DSA's cross-layer index sharing forbids a pipeline stage that starts on a skip layer. `train` copies the converted checkpoint from shared NFS to node-local disk on every trainer pod before the job starts. ## 4. Launch @@ -54,7 +54,7 @@ Full model (≥ 16 nodes): python scripts/run_glm5_2_744b_a40b.py full-train --model-name GLM-5.2 --num-nodes 32 ``` -The Typer app exposes four subcommands: +The Typer app exposes three subcommands: ```bash python scripts/run_glm5_2_744b_a40b.py full-train --model-name GLM-5.2 --num-nodes @@ -62,10 +62,7 @@ python scripts/run_glm5_2_744b_a40b.py full-train --model-name GLM-5.2 --num-nod # Just download model + datasets and convert to Megatron python scripts/run_glm5_2_744b_a40b.py prepare --model-name GLM-5.2 --num-nodes -# Copy converted checkpoint from shared NFS to local disk (run on every node) -python scripts/run_glm5_2_744b_a40b.py prepare-cp --model-name GLM-5.2 --num-nodes - -# Train only (assumes prepare/prepare-cp done) +# Train only (assumes prepare done) python scripts/run_glm5_2_744b_a40b.py train --model-name GLM-5.2 --num-nodes ``` diff --git a/docs/models/glm/glm5.md b/docs/models/glm/glm5.md index a74bc00bc6..2103554528 100644 --- a/docs/models/glm/glm5.md +++ b/docs/models/glm/glm5.md @@ -32,7 +32,7 @@ python scripts/run_glm5_744b_a40b.py prepare --model-name GLM-5 --num-nodes 16 ### 3.2 HF → Megatron `torch_dist` conversion -Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`) and fails fast if it does not, then converts it to the `glm5-744B-A40B` Megatron model type. Run `prepare-cp` afterwards on every node to copy the converted checkpoint from shared NFS to local disk. +Also handled by `prepare`. Before conversion the launcher validates, via `_validate_glm_checkpoint`, that the checkpoint uses the native GLM-5 config (`model_type=glm_moe_dsa`, `architectures=[GlmMoeDsaForCausalLM]`) and fails fast if it does not, then converts it to the `glm5-744B-A40B` Megatron model type. Training itself copies the converted checkpoint from shared NFS to each node's local disk before it starts. ## 4. Launch @@ -42,7 +42,7 @@ Also handled by `prepare`. Before conversion the launcher validates, via `_valid python scripts/run_glm5_744b_a40b.py full-train --model-name GLM-5 --num-nodes 16 ``` -The Typer app exposes four subcommands: +The Typer app exposes three subcommands: ```bash python scripts/run_glm5_744b_a40b.py full-train --model-name GLM-5 --num-nodes @@ -50,10 +50,7 @@ python scripts/run_glm5_744b_a40b.py full-train --model-name GLM-5 --num-nodes < # Just download model + datasets and convert to Megatron python scripts/run_glm5_744b_a40b.py prepare --model-name GLM-5 --num-nodes -# Copy converted checkpoint from shared NFS to local disk (run on every node) -python scripts/run_glm5_744b_a40b.py prepare-cp --model-name GLM-5 --num-nodes - -# Train only (assumes prepare/prepare-cp done) +# Train only (assumes prepare done); copies the checkpoint to each node's local disk first python scripts/run_glm5_744b_a40b.py train --model-name GLM-5 --num-nodes ``` diff --git a/scripts/amd/run_deepseek_v4.py b/scripts/amd/run_deepseek_v4.py index c800489313..486a6cc151 100644 --- a/scripts/amd/run_deepseek_v4.py +++ b/scripts/amd/run_deepseek_v4.py @@ -16,13 +16,12 @@ --model-name DeepSeek-V4-Flash-FP8-4layer \ --num-nodes 1 --num-gpus-per-node 8 - 2. Individual steps (download -> FP8->BF16 -> BF16->torch_dist -> rsync -> train): + 2. Individual steps (download -> FP8->BF16 -> BF16->torch_dist -> train): python scripts/run_deepseek_v4.py prepare-download --model-name DeepSeek-V4-Flash-FP8 python scripts/run_deepseek_v4.py prepare-single --model-name DeepSeek-V4-Flash-FP8 \ --hf-checkpoint /root/models/DeepSeek-V4-Flash-FP8 python scripts/run_deepseek_v4.py prepare-spmd --model-name DeepSeek-V4-Flash-FP8 \ --num-nodes 1 --num-gpus-per-node 8 - python scripts/run_deepseek_v4.py prepare-cp --model-name DeepSeek-V4-Flash-FP8 python scripts/run_deepseek_v4.py train --model-name DeepSeek-V4-Flash-FP8 \ --num-nodes 4 --num-gpus-per-node 8 \ --hf-checkpoint /root/models/DeepSeek-V4-Flash-FP8 @@ -236,23 +235,17 @@ def prepare_spmd(args: ScriptArgs): _prepare_spmd(args) -@app.command() -@U.dataclass_cli -def prepare_cp(args: ScriptArgs): - _prepare_cp(args) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: + if args.model_local_dir == args.model_dir: + return {} - -def _prepare_cp(args: ScriptArgs): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.torch_dist_name}", - path_dst=f"{args.model_local_dir}/{args.torch_dist_name}", - num_nodes=args.num_nodes, - ) - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}", - path_dst=f"{args.model_local_dir}/{args.model_name}", - num_nodes=args.num_nodes, - ) + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.torch_dist_name}", f"{args.model_local_dir}/{args.torch_dist_name}" + ), + U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"), + ] + return {"trainer": " && ".join(copies)} def _get_parallel_config(args: ScriptArgs) -> str: @@ -516,11 +509,11 @@ def _train(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, extra_env_vars={**extra_env_vars}, megatron_path=args.megatron_path, + prepare_cmd=_prepare_cmd(args), ) @@ -550,11 +543,6 @@ def full_train(args: ScriptArgs): else: print(f"[full_train] Skipping BF16->torch_dist conversion: {torch_dist_sentinel} already exists.") - if args.model_local_dir != args.model_dir: - _prepare_cp(args) - else: - print(f"[full_train] Skipping rsync: model_local_dir == model_dir ({args.model_dir})") - if args.hf_checkpoint is None: args.hf_checkpoint = f"{args.model_local_dir}/{args.model_name}" diff --git a/scripts/run_deepseek.py b/scripts/run_deepseek.py index 24f8f71fae..d81c0c34aa 100644 --- a/scripts/run_deepseek.py +++ b/scripts/run_deepseek.py @@ -4,7 +4,6 @@ import re from dataclasses import dataclass -from functools import partial from typing import Literal import typer @@ -93,15 +92,15 @@ def _prepare_megatron_ckpt(args: ScriptArgs): ) -def _prepare_cp(args: ScriptArgs): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}_torch_dist", - path_dst=f"{args.model_local_dir}/{args.model_name}_torch_dist", - ) - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}", - path_dst=f"{args.model_local_dir}/{args.model_name}", - ) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.model_name}_torch_dist", + f"{args.model_local_dir}/{args.model_name}_torch_dist", + ), + U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"), + ] + return {"trainer": " && ".join(copies)} def _execute_train(args: ScriptArgs, before_ray_job_submit=None): @@ -299,27 +298,37 @@ def _execute_train(args: ScriptArgs, before_ray_job_submit=None): U.execute_train( train_args=train_args, - config=args, # TODO may get it from `config` num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, extra_env_vars={**sglang_extra_env_vars}, megatron_path=args.megatron_path, before_ray_job_submit=before_ray_job_submit, + prepare_cmd=_prepare_cmd(args), ) @app.command() @U.dataclass_cli -def train(args: ScriptArgs): +def full_train(args: ScriptArgs) -> None: _prepare_download(args) _prepare_bf16_ckpt(args) - _execute_train(args, before_ray_job_submit=partial(_prepare_ray_dependent, args)) + _prepare_megatron_ckpt(args) + _execute_train(args) -def _prepare_ray_dependent(args: ScriptArgs): +@app.command() +@U.dataclass_cli +def prepare(args: ScriptArgs) -> None: + _prepare_download(args) + _prepare_bf16_ckpt(args) _prepare_megatron_ckpt(args) - _prepare_cp(args) + + +@app.command() +@U.dataclass_cli +def train(args: ScriptArgs) -> None: + _execute_train(args) @app.callback() diff --git a/scripts/run_deepseek_v32.py b/scripts/run_deepseek_v32.py index 62d93ba49d..3536a6468e 100644 --- a/scripts/run_deepseek_v32.py +++ b/scripts/run_deepseek_v32.py @@ -1,6 +1,4 @@ from dataclasses import dataclass -from functools import partial -from pathlib import Path from typing import Literal import typer @@ -144,15 +142,9 @@ def _prepare_megatron_ckpt(args: ScriptArgs): ) -def _prepare_cp(args: ScriptArgs, skip_existing: bool = False): +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: if args.use_single_node: - return - torch_dist_dst = f"{args.model_local_dir}/{args.model_name}_torch_dist" - if not (skip_existing and Path(torch_dist_dst).exists()): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}_torch_dist", - path_dst=torch_dist_dst, - ) + return {} if args.rollout_mxfp8: hf_suffix = "-MXFP8" @@ -161,12 +153,15 @@ def _prepare_cp(args: ScriptArgs, skip_existing: bool = False): else: hf_suffix = "" hf_name = f"{args.model_name}{hf_suffix}" - hf_dst = f"{args.model_local_dir}/{hf_name}" - if not (skip_existing and Path(hf_dst).exists()): - U.rsync_simple( - path_src=f"{args.model_dir}/{hf_name}", - path_dst=hf_dst, - ) + + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.model_name}_torch_dist", + f"{args.model_local_dir}/{args.model_name}_torch_dist", + ), + U.rsync_cmd(f"{args.model_dir}/{hf_name}", f"{args.model_local_dir}/{hf_name}"), + ] + return {"trainer": " && ".join(copies)} def _execute_train(args: ScriptArgs, before_ray_job_submit=None): @@ -381,7 +376,9 @@ def _execute_train(args: ScriptArgs, before_ray_job_submit=None): config: "bf16" """.strip() if "--te-precision-config-file" not in args.extra_args: - misc_args += f"--te-precision-config-file {U.encode_pseudo_file(te_precision_config_text)} " + misc_args += ( + f"--te-precision-config-file {U.encode_pseudo_file(te_precision_config_text)} " + ) else: if args.use_single_node: sglang_world_size = 2 @@ -441,12 +438,12 @@ def _execute_train(args: ScriptArgs, before_ray_job_submit=None): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, extra_env_vars={**misc_env_vars}, megatron_path=args.megatron_path, before_ray_job_submit=before_ray_job_submit, + prepare_cmd=_prepare_cmd(args), ) @@ -458,7 +455,8 @@ def full_train(args: ScriptArgs): _prepare_bf16_ckpt(args) _prepare_mxfp8_ckpt(args) _prepare_fp8_ckpt(args) - _execute_train(args, before_ray_job_submit=partial(_prepare_megatron_ckpt, args)) + _prepare_megatron_ckpt(args) + _execute_train(args) @app.command() @@ -478,17 +476,10 @@ def prepare_megatron_ckpt(args: ScriptArgs): _prepare_megatron_ckpt(args) -@app.command() -@U.dataclass_cli -def prepare_cp(args: ScriptArgs): - """Copy model/checkpoint to local storage (run on each node).""" - _prepare_cp(args) - - @app.command() @U.dataclass_cli def train(args: ScriptArgs): - """Run training only (assumes prepare and optional prepare-cp are done).""" + """Run training only (assumes prepare is done).""" _execute_train(args) diff --git a/scripts/run_deepseek_v4.py b/scripts/run_deepseek_v4.py index 7c760b107f..00d298ec5b 100644 --- a/scripts/run_deepseek_v4.py +++ b/scripts/run_deepseek_v4.py @@ -21,14 +21,13 @@ --model-name DeepSeek-V4-Flash-FP8-4layer \ --num-nodes 1 --num-gpus-per-node 8 - 2. Individual steps (download [-> MXFP4->FP8] -> FP8->BF16 -> BF16->torch_dist -> rsync -> train): + 2. Individual steps (download [-> MXFP4->FP8] -> FP8->BF16 -> BF16->torch_dist -> train): python scripts/run_deepseek_v4.py prepare-download --model-name DeepSeek-V4-Flash-FP8 python scripts/run_deepseek_v4.py prepare-fp8 --model-name DeepSeek-V4-Flash-0731 python scripts/run_deepseek_v4.py prepare-single --model-name DeepSeek-V4-Flash-FP8 \ --hf-checkpoint /root/models/DeepSeek-V4-Flash-FP8 python scripts/run_deepseek_v4.py prepare-spmd --model-name DeepSeek-V4-Flash-FP8 \ --num-nodes 8 --num-gpus-per-node 8 - python scripts/run_deepseek_v4.py prepare-cp --model-name DeepSeek-V4-Flash-FP8 python scripts/run_deepseek_v4.py train --model-name DeepSeek-V4-Flash-FP8 \ --num-nodes 8 --num-gpus-per-node 8 \ --hf-checkpoint /root/models/DeepSeek-V4-Flash-FP8 @@ -366,23 +365,19 @@ def prepare_spmd(args: ScriptArgs): _prepare_spmd(args) -@app.command() -@U.dataclass_cli -def prepare_cp(args: ScriptArgs): - _prepare_cp(args) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: + if args.model_local_dir == args.model_dir: + return {} - -def _prepare_cp(args: ScriptArgs): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.torch_dist_name}", - path_dst=f"{args.model_local_dir}/{args.torch_dist_name}", - num_nodes=args.num_nodes, - ) - U.rsync_simple( - path_src=f"{args.model_dir}/{args.rollout_name}", - path_dst=f"{args.model_local_dir}/{args.rollout_name}", - num_nodes=args.num_nodes, - ) + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.torch_dist_name}", f"{args.model_local_dir}/{args.torch_dist_name}" + ), + U.rsync_cmd( + f"{args.model_dir}/{args.rollout_name}", f"{args.model_local_dir}/{args.rollout_name}" + ), + ] + return {"trainer": " && ".join(copies)} def _get_parallel_config(args: ScriptArgs) -> str: @@ -730,11 +725,11 @@ def _train(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, extra_env_vars={**extra_env_vars}, megatron_path=args.megatron_path, + prepare_cmd=_prepare_cmd(args), ) @@ -778,11 +773,6 @@ def full_train(args: ScriptArgs): else: print(f"[full_train] Skipping BF16->torch_dist conversion: {torch_dist_sentinel} already exists.") - if args.model_local_dir != args.model_dir: - _prepare_cp(args) - else: - print(f"[full_train] Skipping rsync: model_local_dir == model_dir ({args.model_dir})") - if args.hf_checkpoint is None: args.hf_checkpoint = f"{args.model_local_dir}/{args.rollout_name}" diff --git a/scripts/run_glm45_355b_a32b.py b/scripts/run_glm45_355b_a32b.py index b9ac5ecabf..7defaa98d1 100644 --- a/scripts/run_glm45_355b_a32b.py +++ b/scripts/run_glm45_355b_a32b.py @@ -86,20 +86,21 @@ def _prepare_megatron_ckpt(args: ScriptArgs): ) -def _prepare_cp(args: ScriptArgs): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}_torch_dist", - path_dst=f"{args.model_local_dir}/{args.model_name}_torch_dist", - ) - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}", - path_dst=f"{args.model_local_dir}/{args.model_name}", - ) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.model_name}_torch_dist", + f"{args.model_local_dir}/{args.model_name}_torch_dist", + ), + U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"), + ] if args.rollout_fp8: - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}-FP8", - path_dst=f"{args.model_local_dir}/{args.model_name}-FP8", + copies.append( + U.rsync_cmd( + f"{args.model_dir}/{args.model_name}-FP8", f"{args.model_local_dir}/{args.model_name}-FP8" + ) ) + return {"trainer": " && ".join(copies)} def _execute_train(args: ScriptArgs): @@ -353,7 +354,6 @@ def _execute_train(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, megatron_path=args.megatron_path, @@ -382,15 +382,28 @@ def _execute_train(args: ScriptArgs): # "OMPI_MCA_oob_tcp_if_include": "${MLP_SOCKET_IFNAME}", # "OMPI_MCA_btl_tcp_if_include": "${MLP_SOCKET_IFNAME}", }, + prepare_cmd=_prepare_cmd(args), ) @app.command() @U.dataclass_cli -def train(args: ScriptArgs): +def full_train(args: ScriptArgs) -> None: + _prepare_download(args) + _prepare_megatron_ckpt(args) + _execute_train(args) + + +@app.command() +@U.dataclass_cli +def prepare(args: ScriptArgs) -> None: _prepare_download(args) _prepare_megatron_ckpt(args) - _prepare_cp(args) + + +@app.command() +@U.dataclass_cli +def train(args: ScriptArgs) -> None: _execute_train(args) diff --git a/scripts/run_glm5_2_744b_a40b.py b/scripts/run_glm5_2_744b_a40b.py index 73a69eeede..99f37caa61 100644 --- a/scripts/run_glm5_2_744b_a40b.py +++ b/scripts/run_glm5_2_744b_a40b.py @@ -48,10 +48,8 @@ 3. Download model/data + validate checkpoint + convert to megatron (run on head node). `python scripts/run_glm5_2_744b_a40b.py prepare --model-name GLM-5.2 --num-nodes 32 --fp8-rollout` - 4. (Optional) Copy model from shared NFS to local disk on each node. - python scripts/run_glm5_2_744b_a40b.py prepare-cp --model-name GLM-5.2 --num-nodes 32 - - 5. Run training. Execute on head node; uses Ray internally for distributed training. + 4. Run training; it first copies the model from shared NFS to each node's local disk. + Execute on head node; uses Ray internally for distributed training. python scripts/run_glm5_2_744b_a40b.py train --model-name GLM-5.2 --num-nodes 32 --fp8-rollout --enable-mtp """ @@ -219,20 +217,16 @@ def _prepare_megatron_ckpt(args: ScriptArgs): ) -def _prepare_cp(args: ScriptArgs, skip_existing: bool = False): - torch_dist_dst = f"{args.model_local_dir}/{args.model_name}_torch_dist" - if not (skip_existing and Path(torch_dist_dst).exists()): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}_torch_dist", - path_dst=torch_dist_dst, - ) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: hf_name = f"{args.model_name}_fp8" if args.fp8_rollout else args.model_name - hf_dst = f"{args.model_local_dir}/{hf_name}" - if not (skip_existing and Path(hf_dst).exists()): - U.rsync_simple( - path_src=f"{args.model_dir}/{hf_name}", - path_dst=hf_dst, - ) + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.model_name}_torch_dist", + f"{args.model_local_dir}/{args.model_name}_torch_dist", + ), + U.rsync_cmd(f"{args.model_dir}/{hf_name}", f"{args.model_local_dir}/{hf_name}"), + ] + return {"trainer": " && ".join(copies)} def _execute_train(args: ScriptArgs): @@ -465,7 +459,6 @@ def _execute_train(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, extra_env_vars={ @@ -474,6 +467,7 @@ def _execute_train(args: ScriptArgs): "NVSHMEM_DISABLE_NCCL": "1", }, megatron_path=args.megatron_path, + prepare_cmd=_prepare_cmd(args), ) @@ -486,7 +480,6 @@ def full_train(args: ScriptArgs): if args.fp8_rollout: _convert_to_fp8(args) _prepare_megatron_ckpt(args) - _prepare_cp(args, skip_existing=True) _execute_train(args) @@ -501,13 +494,6 @@ def prepare(args: ScriptArgs): _prepare_megatron_ckpt(args) -@app.command() -@U.dataclass_cli -def prepare_cp(args: ScriptArgs): - """Copy model to local storage (run on all nodes via run_spmd).""" - _prepare_cp(args) - - @app.command() @U.dataclass_cli def train(args: ScriptArgs): diff --git a/scripts/run_glm5_744b_a40b.py b/scripts/run_glm5_744b_a40b.py index 39b06f12d3..2236b68607 100644 --- a/scripts/run_glm5_744b_a40b.py +++ b/scripts/run_glm5_744b_a40b.py @@ -45,11 +45,8 @@ Run on **head node**; megatron conversion uses Ray to coordinate multi-node work. `python scripts/run_glm5_744b_a40b.py prepare --model-name GLM-5_20layer --num-nodes 6` - 4. (Optional) Copy model from shared NFS to local disk on each node. - Run independently on every node. - python scripts/run_glm5_744b_a40b.py prepare-cp --model-name GLM-5_20layer --num-nodes 6 - - 5. Run training. Execute on head node; uses Ray internally for distributed training. + 4. Run training; it first copies the model from shared NFS to each node's local disk. + Execute on head node; uses Ray internally for distributed training. python scripts/run_glm5_744b_a40b.py train --model-name GLM-5_20layer --num-nodes 6 """ @@ -200,20 +197,16 @@ def _prepare_megatron_ckpt(args: ScriptArgs): ) -def _prepare_cp(args: ScriptArgs, skip_existing: bool = False): - torch_dist_dst = f"{args.model_local_dir}/{args.model_name}_torch_dist" - if not (skip_existing and Path(torch_dist_dst).exists()): - U.rsync_simple( - path_src=f"{args.model_dir}/{args.model_name}_torch_dist", - path_dst=torch_dist_dst, - ) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: hf_name = f"{args.model_name}_fp8" if args.fp8_rollout else args.model_name - hf_dst = f"{args.model_local_dir}/{hf_name}" - if not (skip_existing and Path(hf_dst).exists()): - U.rsync_simple( - path_src=f"{args.model_dir}/{hf_name}", - path_dst=hf_dst, - ) + copies = [ + U.rsync_cmd( + f"{args.model_dir}/{args.model_name}_torch_dist", + f"{args.model_local_dir}/{args.model_name}_torch_dist", + ), + U.rsync_cmd(f"{args.model_dir}/{hf_name}", f"{args.model_local_dir}/{hf_name}"), + ] + return {"trainer": " && ".join(copies)} def _execute_train(args: ScriptArgs): @@ -407,7 +400,6 @@ def _execute_train(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, extra_env_vars={ @@ -416,6 +408,7 @@ def _execute_train(args: ScriptArgs): "NVSHMEM_DISABLE_NCCL": "1", }, megatron_path=args.megatron_path, + prepare_cmd=_prepare_cmd(args), ) @@ -428,7 +421,6 @@ def full_train(args: ScriptArgs): if args.fp8_rollout: _convert_to_fp8(args) _prepare_megatron_ckpt(args) - _prepare_cp(args, skip_existing=True) _execute_train(args) @@ -443,13 +435,6 @@ def prepare(args: ScriptArgs): _prepare_megatron_ckpt(args) -@app.command() -@U.dataclass_cli -def prepare_cp(args: ScriptArgs): - """Copy model to local storage (run on all nodes via run_spmd).""" - _prepare_cp(args) - - @app.command() @U.dataclass_cli def train(args: ScriptArgs): diff --git a/scripts/run_inkling.py b/scripts/run_inkling.py index 131a946442..45f616b674 100644 --- a/scripts/run_inkling.py +++ b/scripts/run_inkling.py @@ -40,19 +40,12 @@ Usage patterns: - 1. Train on pre-staged checkpoints: + 1. Train, copying torch_dist to node-local NVMe first when torch_dist_local differs: python scripts/run_inkling.py train \ --model-name Inkling --train-mode full --task dapo_math \ --num-nodes 16 --num-gpus-per-node 4 - 2. Individual steps (rsync shared -> node-local NVMe, then train): - python scripts/run_inkling.py prepare-cp --model-name Inkling - python scripts/run_inkling.py train --model-name Inkling ... - - 3. One-shot (prepare-cp when model_local_dir differs, then train): - python scripts/run_inkling.py full-train --model-name Inkling ... - - 4. Fully-async disaggregated training: + 2. Fully-async disaggregated training: MILES_SCRIPT_NUM_NODES=12 python scripts/run_inkling.py train \ --model-name Inkling --fully-async --rollout-num-nodes 4 \ --num-gpus-per-node 4 --num-rollout 100 @@ -412,41 +405,31 @@ def _train(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=_MODEL_REGISTRY[args.model_name], train_script="train_async.py" if args.fully_async else "train.py", extra_env_vars=extra_env_vars, megatron_path=args.megatron_path, + prepare_cmd=_prepare_cmd(args), ) @app.command() @U.dataclass_cli def train(args: ScriptArgs): - """Run training. Assumes HF checkpoint / torch_dist are already staged.""" + """Run training; each trainer pod first copies torch_dist to node-local NVMe when the two differ.""" _train(args) -def _prepare_cp(args: ScriptArgs): - U.rsync_simple(path_src=args.torch_dist, path_dst=args.torch_dist_local) - - -@app.command() -@U.dataclass_cli -def prepare_cp(args: ScriptArgs): - """Copy the shared torch_dist checkpoint to node-local NVMe (torch_dist_local).""" - _prepare_cp(args) +def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: + if args.torch_dist_local == args.torch_dist: + return {} + return {"trainer": U.rsync_cmd(args.torch_dist, args.torch_dist_local)} -@app.command() -@U.dataclass_cli -def full_train(args: ScriptArgs): - if args.torch_dist_local != args.torch_dist: - _prepare_cp(args) - else: - print(f"[full_train] Skipping rsync: torch_dist_local == torch_dist ({args.torch_dist})") - _train(args) +@app.callback() +def _callback() -> None: + pass if __name__ == "__main__": diff --git a/scripts/run_qwen3_4b.py b/scripts/run_qwen3_4b.py index 793287c87b..83f6d06ebb 100644 --- a/scripts/run_qwen3_4b.py +++ b/scripts/run_qwen3_4b.py @@ -10,6 +10,8 @@ get_megatron_model_type, ) +app = typer.Typer() + @dataclass class ScriptArgs(U.ExecuteTrainConfig): @@ -61,7 +63,7 @@ def __post_init__(self): ) -def prepare(args: ScriptArgs): +def _prepare_download(args: ScriptArgs): U.exec_command_cpu(f"mkdir -p {args.model_dir} {args.data_dir}") U.exec_command_cpu(f"hf download Qwen/{args.model_name} --local-dir {args.model_dir}/{args.model_name}") U.hf_download_dataset("zhuzilin/dapo-math-17k", data_dir=args.data_dir) @@ -76,6 +78,8 @@ def prepare(args: ScriptArgs): f"hf download Qwen/{args.model_name}-FP8 --local-dir {args.model_dir}/{args.model_name}-FP8" ) + +def _prepare_megatron_ckpt(args: ScriptArgs): if (args.train_backend == "megatron") and not args.enable_megatron_bridge: U.convert_checkpoint( model_name=args.model_name, @@ -87,7 +91,7 @@ def prepare(args: ScriptArgs): ) -def execute(args: ScriptArgs): +def _execute_train(args: ScriptArgs): is_debug_mode = args.mode != "normal" is_debug_one_sample = args.mode == "debug_one_sample" model_parallel_size = ( @@ -309,7 +313,6 @@ def execute(args: ScriptArgs): U.execute_train( train_args=train_args, - config=args, # TODO may get it from `config` num_gpus_per_node=args.num_gpus_per_node, megatron_model_type=args.megatron_model_type, @@ -321,11 +324,31 @@ def execute(args: ScriptArgs): ) +@app.command() +@U.dataclass_cli +def full_train(args: ScriptArgs) -> None: + _prepare_download(args) + _prepare_megatron_ckpt(args) + _execute_train(args) + + +@app.command() @U.dataclass_cli -def main(args: ScriptArgs): - prepare(args) - execute(args) +def prepare(args: ScriptArgs) -> None: + _prepare_download(args) + _prepare_megatron_ckpt(args) + + +@app.command() +@U.dataclass_cli +def train(args: ScriptArgs) -> None: + _execute_train(args) + + +@app.callback() +def _callback() -> None: + pass if __name__ == "__main__": - typer.run(main) + app() diff --git a/tests/snapshots/launch_scripts/py/scripts/amd/run_deepseek_v4.py/prepare_cp.txt b/tests/snapshots/launch_scripts/py/scripts/amd/run_deepseek_v4.py/prepare_cp.txt deleted file mode 100644 index a55d0d820d..0000000000 --- a/tests/snapshots/launch_scripts/py/scripts/amd/run_deepseek_v4.py/prepare_cp.txt +++ /dev/null @@ -1,7 +0,0 @@ -### 0 -[multi_node num_nodes=1] mkdir -p /root/models/DeepSeek-V4-Flash-FP8_torch_dist && rsync -a - --info=progress2 /root/models/DeepSeek-V4-Flash-FP8_torch_dist/ /root/models/DeepSeek-V4-Flash-FP8_torch_dist - -### 1 -[multi_node num_nodes=1] mkdir -p /root/models/DeepSeek-V4-Flash-FP8 && rsync -a - --info=progress2 /root/models/DeepSeek-V4-Flash-FP8/ /root/models/DeepSeek-V4-Flash-FP8 diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/full_train.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/full_train.txt new file mode 100644 index 0000000000..1690b65ca2 --- /dev/null +++ b/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/full_train.txt @@ -0,0 +1,228 @@ +### 0 +mkdir -p /root/models /root/datasets + +### 1 +hf download deepseek-ai/DeepSeek-V3 + --local-dir /root/models/DeepSeek-V3 + +### 2 +hf download + --repo-type dataset zhuzilin/dapo-math-17k + --local-dir /root/datasets/dapo-math-17k + +### 3 +hf download + --repo-type dataset zhuzilin/aime-2024 + --local-dir /root/datasets/aime-2024 + +### 4 +python /tools/fp8_cast_bf16.py + --input-fp8-hf-path /root/models/DeepSeek-V3 + --output-bf16-hf-path /root/models/DeepSeek-V3-bf16/ + +### 5 +[multi_node num_nodes=None] PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun + --nproc-per-node 4 + --master-addr {{master_addr}} + --master-port 23456 + --nnodes={{nnodes}} + --node-rank {{node_rank}} /tools/convert_hf_to_torch_dist.py + --disable-bias-linear + --num-layers 61 + --hidden-size 7168 + --ffn-hidden-size 18432 + --num-attention-heads 128 + --kv-channels 128 + --normalization RMSNorm + --position-embedding-type rope + --norm-epsilon 1e-6 + --swiglu + --untie-embeddings-and-output-weights + --vocab-size 129280 + --multi-latent-attention + --q-lora-rank 1536 + --kv-lora-rank 512 + --qk-head-dim 128 + --qk-pos-emb-head-dim 64 + --v-head-dim 128 + --qk-layernorm + --rotary-scaling-factor 40 + --rotary-base 10000 + --mscale 1.0 + --mscale-all-dim 1.0 + --attention-softmax-in-fp32 + --no-rope-fusion + --num-experts 256 + --moe-layer-freq '[0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]' + --moe-ffn-hidden-size 2048 + --moe-router-topk 8 + --moe-shared-expert-intermediate-size 2048 + --moe-router-pre-softmax + --moe-router-score-function sigmoid + --moe-router-enable-expert-bias + --moe-router-load-balancing-type seq_aux_loss + --moe-token-dispatcher-type alltoall + --moe-aux-loss-coeff 0 + --moe-router-bias-update-rate 0 + --moe-router-group-topk 4 + --moe-router-num-groups 8 + --moe-grouped-gemm + --moe-router-topk-scaling-factor 2.5 + --moe-router-dtype fp32 + --moe-permute-fusion + --hf-checkpoint /root/models/DeepSeek-V3-bf16 + --save /root/models/DeepSeek-V3_torch_dist + --tensor-model-parallel-size 1 + --expert-tensor-parallel-size 1 + --pipeline-model-parallel-size 8 + --expert-model-parallel-size 4 + --decoder-first-pipeline-num-layers 7 + --decoder-last-pipeline-num-layers 6 + +### 6 +pkill -9 sglang; sleep 3; ray stop + --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; + +### 7 +export PYTHONUNBUFFERED=1 && ray start + --head + --node-ip-address 127.0.0.1 + --num-gpus 4 + --disable-usage-stats + +### 8 +[multi_node num_nodes=None] mkdir -p /root/local_data/DeepSeek-V3_torch_dist && rsync -a + --info=progress2 /root/models/DeepSeek-V3_torch_dist/ /root/local_data/DeepSeek-V3_torch_dist && mkdir -p /root/local_data/DeepSeek-V3 && rsync -a + --info=progress2 /root/models/DeepSeek-V3/ /root/local_data/DeepSeek-V3 + +### 9 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 10 +export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit + --address="http://127.0.0.1:8265" + --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "256", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' + -- python3 /train.py + --disable-bias-linear + --num-layers 61 + --hidden-size 7168 + --ffn-hidden-size 18432 + --num-attention-heads 128 + --kv-channels 128 + --normalization RMSNorm + --position-embedding-type rope + --norm-epsilon 1e-6 + --swiglu + --untie-embeddings-and-output-weights + --vocab-size 129280 + --multi-latent-attention + --q-lora-rank 1536 + --kv-lora-rank 512 + --qk-head-dim 128 + --qk-pos-emb-head-dim 64 + --v-head-dim 128 + --qk-layernorm + --rotary-scaling-factor 40 + --rotary-base 10000 + --mscale 1.0 + --mscale-all-dim 1.0 + --attention-softmax-in-fp32 + --no-rope-fusion + --num-experts 256 + --moe-layer-freq '[0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]' + --moe-ffn-hidden-size 2048 + --moe-router-topk 8 + --moe-shared-expert-intermediate-size 2048 + --moe-router-pre-softmax + --moe-router-score-function sigmoid + --moe-router-enable-expert-bias + --moe-router-load-balancing-type seq_aux_loss + --moe-token-dispatcher-type alltoall + --moe-aux-loss-coeff 0 + --moe-router-bias-update-rate 0 + --moe-router-group-topk 4 + --moe-router-num-groups 8 + --moe-grouped-gemm + --moe-router-topk-scaling-factor 2.5 + --moe-router-dtype fp32 + --moe-permute-fusion + --hf-checkpoint /root/local_data/DeepSeek-V3 + --ref-load /root/local_data/DeepSeek-V3_torch_dist + --load /root/shared_data/260101-000000-000/checkpoints + --save /root/shared_data/260101-000000-000/checkpoints + --save-interval 20 + --save-retain-interval 20 + --label-key label + --apply-chat-template + --rollout-shuffle + --rm-type math + --num-rollout 3000 + --rollout-batch-size 128 + --n-samples-per-prompt 8 + --rollout-temperature 1 + --num-steps-per-rollout 4 + --balance-data + --over-sampling-batch-size 256 + --dynamic-sampling-filter-path miles.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std + --prompt-data /root/datasets/dapo-math-17k/dapo-math-17k.jsonl + --input-key prompt + --rollout-max-response-len 32768 + --optimizer adam + --lr 1e-6 + --lr-decay-style constant + --weight-decay 0.1 + --adam-beta1 0.9 + --adam-beta2 0.98 + --advantage-estimator grpo + --kl-loss-coef 0.00 + --kl-loss-type low_var_kl + --entropy-coef 0.00 + --eps-clip 0.2 + --eps-clip-high 0.28 + --use-wandb + --wandb-project miles-run_deepseek + --wandb-group 260101-000000-000 + --wandb-key 'frozen-wandb-api-key' + --disable-wandb-random-suffix + --tensor-model-parallel-size 1 + --sequence-parallel + --pipeline-model-parallel-size 1 + --context-parallel-size 4 + --expert-model-parallel-size 4 + --expert-tensor-parallel-size 1 + --recompute-granularity full + --recompute-method uniform + --recompute-num-layers 1 + --use-dynamic-batch-size + --max-tokens-per-gpu 2048 + --eval-interval 20 + --eval-top-p 1 + --eval-prompt-data aime /root/datasets/aime-2024/aime-2024.jsonl + --n-samples-per-eval-prompt 8 + --eval-max-response-len 32768 + --rollout-num-gpus-per-engine 4 + --sglang-mem-fraction-static 0.7 + --sglang-tp-size 4 + --sglang-ep-size 4 + --sglang-enable-dp-attention + --sglang-dp-size 1 + --sglang-moe-dense-tp-size 1 + --sglang-enable-dp-lm-head + --sglang-moe-a2a-backend deepep + --sglang-deepep-mode low_latency + --sglang-server-concurrency 1024 + --sglang-max-running-requests 256 + --sglang-chunked-prefill-size 1024 + --sglang-cuda-graph-max-bs 256 + --attention-dropout 0.0 + --hidden-dropout 0.0 + --accumulate-allreduce-grads-in-fp32 + --attention-softmax-in-fp32 + --update-weight-buffer-size 4294967296 + --actor-num-nodes 1 + --actor-num-gpus-per-node 4 + --num-gpus-per-node 4 + --colocate + --use-fault-tolerance + --dump-details /root/shared_data/260101-000000-000/dump_details + --disable-weights-backuper diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/prepare.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/prepare.txt new file mode 100644 index 0000000000..de1c065579 --- /dev/null +++ b/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/prepare.txt @@ -0,0 +1,80 @@ +### 0 +mkdir -p /root/models /root/datasets + +### 1 +hf download deepseek-ai/DeepSeek-V3 + --local-dir /root/models/DeepSeek-V3 + +### 2 +hf download + --repo-type dataset zhuzilin/dapo-math-17k + --local-dir /root/datasets/dapo-math-17k + +### 3 +hf download + --repo-type dataset zhuzilin/aime-2024 + --local-dir /root/datasets/aime-2024 + +### 4 +python /tools/fp8_cast_bf16.py + --input-fp8-hf-path /root/models/DeepSeek-V3 + --output-bf16-hf-path /root/models/DeepSeek-V3-bf16/ + +### 5 +[multi_node num_nodes=None] PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun + --nproc-per-node 4 + --master-addr {{master_addr}} + --master-port 23456 + --nnodes={{nnodes}} + --node-rank {{node_rank}} /tools/convert_hf_to_torch_dist.py + --disable-bias-linear + --num-layers 61 + --hidden-size 7168 + --ffn-hidden-size 18432 + --num-attention-heads 128 + --kv-channels 128 + --normalization RMSNorm + --position-embedding-type rope + --norm-epsilon 1e-6 + --swiglu + --untie-embeddings-and-output-weights + --vocab-size 129280 + --multi-latent-attention + --q-lora-rank 1536 + --kv-lora-rank 512 + --qk-head-dim 128 + --qk-pos-emb-head-dim 64 + --v-head-dim 128 + --qk-layernorm + --rotary-scaling-factor 40 + --rotary-base 10000 + --mscale 1.0 + --mscale-all-dim 1.0 + --attention-softmax-in-fp32 + --no-rope-fusion + --num-experts 256 + --moe-layer-freq '[0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]' + --moe-ffn-hidden-size 2048 + --moe-router-topk 8 + --moe-shared-expert-intermediate-size 2048 + --moe-router-pre-softmax + --moe-router-score-function sigmoid + --moe-router-enable-expert-bias + --moe-router-load-balancing-type seq_aux_loss + --moe-token-dispatcher-type alltoall + --moe-aux-loss-coeff 0 + --moe-router-bias-update-rate 0 + --moe-router-group-topk 4 + --moe-router-num-groups 8 + --moe-grouped-gemm + --moe-router-topk-scaling-factor 2.5 + --moe-router-dtype fp32 + --moe-permute-fusion + --hf-checkpoint /root/models/DeepSeek-V3-bf16 + --save /root/models/DeepSeek-V3_torch_dist + --tensor-model-parallel-size 1 + --expert-tensor-parallel-size 1 + --pipeline-model-parallel-size 8 + --expert-model-parallel-size 4 + --decoder-first-pipeline-num-layers 7 + --decoder-last-pipeline-num-layers 6 diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/train.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/train.txt index 1d5398d012..a5f479e3e0 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_deepseek.py/train.txt @@ -1,107 +1,23 @@ ### 0 -mkdir -p /root/models /root/datasets - -### 1 -hf download deepseek-ai/DeepSeek-V3 - --local-dir /root/models/DeepSeek-V3 - -### 2 -hf download - --repo-type dataset zhuzilin/dapo-math-17k - --local-dir /root/datasets/dapo-math-17k - -### 3 -hf download - --repo-type dataset zhuzilin/aime-2024 - --local-dir /root/datasets/aime-2024 - -### 4 -python /tools/fp8_cast_bf16.py - --input-fp8-hf-path /root/models/DeepSeek-V3 - --output-bf16-hf-path /root/models/DeepSeek-V3-bf16/ - -### 5 pkill -9 sglang; sleep 3; ray stop --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; -### 6 +### 1 export PYTHONUNBUFFERED=1 && ray start --head --node-ip-address 127.0.0.1 --num-gpus 4 --disable-usage-stats -### 7 -[multi_node num_nodes=None] PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun - --nproc-per-node 4 - --master-addr {{master_addr}} - --master-port 23456 - --nnodes={{nnodes}} - --node-rank {{node_rank}} /tools/convert_hf_to_torch_dist.py - --disable-bias-linear - --num-layers 61 - --hidden-size 7168 - --ffn-hidden-size 18432 - --num-attention-heads 128 - --kv-channels 128 - --normalization RMSNorm - --position-embedding-type rope - --norm-epsilon 1e-6 - --swiglu - --untie-embeddings-and-output-weights - --vocab-size 129280 - --multi-latent-attention - --q-lora-rank 1536 - --kv-lora-rank 512 - --qk-head-dim 128 - --qk-pos-emb-head-dim 64 - --v-head-dim 128 - --qk-layernorm - --rotary-scaling-factor 40 - --rotary-base 10000 - --mscale 1.0 - --mscale-all-dim 1.0 - --attention-softmax-in-fp32 - --no-rope-fusion - --num-experts 256 - --moe-layer-freq '[0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]' - --moe-ffn-hidden-size 2048 - --moe-router-topk 8 - --moe-shared-expert-intermediate-size 2048 - --moe-router-pre-softmax - --moe-router-score-function sigmoid - --moe-router-enable-expert-bias - --moe-router-load-balancing-type seq_aux_loss - --moe-token-dispatcher-type alltoall - --moe-aux-loss-coeff 0 - --moe-router-bias-update-rate 0 - --moe-router-group-topk 4 - --moe-router-num-groups 8 - --moe-grouped-gemm - --moe-router-topk-scaling-factor 2.5 - --moe-router-dtype fp32 - --moe-permute-fusion - --hf-checkpoint /root/models/DeepSeek-V3-bf16 - --save /root/models/DeepSeek-V3_torch_dist - --tensor-model-parallel-size 1 - --expert-tensor-parallel-size 1 - --pipeline-model-parallel-size 8 - --expert-model-parallel-size 4 - --decoder-first-pipeline-num-layers 7 - --decoder-last-pipeline-num-layers 6 - -### 8 +### 2 [multi_node num_nodes=None] mkdir -p /root/local_data/DeepSeek-V3_torch_dist && rsync -a - --info=progress2 /root/models/DeepSeek-V3_torch_dist/ /root/local_data/DeepSeek-V3_torch_dist - -### 9 -[multi_node num_nodes=None] mkdir -p /root/local_data/DeepSeek-V3 && rsync -a + --info=progress2 /root/models/DeepSeek-V3_torch_dist/ /root/local_data/DeepSeek-V3_torch_dist && mkdir -p /root/local_data/DeepSeek-V3 && rsync -a --info=progress2 /root/models/DeepSeek-V3/ /root/local_data/DeepSeek-V3 -### 10 +### 3 nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l -### 11 +### 4 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "256", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/full_train.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/full_train.txt index 60090eb5bf..c5c481023a 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/full_train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/full_train.txt @@ -91,9 +91,14 @@ export PYTHONUNBUFFERED=1 && ray start --expert-tensor-parallel-size 1 ### 8 -nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l +[multi_node num_nodes=None] mkdir -p /root/models/DeepSeek-V3.2_torch_dist && rsync -a + --info=progress2 /root/models/DeepSeek-V3.2_torch_dist/ /root/models/DeepSeek-V3.2_torch_dist && mkdir -p /root/models/DeepSeek-V3.2 && rsync -a + --info=progress2 /root/models/DeepSeek-V3.2/ /root/models/DeepSeek-V3.2 ### 9 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 10 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_NSA_FORCE_MLA": "1", "SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD": "0", "NVSHMEM_DISABLE_NCCL": "1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/prepare_cp.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/prepare_cp.txt deleted file mode 100644 index 52049a6660..0000000000 --- a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/prepare_cp.txt +++ /dev/null @@ -1,7 +0,0 @@ -### 0 -[multi_node num_nodes=None] mkdir -p /root/models/DeepSeek-V3.2_torch_dist && rsync -a - --info=progress2 /root/models/DeepSeek-V3.2_torch_dist/ /root/models/DeepSeek-V3.2_torch_dist - -### 1 -[multi_node num_nodes=None] mkdir -p /root/models/DeepSeek-V3.2 && rsync -a - --info=progress2 /root/models/DeepSeek-V3.2/ /root/models/DeepSeek-V3.2 diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/train.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/train.txt index 80016da91d..41a245d8da 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v32.py/train.txt @@ -10,9 +10,14 @@ export PYTHONUNBUFFERED=1 && ray start --disable-usage-stats ### 2 -nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l +[multi_node num_nodes=None] mkdir -p /root/models/DeepSeek-V3.2_torch_dist && rsync -a + --info=progress2 /root/models/DeepSeek-V3.2_torch_dist/ /root/models/DeepSeek-V3.2_torch_dist && mkdir -p /root/models/DeepSeek-V3.2 && rsync -a + --info=progress2 /root/models/DeepSeek-V3.2/ /root/models/DeepSeek-V3.2 ### 3 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 4 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_NSA_FORCE_MLA": "1", "SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD": "0", "NVSHMEM_DISABLE_NCCL": "1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v4.py/prepare_cp.txt b/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v4.py/prepare_cp.txt deleted file mode 100644 index 703fd68445..0000000000 --- a/tests/snapshots/launch_scripts/py/scripts/run_deepseek_v4.py/prepare_cp.txt +++ /dev/null @@ -1,7 +0,0 @@ -### 0 -[multi_node num_nodes=1] mkdir -p /root/models/DeepSeek-V4-Flash-FP8-4layer_torch_dist && rsync -a - --info=progress2 /root/models/DeepSeek-V4-Flash-FP8-4layer_torch_dist/ /root/models/DeepSeek-V4-Flash-FP8-4layer_torch_dist - -### 1 -[multi_node num_nodes=1] mkdir -p /root/models/DeepSeek-V4-Flash-FP8-4layer && rsync -a - --info=progress2 /root/models/DeepSeek-V4-Flash-FP8-4layer/ /root/models/DeepSeek-V4-Flash-FP8-4layer diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/full_train.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/full_train.txt new file mode 100644 index 0000000000..f5a80886cf --- /dev/null +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/full_train.txt @@ -0,0 +1,197 @@ +### 0 +mkdir -p /root/models /root/datasets + +### 1 +hf download zai-org/GLM-4.5 + --local-dir /root/models/GLM-4.5 + +### 2 +hf download + --repo-type dataset zhuzilin/dapo-math-17k + --local-dir /root/datasets/dapo-math-17k + +### 3 +hf download + --repo-type dataset zhuzilin/aime-2024 + --local-dir /root/datasets/aime-2024 + +### 4 +hf download + --repo-type dataset zhuzilin/aime-2025 + --local-dir /root/datasets/aime-2025 + +### 5 +[multi_node num_nodes=None] PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun + --nproc-per-node 4 + --master-addr {{master_addr}} + --master-port 23456 + --nnodes={{nnodes}} + --node-rank {{node_rank}} /tools/convert_hf_to_torch_dist.py + --disable-bias-linear + --qk-layernorm + --group-query-attention + --num-attention-heads 96 + --num-query-groups 8 + --kv-channels 128 + --num-layers 92 + --hidden-size 5120 + --ffn-hidden-size 12288 + --add-qkv-bias + --normalization RMSNorm + --position-embedding-type rope + --rotary-percent 0.5 + --swiglu + --untie-embeddings-and-output-weights + --vocab-size 151552 + --rotary-base 1000000 + --moe-ffn-hidden-size 1536 + --moe-shared-expert-intermediate-size 1536 + --moe-router-pre-softmax + --moe-router-score-function sigmoid + --moe-router-enable-expert-bias + --moe-router-bias-update-rate 0 + --moe-router-load-balancing-type seq_aux_loss + --moe-token-dispatcher-type alltoall + --moe-router-topk 8 + --moe-router-topk-scaling-factor 2.5 + --moe-layer-freq '[0]*3+[1]*89' + --num-experts 160 + --moe-grouped-gemm + --moe-router-dtype fp32 + --moe-permute-fusion + --moe-aux-loss-coeff 0 + --hf-checkpoint /root/models/GLM-4.5 + --save /root/models/GLM-4.5_torch_dist + +### 6 +pkill -9 sglang; sleep 3; ray stop + --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; + +### 7 +export PYTHONUNBUFFERED=1 && ray start + --head + --node-ip-address 127.0.0.1 + --num-gpus 4 + --disable-usage-stats + +### 8 +[multi_node num_nodes=None] mkdir -p /root/local_data/GLM-4.5_torch_dist && rsync -a + --info=progress2 /root/models/GLM-4.5_torch_dist/ /root/local_data/GLM-4.5_torch_dist && mkdir -p /root/local_data/GLM-4.5 && rsync -a + --info=progress2 /root/models/GLM-4.5/ /root/local_data/GLM-4.5 + +### 9 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 10 +export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit + --address="http://127.0.0.1:8265" + --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' + -- python3 /train.py + --disable-bias-linear + --qk-layernorm + --group-query-attention + --num-attention-heads 96 + --num-query-groups 8 + --kv-channels 128 + --num-layers 92 + --hidden-size 5120 + --ffn-hidden-size 12288 + --add-qkv-bias + --normalization RMSNorm + --position-embedding-type rope + --rotary-percent 0.5 + --swiglu + --untie-embeddings-and-output-weights + --vocab-size 151552 + --rotary-base 1000000 + --moe-ffn-hidden-size 1536 + --moe-shared-expert-intermediate-size 1536 + --moe-router-pre-softmax + --moe-router-score-function sigmoid + --moe-router-enable-expert-bias + --moe-router-bias-update-rate 0 + --moe-router-load-balancing-type seq_aux_loss + --moe-token-dispatcher-type alltoall + --moe-router-topk 8 + --moe-router-topk-scaling-factor 2.5 + --moe-layer-freq '[0]*3+[1]*89' + --num-experts 160 + --moe-grouped-gemm + --moe-router-dtype fp32 + --moe-permute-fusion + --moe-aux-loss-coeff 0 + --hf-checkpoint /root/local_data/GLM-4.5 + --ref-load /root/local_data/GLM-4.5_torch_dist + --load /root/shared_data/260101-000000-000/checkpoints + --save /root/shared_data/260101-000000-000/checkpoints + --save-interval 10 + --save-retain-interval 10 + --label-key label + --apply-chat-template + --rollout-shuffle + --rm-type math + --num-rollout 3000 + --rollout-batch-size 32 + --n-samples-per-prompt 8 + --rollout-temperature 1 + --num-steps-per-rollout 1 + --balance-data + --rollout-stop-token-ids 151329 151336 151338 + --prompt-data /root/datasets/dapo-math-17k/dapo-math-17k.jsonl + --input-key prompt + --rollout-max-response-len 8192 + --optimizer adam + --lr 1e-6 + --lr-decay-style constant + --weight-decay 0.1 + --adam-beta1 0.9 + --adam-beta2 0.98 + --advantage-estimator grpo + --kl-loss-coef 0.00 + --kl-loss-type low_var_kl + --kl-coef 0.00 + --entropy-coef 0.00 + --eps-clip 1e-4 + --eps-clip-high 2e-4 + --use-tis + --use-wandb + --wandb-project miles-run_glm45_355b_a32b + --wandb-group 260101-000000-000 + --wandb-key 'frozen-wandb-api-key' + --disable-wandb-random-suffix + --tensor-model-parallel-size 4 + --sequence-parallel + --pipeline-model-parallel-size 1 + --context-parallel-size 1 + --expert-model-parallel-size 4 + --expert-tensor-parallel-size 1 + --recompute-granularity full + --recompute-method uniform + --recompute-num-layers 1 + --use-dynamic-batch-size + --max-tokens-per-gpu 16384 + --eval-interval 20 + --eval-top-p 1 + --eval-prompt-data aime /root/datasets/aime-2024/aime-2024.jsonl + --n-samples-per-eval-prompt 8 + --eval-max-response-len 8192 + --rollout-num-gpus-per-engine 4 + --sglang-mem-fraction-static 0.7 + --sglang-tp-size 4 + --sglang-chunked-prefill-size 8192 + --sglang-attention-backend trtllm_mha + --attention-dropout 0.0 + --hidden-dropout 0.0 + --accumulate-allreduce-grads-in-fp32 + --attention-softmax-in-fp32 + --attention-backend flash + --update-weight-buffer-size 2147483648 + --actor-num-nodes 1 + --actor-num-gpus-per-node 4 + --num-gpus-per-node 4 + --colocate + --use-fault-tolerance + --dump-details /root/shared_data/260101-000000-000/dump_details + --disable-weights-backuper + --router-health-success-threshold 1 + --router-health-check-interval-secs 15 diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/prepare.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/prepare.txt new file mode 100644 index 0000000000..fc0e2d4fe5 --- /dev/null +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/prepare.txt @@ -0,0 +1,64 @@ +### 0 +mkdir -p /root/models /root/datasets + +### 1 +hf download zai-org/GLM-4.5 + --local-dir /root/models/GLM-4.5 + +### 2 +hf download + --repo-type dataset zhuzilin/dapo-math-17k + --local-dir /root/datasets/dapo-math-17k + +### 3 +hf download + --repo-type dataset zhuzilin/aime-2024 + --local-dir /root/datasets/aime-2024 + +### 4 +hf download + --repo-type dataset zhuzilin/aime-2025 + --local-dir /root/datasets/aime-2025 + +### 5 +[multi_node num_nodes=None] PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun + --nproc-per-node 4 + --master-addr {{master_addr}} + --master-port 23456 + --nnodes={{nnodes}} + --node-rank {{node_rank}} /tools/convert_hf_to_torch_dist.py + --disable-bias-linear + --qk-layernorm + --group-query-attention + --num-attention-heads 96 + --num-query-groups 8 + --kv-channels 128 + --num-layers 92 + --hidden-size 5120 + --ffn-hidden-size 12288 + --add-qkv-bias + --normalization RMSNorm + --position-embedding-type rope + --rotary-percent 0.5 + --swiglu + --untie-embeddings-and-output-weights + --vocab-size 151552 + --rotary-base 1000000 + --moe-ffn-hidden-size 1536 + --moe-shared-expert-intermediate-size 1536 + --moe-router-pre-softmax + --moe-router-score-function sigmoid + --moe-router-enable-expert-bias + --moe-router-bias-update-rate 0 + --moe-router-load-balancing-type seq_aux_loss + --moe-token-dispatcher-type alltoall + --moe-router-topk 8 + --moe-router-topk-scaling-factor 2.5 + --moe-layer-freq '[0]*3+[1]*89' + --num-experts 160 + --moe-grouped-gemm + --moe-router-dtype fp32 + --moe-permute-fusion + --moe-aux-loss-coeff 0 + --hf-checkpoint /root/models/GLM-4.5 + --save /root/models/GLM-4.5_torch_dist diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/train.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/train.txt index 7cc64f3e27..18f6d355eb 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm45_355b_a32b.py/train.txt @@ -1,91 +1,23 @@ ### 0 -mkdir -p /root/models /root/datasets - -### 1 -hf download zai-org/GLM-4.5 - --local-dir /root/models/GLM-4.5 - -### 2 -hf download - --repo-type dataset zhuzilin/dapo-math-17k - --local-dir /root/datasets/dapo-math-17k - -### 3 -hf download - --repo-type dataset zhuzilin/aime-2024 - --local-dir /root/datasets/aime-2024 - -### 4 -hf download - --repo-type dataset zhuzilin/aime-2025 - --local-dir /root/datasets/aime-2025 - -### 5 -[multi_node num_nodes=None] PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun - --nproc-per-node 4 - --master-addr {{master_addr}} - --master-port 23456 - --nnodes={{nnodes}} - --node-rank {{node_rank}} /tools/convert_hf_to_torch_dist.py - --disable-bias-linear - --qk-layernorm - --group-query-attention - --num-attention-heads 96 - --num-query-groups 8 - --kv-channels 128 - --num-layers 92 - --hidden-size 5120 - --ffn-hidden-size 12288 - --add-qkv-bias - --normalization RMSNorm - --position-embedding-type rope - --rotary-percent 0.5 - --swiglu - --untie-embeddings-and-output-weights - --vocab-size 151552 - --rotary-base 1000000 - --moe-ffn-hidden-size 1536 - --moe-shared-expert-intermediate-size 1536 - --moe-router-pre-softmax - --moe-router-score-function sigmoid - --moe-router-enable-expert-bias - --moe-router-bias-update-rate 0 - --moe-router-load-balancing-type seq_aux_loss - --moe-token-dispatcher-type alltoall - --moe-router-topk 8 - --moe-router-topk-scaling-factor 2.5 - --moe-layer-freq '[0]*3+[1]*89' - --num-experts 160 - --moe-grouped-gemm - --moe-router-dtype fp32 - --moe-permute-fusion - --moe-aux-loss-coeff 0 - --hf-checkpoint /root/models/GLM-4.5 - --save /root/models/GLM-4.5_torch_dist - -### 6 -[multi_node num_nodes=None] mkdir -p /root/local_data/GLM-4.5_torch_dist && rsync -a - --info=progress2 /root/models/GLM-4.5_torch_dist/ /root/local_data/GLM-4.5_torch_dist - -### 7 -[multi_node num_nodes=None] mkdir -p /root/local_data/GLM-4.5 && rsync -a - --info=progress2 /root/models/GLM-4.5/ /root/local_data/GLM-4.5 - -### 8 pkill -9 sglang; sleep 3; ray stop --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; -### 9 +### 1 export PYTHONUNBUFFERED=1 && ray start --head --node-ip-address 127.0.0.1 --num-gpus 4 --disable-usage-stats -### 10 +### 2 +[multi_node num_nodes=None] mkdir -p /root/local_data/GLM-4.5_torch_dist && rsync -a + --info=progress2 /root/models/GLM-4.5_torch_dist/ /root/local_data/GLM-4.5_torch_dist && mkdir -p /root/local_data/GLM-4.5 && rsync -a + --info=progress2 /root/models/GLM-4.5/ /root/local_data/GLM-4.5 + +### 3 nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l -### 11 +### 4 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/full_train.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/full_train.txt index b902f8c17c..c1ec95e92f 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/full_train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/full_train.txt @@ -65,28 +65,25 @@ hf download --decoder-last-pipeline-num-layers 20 ### 4 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5.2_torch_dist && rsync -a - --info=progress2 /models/GLM-5.2_torch_dist/ /root/models/GLM-5.2_torch_dist - -### 5 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5.2 && rsync -a - --info=progress2 /models/GLM-5.2/ /root/models/GLM-5.2 - -### 6 pkill -9 sglang; sleep 3; ray stop --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; -### 7 +### 5 export PYTHONUNBUFFERED=1 && ray start --head --node-ip-address 127.0.0.1 --num-gpus 8 --disable-usage-stats -### 8 +### 6 +[multi_node num_nodes=None] mkdir -p /root/models/GLM-5.2_torch_dist && rsync -a + --info=progress2 /models/GLM-5.2_torch_dist/ /root/models/GLM-5.2_torch_dist && mkdir -p /root/models/GLM-5.2 && rsync -a + --info=progress2 /models/GLM-5.2/ /root/models/GLM-5.2 + +### 7 nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l -### 9 +### 8 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "256", "SGLANG_NSA_FORCE_MLA": "1", "INDEXER_ROPE_NEOX_STYLE": "0", "NVSHMEM_DISABLE_NCCL": "1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/prepare_cp.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/prepare_cp.txt deleted file mode 100644 index 53aa839517..0000000000 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/prepare_cp.txt +++ /dev/null @@ -1,7 +0,0 @@ -### 0 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5.2_torch_dist && rsync -a - --info=progress2 /models/GLM-5.2_torch_dist/ /root/models/GLM-5.2_torch_dist - -### 1 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5.2 && rsync -a - --info=progress2 /models/GLM-5.2/ /root/models/GLM-5.2 diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/train.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/train.txt index 281a2cd342..2f5cc159e8 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm5_2_744b_a40b.py/train.txt @@ -10,9 +10,14 @@ export PYTHONUNBUFFERED=1 && ray start --disable-usage-stats ### 2 -nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l +[multi_node num_nodes=None] mkdir -p /root/models/GLM-5.2_torch_dist && rsync -a + --info=progress2 /models/GLM-5.2_torch_dist/ /root/models/GLM-5.2_torch_dist && mkdir -p /root/models/GLM-5.2 && rsync -a + --info=progress2 /models/GLM-5.2/ /root/models/GLM-5.2 ### 3 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 4 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "256", "SGLANG_NSA_FORCE_MLA": "1", "INDEXER_ROPE_NEOX_STYLE": "0", "NVSHMEM_DISABLE_NCCL": "1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/full_train.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/full_train.txt index 855801768e..c6aebaab6c 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/full_train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/full_train.txt @@ -64,28 +64,25 @@ hf download --decoder-last-pipeline-num-layers 18 ### 4 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5_torch_dist && rsync -a - --info=progress2 /models/GLM-5_torch_dist/ /root/models/GLM-5_torch_dist - -### 5 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5 && rsync -a - --info=progress2 /models/GLM-5/ /root/models/GLM-5 - -### 6 pkill -9 sglang; sleep 3; ray stop --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; -### 7 +### 5 export PYTHONUNBUFFERED=1 && ray start --head --node-ip-address 127.0.0.1 --num-gpus 8 --disable-usage-stats -### 8 +### 6 +[multi_node num_nodes=None] mkdir -p /root/models/GLM-5_torch_dist && rsync -a + --info=progress2 /models/GLM-5_torch_dist/ /root/models/GLM-5_torch_dist && mkdir -p /root/models/GLM-5 && rsync -a + --info=progress2 /models/GLM-5/ /root/models/GLM-5 + +### 7 nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l -### 9 +### 8 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "256", "SGLANG_NSA_FORCE_MLA": "1", "INDEXER_ROPE_NEOX_STYLE": "0", "NVSHMEM_DISABLE_NCCL": "1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/prepare_cp.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/prepare_cp.txt deleted file mode 100644 index 52776b98ff..0000000000 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/prepare_cp.txt +++ /dev/null @@ -1,7 +0,0 @@ -### 0 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5_torch_dist && rsync -a - --info=progress2 /models/GLM-5_torch_dist/ /root/models/GLM-5_torch_dist - -### 1 -[multi_node num_nodes=None] mkdir -p /root/models/GLM-5 && rsync -a - --info=progress2 /models/GLM-5/ /root/models/GLM-5 diff --git a/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/train.txt b/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/train.txt index 49991ed85d..fa085ea1ab 100644 --- a/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/train.txt +++ b/tests/snapshots/launch_scripts/py/scripts/run_glm5_744b_a40b.py/train.txt @@ -10,9 +10,14 @@ export PYTHONUNBUFFERED=1 && ray start --disable-usage-stats ### 2 -nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l +[multi_node num_nodes=None] mkdir -p /root/models/GLM-5_torch_dist && rsync -a + --info=progress2 /models/GLM-5_torch_dist/ /root/models/GLM-5_torch_dist && mkdir -p /root/models/GLM-5 && rsync -a + --info=progress2 /models/GLM-5/ /root/models/GLM-5 ### 3 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 4 export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "256", "SGLANG_NSA_FORCE_MLA": "1", "INDEXER_ROPE_NEOX_STYLE": "0", "NVSHMEM_DISABLE_NCCL": "1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' diff --git a/tests/snapshots/launch_scripts/py/scripts/run_inkling.py/prepare_cp.txt b/tests/snapshots/launch_scripts/py/scripts/run_inkling.py/prepare_cp.txt deleted file mode 100644 index 8c7f29097d..0000000000 --- a/tests/snapshots/launch_scripts/py/scripts/run_inkling.py/prepare_cp.txt +++ /dev/null @@ -1,3 +0,0 @@ -### 0 -[multi_node num_nodes=None] mkdir -p /root/models/Inkling-4layer_torch_dist && rsync -a - --info=progress2 /root/models/Inkling-4layer_torch_dist/ /root/models/Inkling-4layer_torch_dist diff --git a/tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/full_train.txt b/tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/full_train.txt new file mode 100644 index 0000000000..78641f07ef --- /dev/null +++ b/tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/full_train.txt @@ -0,0 +1,141 @@ +### 0 +mkdir -p /root/models /root/datasets + +### 1 +hf download Qwen/Qwen3-4B + --local-dir /root/models/Qwen3-4B + +### 2 +hf download + --repo-type dataset zhuzilin/dapo-math-17k + --local-dir /root/datasets/dapo-math-17k + +### 3 +hf download + --repo-type dataset zhuzilin/aime-2024 + --local-dir /root/datasets/aime-2024 + +### 4 +PYTHONPATH=:/root/Megatron-LM:/frozen/pythonpath torchrun + --nproc-per-node 8 /tools/convert_hf_to_torch_dist.py + --swiglu + --num-layers 36 + --hidden-size 2560 + --ffn-hidden-size 9728 + --num-attention-heads 32 + --group-query-attention + --num-query-groups 8 + --use-rotary-position-embeddings + --disable-bias-linear + --normalization RMSNorm + --norm-epsilon 1e-6 + --rotary-base 1000000 + --vocab-size 151936 + --kv-channels 128 + --qk-layernorm + --hf-checkpoint /root/models/Qwen3-4B + --save /root/models/Qwen3-4B_torch_dist + +### 5 +pkill -9 sglang; sleep 3; ray stop + --force; pkill -9 ray; pkill -9 miles; sleep 3; pkill -9 ray; pkill -9 miles; pkill -9 redis; true; + +### 6 +export PYTHONUNBUFFERED=1 && ray start + --head + --node-ip-address 127.0.0.1 + --num-gpus 8 + --disable-usage-stats + +### 7 +nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l + +### 8 +export no_proxy=127.0.0.1 && export PYTHONUNBUFFERED=1 && ray job submit + --address="http://127.0.0.1:8265" + --runtime-env-json='{"env_vars": {"PYTHONUNBUFFERED": "1", "CUDA_DEVICE_MAX_CONNECTIONS": "1", "NCCL_NVLS_ENABLE": "0", "no_proxy": "127.0.0.1,127.0.0.1", "MASTER_ADDR": "127.0.0.1", "PYTHONPATH": ":/root/Megatron-LM:/frozen/pythonpath"}}' + -- python3 /train.py + --swiglu + --num-layers 36 + --hidden-size 2560 + --ffn-hidden-size 9728 + --num-attention-heads 32 + --group-query-attention + --num-query-groups 8 + --use-rotary-position-embeddings + --disable-bias-linear + --normalization RMSNorm + --norm-epsilon 1e-6 + --rotary-base 1000000 + --vocab-size 151936 + --kv-channels 128 + --qk-layernorm + --hf-checkpoint /root/models/Qwen3-4B + --save /root/shared_data/260101-000000-000/checkpoints + --save-interval 20 + --load /root/models/Qwen3-4B_torch_dist + --ref-load /root/models/Qwen3-4B_torch_dist + --save-retain-interval 20 + --prompt-data /root/datasets/dapo-math-17k/dapo-math-17k.jsonl + --input-key prompt + --label-key label + --apply-chat-template + --rollout-shuffle + --rm-type math + --num-rollout 3000 + --rollout-batch-size 32 + --n-samples-per-prompt 8 + --rollout-max-response-len 8192 + --rollout-temperature 1 + --global-batch-size 256 + --balance-data + --optimizer adam + --lr 1e-6 + --lr-decay-style constant + --weight-decay 0.1 + --adam-beta1 0.9 + --adam-beta2 0.98 + --advantage-estimator grpo + --entropy-coef 0.00 + --eps-clip 0.2 + --eps-clip-high 0.28 + --use-kl-loss + --kl-loss-coef 0.00 + --kl-loss-type low_var_kl + --use-wandb + --wandb-project miles-run_qwen3_4b + --wandb-group 260101-000000-000 + --wandb-key 'frozen-wandb-api-key' + --disable-wandb-random-suffix + --use-dynamic-batch-size + --max-tokens-per-gpu 9216 + --eval-interval 20 + --eval-prompt-data aime /root/datasets/aime-2024/aime-2024.jsonl + --n-samples-per-eval-prompt 16 + --eval-max-response-len 16384 + --eval-top-p 1 + --rollout-num-gpus-per-engine 1 + --sglang-chunked-prefill-size 4096 + --sglang-mem-fraction-static 0.7 + --tensor-model-parallel-size 2 + --sequence-parallel + --pipeline-model-parallel-size 1 + --context-parallel-size 4 + --cp-comm-type a2a + --expert-model-parallel-size 1 + --expert-tensor-parallel-size 1 + --recompute-granularity full + --recompute-method uniform + --recompute-num-layers 1 + --attention-dropout 0.0 + --hidden-dropout 0.0 + --accumulate-allreduce-grads-in-fp32 + --attention-softmax-in-fp32 + --attention-backend flash + --train-memory-margin-bytes 3221225472 + --actor-num-nodes 1 + --actor-num-gpus-per-node 8 + --num-gpus-per-node 8 + --colocate + --use-fault-tolerance + --dump-details /root/shared_data/260101-000000-000/dump_details diff --git a/tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/execute.txt b/tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/train.txt similarity index 100% rename from tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/execute.txt rename to tests/snapshots/launch_scripts/py/scripts/run_qwen3_4b.py/train.txt