Skip to content

Give every launch script the same three subcommands - #2431

Merged
fzyzcjy merged 1 commit into
mainfrom
tom/refactor-miles/op16-45
Sep 26, 2026
Merged

fzyzcjy merged 1 commit into
mainfrom
tom/refactor-miles/op16-45

Conversation

@fzyzcjy

@fzyzcjy fzyzcjy commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Part of #1837

@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 47bc382 to e6ba544 Compare August 12, 2026 14:22
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 1541eac to f457523 Compare August 13, 2026 02:52
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from e6ba544 to 61453a4 Compare August 13, 2026 02:52
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from f457523 to fcf463b Compare August 13, 2026 03:11
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch 3 times, most recently from 80d39e6 to ad9d29b Compare August 13, 2026 06:01
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 0e8b5c8 to b7eb2ff Compare August 13, 2026 06:40
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from ad9d29b to 51f369d Compare August 13, 2026 06:40
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from b7eb2ff to cc9fd72 Compare August 14, 2026 04:45
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch 2 times, most recently from 8566512 to cb31bb4 Compare August 14, 2026 04:49
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 013f985 to c27ff29 Compare August 14, 2026 05:52
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from cb31bb4 to 5c998a4 Compare August 14, 2026 05:53
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 5c998a4 to f1603b0 Compare August 28, 2026 11:41
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 598bdda to f02d568 Compare August 28, 2026 12:37
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from f1603b0 to 517a00e Compare August 28, 2026 12:37
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from f02d568 to 9fbd870 Compare August 28, 2026 14:32
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 517a00e to 4746701 Compare August 28, 2026 14:32
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 9fbd870 to f096336 Compare August 28, 2026 15:03
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 4746701 to e31022a Compare August 28, 2026 15:03
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from f096336 to 1ddfc61 Compare August 29, 2026 12:39
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from e31022a to 18360e3 Compare August 29, 2026 12:39

@fzyzcjy fzyzcjy left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The patch leaves Kubernetes inference workers without staged node-local models and breaks Inkling's documented CLI. It also leaves the advertised command migration and its documentation incomplete, and these defects persist at deliver-2.

Comment thread scripts/run_deepseek.py
),
U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"),
]
return {"trainer": " && ".join(copies)}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Stage checkpoints on inference workers too

When --cluster-backend kubernetes is used with model_local_dir != model_dir (the default here), this map attaches both rsyncs only to the trainer role. In deliver-2, helm_backend/launcher/values/pool_entry.py::_with_prepare_cmd skips inference-engine pods, while compute_engine_launch_cmd passes the local HF path to SGLang as model_path; disaggregated rollout nodes therefore never receive the model, and colocated inference pods can start before the trainer's rsync completes. Add inference-side staging; the same pattern affects the V4/GLM launchers.

Comment thread scripts/run_inkling.py
@U.dataclass_cli
def train(args: ScriptArgs):
"""Run training. Assumes HF checkpoint / torch_dist are already staged."""
"""Run training; each trainer pod first copies torch_dist to node-local NVMe when the two differ."""

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep Inkling's CLI as a command group

After prepare_cp and full_train are removed, this app has exactly one registered command and no callback. Typer collapses such an app into a root command, so the newly documented python scripts/run_inkling.py train ... treats train as an unexpected argument; deliver-2 retains this shape. Restore the common commands or add a callback so train remains a subcommand.

@U.dataclass_cli
def prepare_cp(args: ScriptArgs):
_prepare_cp(args)
def _prepare_cmd(args: ScriptArgs) -> dict[str, str]:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Add the common prepare command to both V4 launchers

At deliver-2, neither this launcher nor scripts/amd/run_deepseek_v4.py registers a prepare subcommand; both expose only granular prepare-* steps plus train and full-train. Callers relying on this op's advertised three-command contract still get No such command 'prepare', so add a wrapper composing the existing preparation steps (.claude/rules/launch-and-model-scripts.md:63-70).

Comment thread scripts/run_deepseek.py

@app.command()
@U.dataclass_cli
def train(args: ScriptArgs) -> None:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Update every caller of the renamed launcher roles

This split makes train submit-only and removes prepare-cp, but deliver-2 still documents the old interfaces in docs/models/deepseek/deepseek.md:27-38, docs/models/glm/glm4-5.md:60-73, docs/models/deepseek/deepseek-v3-2.md:96-111, and docs/models/glm/glm5-2.md:39-69. The first two commands no longer perform the documented full pipeline, while the latter two fail with an unknown subcommand; update all affected pages as required by .claude/rules/launch-and-model-scripts.md:135-140.

@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 19f5f4d to b5e8d35 Compare September 2, 2026 07:53
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from e435ac1 to b03c433 Compare September 2, 2026 08:07
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch 2 times, most recently from 0a73e27 to 57d447e Compare September 4, 2026 03:27
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from b03c433 to 4ef9303 Compare September 4, 2026 03:27
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 57d447e to 745d6e9 Compare September 4, 2026 09:20
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch 2 times, most recently from 7511610 to 6cd3a21 Compare September 4, 2026 10:08
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch 2 times, most recently from 059cf1c to 7ecd119 Compare September 4, 2026 15:59
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 2e6b63f to 4b5e930 Compare September 4, 2026 20:44
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 7ecd119 to 2d8372d Compare September 4, 2026 20:44
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 4b5e930 to 891fc28 Compare September 4, 2026 22:30
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 2d8372d to 24c2cbf Compare September 4, 2026 22:30
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from 891fc28 to d25a600 Compare September 5, 2026 07:40
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch 2 times, most recently from 63d0a8a to 857c220 Compare September 10, 2026 02:08
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from d25a600 to ea01986 Compare September 10, 2026 02:08
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 857c220 to f78e6aa Compare September 10, 2026 03:57
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from ea01986 to d9b1c68 Compare September 10, 2026 03:58
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from f78e6aa to 857c220 Compare September 10, 2026 14:26
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from d9b1c68 to ea01986 Compare September 10, 2026 14:27
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from 857c220 to d4d081d Compare September 12, 2026 04:36
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch from ea01986 to a433af3 Compare September 12, 2026 04:36
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch from d4d081d to 3eb3093 Compare September 15, 2026 09:39
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-7 branch 2 times, most recently from 3144d0b to f0c4324 Compare September 17, 2026 23:19
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op16-45 branch 2 times, most recently from 563f12b to 3eb3093 Compare September 18, 2026 07:29
Each script had grown its own entry points, so running one meant reading it
first. They settle on prepare, train and full_train, with the body split into
private steps those three compose, and the recorded command snapshots follow the
new subcommand names.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant