Skip to content

Multi-LoRA for DeepSeek-V4.1-Flash on the raw model path - #15

Merged
micahtyong merged 12 commits into
spindle-miles-v0.1.2from
micah/v41-lora
Oct 20, 2026
Merged

micahtyong merged 12 commits into
spindle-miles-v0.1.2from
micah/v41-lora

Conversation

@micahtyong

@micahtyong micahtyong commented Oct 16, 2026 •

Copy link
Copy Markdown

Summary

DeepSeek-V4.1-Flash LoRA, part 2 of 5. Serve adapters (sglang#80) → train them (this PR) → Miles v0.1.2 and BF16 assets (spindle#73) → LoRA on Miles v0.1.2 (spindle#74) → the recipe (spindle#75). The two fork PRs merge first.

Multi-LoRA training for DeepSeek-V4.1 on Miles' own model ("raw" mode). Megatron-Bridge has no V4.1 mapping, so the multi-LoRA path could not build, load or export it.

  • Targets: attention wq_a, wkv, wq_b, wo_b and the shared expert w1/w2/w3. Routed-expert targets fail with a clear error. The router, Engram and wo_a stay frozen.
  • Only V4.1 runs raw: raw multi-LoRA is limited to deepseek_v41, and every dispatch point checks it. bshd is accepted at micro-batch 1 without dynamic batching.
  • Build, load, export:
    • Adapters attach before DDP wrapping and are hidden from mbridge's base-weight load.
    • Export writes a standard PEFT adapter: TP shards gathered, trimmed to the slot's rank, and the fused shared-expert fc1 split into w1/w3.
  • TP gradient fix: Megatron-Bridge's multi-LoRA forward misses a backward all-reduce under tensor parallelism, leaving lora_A gradients 72–84% off at TP 2. A patched forward fixes it on the V4.1 path only. The upstream fix is Megatron-Bridge#6513; once it lands, the copy can go.
  • Eager CUDA module loading: a GEMM kernel loaded lazily mid-forward could deadlock against cross-node expert all-to-alls, the hazard in the CUDA lazy-loading docs. Raw-mode multi-LoRA trainers now start with CUDA_MODULE_LOADING=EAGER, which costs about 1.9 GiB per GPU.
  • Layernorm-fused targets are refused: under sequence parallelism, Transformer Engine drops their input gradient. No V4.1 target is one.

E2E

Tiny random-weight V4.1 (2 layers) on 2×H100, TP 2 with sequence parallelism, through the real MultiLoRATrainRayActor:

Check Result
Gradients vs single-device reference ≤3% relative error in bf16 with the fix (72–84% without)
Base weights through the mbridge load 72 of 72 parameters bit-exact
Training loss 7.08 → 6.18 over 3 steps; only the active slot gets gradients
save_slot / load_slot bit-identical weights and loss

Full-size DeepSeek-V4.1-Flash on 2×8 B300 (TP 4, EP 16), through Spindle and the stock Tinker SDK:

  • Training: loss 77.21 → 76.60 → 58.00 over three steps, at 2.2 s per forward_backward.
  • Checkpoints: save and restore reproduce the pre-save loss exactly.
  • Serving: the exported adapter serves from SGLang, including two adapters at once.
  • Hang: with eager loading, 3 of 3 fresh trainers completed their first forward pass, against about 3 of 7 hanging before.

Upstream

This enables the DeepSeek-V4.1-Flash path now. Upstream, radixark/miles#4205 (with its serving counterpart sgl-project/sglang#45231) adds V4.1 LoRA through Miles' native single-adapter path and will likely supersede the model-side pieces here once it ships in a Miles release. The multi-adapter wrapping and export are the part it doesn't cover. We'll track both and decide from there what still needs upstreaming from this PR. The tensor-parallel gradient fix is already upstream as Megatron-Bridge#6513.

🤖 Generated with Claude Code

micahtyong and others added 9 commits October 15, 2026 14:19
V4.1 checkpoints name modules layers.N.attn.* and layers.N.ffn.*, with
the output head at head.weight. wo_a is a grouped einsum weight rather
than a linear module, so it is not a target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Maps V4.1 HF targets (attention wq_a/wkv/wq_b/wo_b and shared-expert
w1/w2/w3) to the Miles model's Megatron modules, wraps them with
Megatron-Bridge multi-LoRA layers, freezes the router's expert bias, and
exports one slot as HF PEFT tensors: TP shards gathered, the fused
shared-expert linear_fc1 split into w1/w3, trimmed to the slot's rank.

V4.1 attention gathers its sequence-parallel input before its
projections, so its adapters skip sequence-parallel collectives and the
replicated wq_a/wkv adapters are not TP-summed. Routed experts are
rejected for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Multi-LoRA keeps --megatron-to-hf-mode raw for models with a Miles model
definition (deepseek_v41) and resolves their targets from the HF layout
without Megatron-Bridge; every other model is still switched to bridge.

bshd with --micro-batch-size 1 and no dynamic batch size is accepted:
one sample per micro-batch keeps each adapter's tokens contiguous.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In raw mode the multi-LoRA actor builds the model from the Miles model
provider with adapters applied before DDP wrapping. Base weights load
with every multi-LoRA wrapper swapped for its base linear, so mbridge
and torch_dist see the base model's exact modules and names. Adapter
export goes through the model's raw exporter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fused multi-LoRA forward multiplies raw weight shards, so under
tensor parallelism the gradient reaching a TP-sharded lora_B input (and
the adapter input of a column-parallel base without sequence
parallelism) holds only the local rank's share. lora_A gradients of
every column- and row-parallel adapter were off by roughly the TP size,
and the partial input gradient leaked into upstream layers.

use_tp_exact_multi_lora_forward reproduces the fused forward with the
same copy-to-TP-region boundaries Megatron's parallel linears use. The
DeepSeek-V4.1 multi-LoRA path enables it; on a TP=2 run its A/B
gradients match a single-device reference to bf16 precision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under sequence parallelism Megatron-Bridge asks a layernorm-fused column
linear to return its gathered layernorm output, and Transformer Engine
then drops the gradient flowing back through that output, so the
adapter's input gradient never reaches upstream layers. None of the
V4.1 targets is layernorm-fused; fail at build time if one ever is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With CUDA lazy module loading, the first use of a kernel can synchronize
the CUDA context. DeepSeek-V4.1 multi-LoRA runs expert parallelism across
nodes with per-rank token counts, so cuBLAS picks first-use grouped-GEMM
kernels on only some ranks, mid-forward, while expert all-to-all kernels
are in flight. A hung run had four ranks blocked in cuKernelSetAttribute
inside cublasLtMatmul, waiting on their own GPUs, and every other rank
waiting on them.

Raw multi-LoRA trainers now start with CUDA_MODULE_LOADING=EAGER unless
--train-env-vars sets it. Eager loading costs about 2 GiB per GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread miles/ray/specs/train.py
)


def raw_multi_lora_needs_eager_module_loading(args) -> bool:

@micahtyong micahtyong Oct 16, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing, we hit the deadlock described in the NVIDIA lazy loading docs here:

Some programs incorrectly assume that concurrent kernel execution is guaranteed. A deadlock can occur if cross-kernel synchronization is required, but kernel execution has been serialized.

DeepSeek-V4.1 multi-LoRA runs expert parallelism across nodes, and each rank gets a different number of tokens per expert. On one rank, the CPU launched a routed-expert GEMM kernel (kernel B) for the first time. With lazy loading, the first launch loads B's code, and that load waits for kernels already running on the GPU, blocking the CPU. One of those running kernels was kernel A, a NCCL collective that can't finish until the other ranks send their data. But the other ranks were waiting on this rank: their next all-to-all needs the expert output kernel B was supposed to produce. B waits on A, A waits on the other ranks, and the other ranks wait on B, so every rank hangs.

So the stall starts within one rank, but it becomes a deadlock because kernel A is a collective that depends on the other ranks. Loading CUDA modules eagerly at startup removes the first-time load from the forward pass

micahtyong added a commit to modal-labs/spindle that referenced this pull request Oct 16, 2026
…anch

V4.1 raw-mode multi-LoRA lands there via modal-projects/miles#15; the build fails if it has not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-level imports

Target resolution has no Megatron imports so argument parsing stays light; export needs only torch and megatron.core so its CPU tests keep running; the Bridge-dependent wrapping, TP-exact forward and unwrap helpers move out of slots.py into the V4.1 module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mid = reduce_from_tensor_model_parallel_region(mid)
else:
mid = gather_from_tensor_model_parallel_region(mid)
mid = copy_to_tensor_model_parallel_region(mid)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the TP gradient fix. With TP-sharded adapters, every rank sees the full rank-r activation mid, but each rank holds only its shard of lora_B and produces only its slice of the output:

forward   mid ──► B_0·mid ─► y_0   (rank 0)
              └─► B_1·mid ─► y_1   (rank 1)
backward  dL/dmid = B_0ᵀ·dy_0 + B_1ᵀ·dy_1   ← each rank computes one term

copy_to_tensor_model_parallel_region is the identity in forward and an all-reduce in backward, so it sums those terms before they reach lora_A. The upstream fused forward has no such op, so lora_A (and everything upstream of the adapter) trains on this rank's share alone: 72–82% gradient error at TP 2 against a single-device reference. Megatron's ColumnParallelLinear uses the same op on its input for the same reason.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're upstreaming a fix to Megatron bridge here. After that, we can remove the diff added in miles_plugins/models/deepseek_v4_1/lora.py in this PR.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note on _tp_exact_multi_lora_forward: remove once the pinned Megatron-Bridge includes NVIDIA-NeMo/Megatron-Bridge#6513. Miles pins the radixark fork (radixark/Megatron-Bridge@8cd3466), so the fix also needs to be cherry-picked there before this copy can go.

micahtyong added a commit to modal-labs/spindle that referenced this pull request Oct 16, 2026
…anch

V4.1 raw-mode multi-LoRA lands there via modal-projects/miles#15; the build fails if it has not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread miles/backends/megatron_utils/model.py Resolved
mid = reduce_from_tensor_model_parallel_region(mid)
else:
mid = gather_from_tensor_model_parallel_region(mid)
mid = copy_to_tensor_model_parallel_region(mid)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're upstreaming a fix to Megatron bridge here. After that, we can remove the diff added in miles_plugins/models/deepseek_v4_1/lora.py in this PR.

@micahtyong
micahtyong marked this pull request as ready for review October 16, 2026 22:43
micahtyong and others added 2 commits October 16, 2026 15:52
is_raw_multi_lora_enabled raises for models outside RAW_MULTI_LORA_MODEL_TYPES, so model building, base-weight loading and adapter export can no longer route another model into the DeepSeek-V4.1 path. Note when the copied TP-exact forward can go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
train.py already forces CUDA_MODULE_LOADING=EAGER for raw multi-LoRA trainers. The routed-expert rejection and the fused fc1 split were each tested twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong

Copy link
Copy Markdown
Author

Merging to enable the DeepSeek-V4.1-Flash path now. Upstream, radixark#4205 adds V4.1 LoRA through Miles' native path and will likely supersede the model-side pieces here; the multi-adapter wrapping and export are what it doesn't cover. We'll track it and decide from there what still needs upstreaming. The TP gradient fix is already upstream as NVIDIA-NeMo/Megatron-Bridge#6513.

@micahtyong
micahtyong merged commit 5562bdb into spindle-miles-v0.1.2 Oct 20, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant