Skip to content

[LoRA] opt8: piecewise CUDA graph for LoRA prefill — correct & opt-in, e2e-negative in current split-op form (default OFF) - #9

Closed
yushengsu-thu wants to merge 13 commits into
qwen3-30b-a3b-2507-bf16from
opt8-piecewise-lora
Closed

yushengsu-thu wants to merge 13 commits into
qwen3-30b-a3b-2507-bf16from
opt8-piecewise-lora

Conversation

@yushengsu-thu

@yushengsu-thu yushengsu-thu commented Jun 13, 2026 •

Copy link
Copy Markdown
Owner

What

13 commits making the piecewise CUDA graph work with LoRA serving (today server_args condition 7 force-disables it whenever --enable-lora). Opt-in via --enforce-piecewise-cuda-graph; zero behavior change without the flag.

Motivation (run 20260612-051818): LoRA prefill is host-bound — graph-off idle 49% (13.1k launches), and in production the host skew shows up as allreduce spin (1672 µs/call vs base 149 µs; 73% of the 4096-tok chunk wall). Ceiling estimate was prefill ~3.6×.

The implementation ladder (each step = a named, diagnosed blocker)

  1. step1: bf16 trtllm MoE custom-op wrappers (raw flashinfer call JIT-loads + logs in-trace → dynamo hard fail; mirrors the existing fp8 wrappers — also benefits the no-lora bf16 cell)
  2. step2: MoE-LoRA dispatch as an opaque split op (RadixAttention escape-hatch pattern; dispatch subtree is untraceable host Python)
  3. step3: two-stream forced OFF under the piecewise forward (Stream objects can't cross FX split partitions)
  4. step4–6: capture-batch fixes (upstream lora_ids hardcoded-None bug; extend_seq_lens_cpu is a tensor on the capture batch — 0-d-tensor max_len becomes a POINTER triton arg)
  5. step7a–e: replay guard ladder (TORCH_LOGS=recompiles-named): per-forward context reads, per-batch batch_info tensor identity (→ persistent piecewise batch_info), host-int kernel-grid guards (→ capacity constants + seg-tensor masking)
  6. step7f + step8x: embedding and ALL dense (qkv/o/column/replicated/lm-head) LoRA applies as split ops — no in-graph batch_info/sgemm_info reads remain

Result (probe runs/.../opt8-probe/, GB300 TP4/EP4)

  • Correctness: PASS. Full 50-size capture (104 s), stable replay, full bench, coherence; acc ON: KL(sglang-lora, trainer) = 0.003761 < noise floor 0.004243.
  • Performance: NEGATIVE — prefill ON/OFF = 62% (24.2k vs 39.1k tok/s at bs16/32/64), decode flat, e2e ~+4%.
  • Cause: correctness currently requires ~6 split ops/layer (4 dense LoRA + MoE + attention) × 48 layers ≈ 300 graph pieces — the host gaps return through the eager seams, plus per-op output copies.

Disposition & the payoff path

  • Default OFF (no flag → byte-identical; FP8/NVFP4 untouched — the split ops only engage under the piecewise forward, which remains disabled by condition 7 unless enforced).
  • The payoff requires making the LoRA kernels traceable in-graph (tensor-borne batch metadata everywhere, capacity grids + seg-tensor masking like the decode graph — upstream-aligned), keeping at most the MoE dispatch as a split op (~2 pieces/layer). This PR is the working foundation + full diagnosis ledger for that work.
  • Full ladder, guard names, and measurement methodology: journal_opti.md (runs repo) §opt8.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #27453226793
Latest PR Test (Extra): ❌ Run #27453226766

…piecewise compatibility)

The bf16 runner called flashinfer's trtllm_bf16(_routed)_moe RAW — under the
piecewise-cuda-graph compile, dynamo traces into flashinfer's call-time JIT
module load + logger.debug and dies (torch._dynamo Unsupported: logging.Logger).
The fp8 paths already route through register_custom_op wrappers for exactly this
reason; this adds the missing bf16 twins and rewires the two call sites.

Eager behavior identical (wrapper is a direct passthrough); benefits the no-lora
bf16 cell equally; FP8/NVFP4 untouched. Probe: --enforce-piecewise-cuda-graph.
…(RadixAttention escape-hatch pattern)

Second dynamo break (after step1's flashinfer wrapper) was inside the LoRA
Triton dispatch: get_moe_configs -> torch.cuda.get_device_name() returns a str
(Unsupported: non-Tensor torch.* return). The dispatch subtree is full of such
host-side Python — wrap the WHOLE thing as one custom op + register_split_op,
exactly like unified_attention_with_output: graph pieces capture around it, the
MoE-LoRA block runs eagerly between pieces with the real per-batch LoRA state
(capture-time lora_ids=[None]*bs is never baked into a graph).

Layer objects fetched via a layer_id registry; padded tokens trimmed via the
forward context (num_token_non_padded_cpu), output written into a preallocated
buffer (mutates_args), both mirroring attention. Carries opt2's packed_topk_ids.
Gating: piecewise context present + experimental_sgl_trtllm + extend-mode only —
decode and eager paths byte-identical; FP8/NVFP4 share the same lora layer but
only enter this branch under piecewise (same opaque-op semantics).
…central gate in is_two_stream_active)

Break #3: split_module died on torch.cuda.Stream values crossing partition
boundaries — the dense-layer LoRA side-stream forwards (qkv/o/column/replicated/
merged-column) trace their stream choreography into the FX graph at the small
piecewise sizes where is_two_stream_active() is True. Gate the shared predicate
on the piecewise forward context: every side-stream forward falls back to its
serial saved-original (same kernels, same math); inside graphed pieces the graph
already removes the launch gaps. Eager and decode-graph paths unchanged.
… just computed

capture_one_batch_size computes lora_ids=[None]*bs for enable_lora but then
hardcodes lora_ids=None into the ForwardBatch — prepare_lora_batch does
len(forward_batch.lora_ids) and dies. Never exercised upstream (condition 7
force-disables piecewise whenever LoRA is on).
…_lens_cpu is a CPU tensor

The piecewise-capture ForwardBatch builds extend_seq_lens_cpu as torch.tensor
(scheduler path: list); max() of it is a 0-d tensor that triton specializes as
a pointer kernel arg -> IncompatibleTypeErrorImpl in cdiv at capture.
…o too (same 0-d-tensor trap as step5, second call site)
…_batch reads install per-forward guards -> runtime recompilation at replay)
…ror decode's cuda_graph_batch_info)

The dense/embedding LoRA layers consume batch_info tensors INSIDE captured
graph pieces; per-batch fresh allocations change tensor identity (dynamo guard
miss -> runtime recompilation -> 'PCG capture stream is not set') and would
bind stale storages into replays. One persistent buffer set, updated in place;
kernels already mask by seg tensors with capture-max grids (same contract the
decode cuda graph relies on).
…h — that selects the decode-sized moe_cg_buffers (token_lora_mapping overflows at 4096-token extend)
…namo guards the exact int (TORCH_LOGS=recompiles: embed_tokens ... batch_info.max_len == 2816/3072/...), per-batch values recompile every new length and assert at replay
…(guard #2: batch_info.bs == 1) — zeroed seg/weight tails make excess segments no-ops; captured grids are frozen at capture so per-batch bs could never work anyway
… in-graph batch_info consumer (its host-int kernel grids guard exact values; decode/extend batch_info object swaps overflow the recompile cache -> replay assert)
…ard #3: shared backend's sgemm_batch_info type_id) — base GEMMs stay in-graph, LoRA deltas run eagerly between pieces
@yushengsu-thu

Copy link
Copy Markdown
Owner Author

Closing in favor of the in-graph approach (upcoming PR #10).

Diagnosis update (per-rank prefill trace, TP0–3): the allreduce cost is host-dispatch skew, not load imbalance — compute is balanced across ranks (4% spread) but allreduce ranges 447–3525µs/layer, with TP3 the structural laggard in 10/12 chunks. The no-lora cell (piecewise-by-default) shows the same TP3-laggard pattern but only 2× spread vs LoRA's 7.9× — confirming the eager LoRA launch storm (13.6 MoE-dispatch launches/layer) is what amplifies a tiny structural imbalance into the 3000µs spin.

This split-op form keeps that storm eager (correct but e2e-negative, prefill 62%). PR #10 will make the MoE-LoRA dispatch itself graph-traceable so the storm collapses into graph replay and the skew tightens toward the no-lora floor (~149µs/call). The step1–8x commits here remain the working piecewise+LoRA foundation that #10 builds on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant