Repository navigation
[LoRA] opt8: piecewise CUDA graph for LoRA prefill — correct & opt-in, e2e-negative in current split-op form (default OFF) - #9
Conversation
…piecewise compatibility) The bf16 runner called flashinfer's trtllm_bf16(_routed)_moe RAW — under the piecewise-cuda-graph compile, dynamo traces into flashinfer's call-time JIT module load + logger.debug and dies (torch._dynamo Unsupported: logging.Logger). The fp8 paths already route through register_custom_op wrappers for exactly this reason; this adds the missing bf16 twins and rewires the two call sites. Eager behavior identical (wrapper is a direct passthrough); benefits the no-lora bf16 cell equally; FP8/NVFP4 untouched. Probe: --enforce-piecewise-cuda-graph.
…(RadixAttention escape-hatch pattern) Second dynamo break (after step1's flashinfer wrapper) was inside the LoRA Triton dispatch: get_moe_configs -> torch.cuda.get_device_name() returns a str (Unsupported: non-Tensor torch.* return). The dispatch subtree is full of such host-side Python — wrap the WHOLE thing as one custom op + register_split_op, exactly like unified_attention_with_output: graph pieces capture around it, the MoE-LoRA block runs eagerly between pieces with the real per-batch LoRA state (capture-time lora_ids=[None]*bs is never baked into a graph). Layer objects fetched via a layer_id registry; padded tokens trimmed via the forward context (num_token_non_padded_cpu), output written into a preallocated buffer (mutates_args), both mirroring attention. Carries opt2's packed_topk_ids. Gating: piecewise context present + experimental_sgl_trtllm + extend-mode only — decode and eager paths byte-identical; FP8/NVFP4 share the same lora layer but only enter this branch under piecewise (same opaque-op semantics).
…central gate in is_two_stream_active) Break #3: split_module died on torch.cuda.Stream values crossing partition boundaries — the dense-layer LoRA side-stream forwards (qkv/o/column/replicated/ merged-column) trace their stream choreography into the FX graph at the small piecewise sizes where is_two_stream_active() is True. Gate the shared predicate on the piecewise forward context: every side-stream forward falls back to its serial saved-original (same kernels, same math); inside graphed pieces the graph already removes the launch gaps. Eager and decode-graph paths unchanged.
… just computed capture_one_batch_size computes lora_ids=[None]*bs for enable_lora but then hardcodes lora_ids=None into the ForwardBatch — prepare_lora_batch does len(forward_batch.lora_ids) and dies. Never exercised upstream (condition 7 force-disables piecewise whenever LoRA is on).
…_lens_cpu is a CPU tensor The piecewise-capture ForwardBatch builds extend_seq_lens_cpu as torch.tensor (scheduler path: list); max() of it is a 0-d tensor that triton specializes as a pointer kernel arg -> IncompatibleTypeErrorImpl in cdiv at capture.
…o too (same 0-d-tensor trap as step5, second call site)
…_batch reads install per-forward guards -> runtime recompilation at replay)
…ror decode's cuda_graph_batch_info) The dense/embedding LoRA layers consume batch_info tensors INSIDE captured graph pieces; per-batch fresh allocations change tensor identity (dynamo guard miss -> runtime recompilation -> 'PCG capture stream is not set') and would bind stale storages into replays. One persistent buffer set, updated in place; kernels already mask by seg tensors with capture-max grids (same contract the decode cuda graph relies on).
…h — that selects the decode-sized moe_cg_buffers (token_lora_mapping overflows at 4096-token extend)
…namo guards the exact int (TORCH_LOGS=recompiles: embed_tokens ... batch_info.max_len == 2816/3072/...), per-batch values recompile every new length and assert at replay
…(guard #2: batch_info.bs == 1) — zeroed seg/weight tails make excess segments no-ops; captured grids are frozen at capture so per-batch bs could never work anyway
… in-graph batch_info consumer (its host-int kernel grids guard exact values; decode/extend batch_info object swaps overflow the recompile cache -> replay assert)
…ard #3: shared backend's sgemm_batch_info type_id) — base GEMMs stay in-graph, LoRA deltas run eagerly between pieces
|
Closing in favor of the in-graph approach (upcoming PR #10). Diagnosis update (per-rank prefill trace, TP0–3): the allreduce cost is host-dispatch skew, not load imbalance — compute is balanced across ranks (4% spread) but allreduce ranges 447–3525µs/layer, with TP3 the structural laggard in 10/12 chunks. The no-lora cell (piecewise-by-default) shows the same TP3-laggard pattern but only 2× spread vs LoRA's 7.9× — confirming the eager LoRA launch storm (13.6 MoE-dispatch launches/layer) is what amplifies a tiny structural imbalance into the 3000µs spin. This split-op form keeps that storm eager (correct but e2e-negative, prefill 62%). PR #10 will make the MoE-LoRA dispatch itself graph-traceable so the storm collapses into graph replay and the skew tightens toward the no-lora floor (~149µs/call). The step1–8x commits here remain the working piecewise+LoRA foundation that #10 builds on. |
What
13 commits making the piecewise CUDA graph work with LoRA serving (today
server_argscondition 7 force-disables it whenever--enable-lora). Opt-in via--enforce-piecewise-cuda-graph; zero behavior change without the flag.Motivation (run
20260612-051818): LoRA prefill is host-bound — graph-off idle 49% (13.1k launches), and in production the host skew shows up as allreduce spin (1672 µs/call vs base 149 µs; 73% of the 4096-tok chunk wall). Ceiling estimate was prefill ~3.6×.The implementation ladder (each step = a named, diagnosed blocker)
lora_idshardcoded-None bug;extend_seq_lens_cpuis a tensor on the capture batch — 0-d-tensormax_lenbecomes a POINTER triton arg)TORCH_LOGS=recompiles-named): per-forward context reads, per-batch batch_info tensor identity (→ persistent piecewise batch_info), host-int kernel-grid guards (→ capacity constants + seg-tensor masking)Result (probe
runs/.../opt8-probe/, GB300 TP4/EP4)Disposition & the payoff path
journal_opti.md(runs repo) §opt8.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #27453226793
Latest PR Test (Extra): ❌ Run #27453226766