Repository navigation
Conversation
Woody-Hu
requested review from
Fridge003,
Ying1123,
hnyls2002,
ispobock and
merrymercy
as code owners
October 22, 2026 00:49
…able static ForwardBatch The breakable prefill graph rebuilds its static ForwardBatch field by field and runs the outer tail eagerly on the replayed hidden states. Two fields the tail consumes were not carried over, so the eager readout saw None: - decision_layouts: a joint schema head (Clef) scores each request from its layout and raised "the joint schema head did not score this request" (HTTP 500). - token_indices_to_pool: setwise readout pools the score head at caller positions; dropping it silently falls back to standard pooling, i.e. wrong scores instead of an error. Both are only populated for prefill-only batches, so the decode runner is unaffected. The other prefill-only readout fields (dimensions, return_pooled_hidden_states, multi_item_delimiter_indices) are already forwarded.
The Clef joint-head hook declares both CUDA graph phases DISABLED, and a declared resolution outranks raw CLI input (resolution_result), so --cuda-graph-backend-prefill was silently overridden for these checkpoints and the breakable prefill graph stayed unreachable. Keep the default unchanged -- decode still has nothing to capture, and BCG is not the mode this encoder-style prefill was validated with -- but stop overriding a prefill backend the caller picked explicitly, the same way the DeepSeek-V3 and EmbeddingGemma paths consult _cuda_graph_config_locked. A Clef request is one complete prefill, so the graph captures the transformer stack and the joint schema head keeps running eagerly on the replayed hidden states -- which is only correct with the prefill-only readout fix in the previous commit. The manual repro therefore serves with --cuda-graph-backend-prefill breakable; the default path still disables both phases.
Woody-Hu
force-pushed
the
feat/clef-prefill-cuda-graph
branch
from
November 16, 2026 07:46
3f3f004 to
6a21e57
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A Clef checkpoint serves one whole prompt as a single extend and never decodes, so the joint-head
hook declares both CUDA graph phases disabled. A declared resolution outranks raw CLI input, so
--cuda-graph-backend-prefillis silently overridden for these checkpoints and the prefill graph isunreachable.
/v1/systemonetherefore runs fully eager: on a short prompt the launch train between~1000 tiny kernels costs more wall time than the GPU work (A100, clef-flash-int8: 70.9 ms per
request, GPU busy 30.7%).
The prefill body is graph-eligible: BCG captures the transformer stack and replays the outer tail
eagerly, and the
EmbeddingGemmabranch in the same file already uses BCG prefill in embedding mode,so this is not a new contract. Opting the phase in also exposed a bug in the breakable runner, fixed
by the first commit of this PR.
Modifications
prefill_cuda_graph_runner.py(load_batch): the staticForwardBatchis rebuilt field by fieldand the tail then runs eagerly on the replayed hidden states, so the prefill-only readout metadata
it consumes must survive the rebuild. Forward
decision_layouts(a joint schema head raisesthe joint schema head did not score this request-> HTTP 500 without it) andtoken_indices_to_pool(setwise readout; forwarding it keeps the rebuild faithful rather thansilently falling back to standard pooling if the runner's guard is ever relaxed -- today that
guard keeps setwise batches off the graph).
model_hook.py: keep disabling the decode phase (nothing to capture), but stop overriding aprefill backend the caller selected explicitly -- the same
_cuda_graph_config_lockedcheck theDeepSeek-V3 and
EmbeddingGemmapaths use. BCG is CUDA-only, so non-CUDA platforms stay eager.Without an explicit selection the behaviour is unchanged: both phases stay disabled. (Note:
--cuda-graph-backend-prefill fullremains unusable for this stack -- the hybrid GDN backendcrashes while capturing -- so
breakableis the working mode.)test/manual/systemone/test_decision_layouts_prefill_graph.py: reproduction (see below). A CIunit test is not practical here -- the bug needs CUDA graph capture plus a joint-head checkpoint,
which the CPU CI cannot construct.
Reproduction (needs a Clef checkpoint and a GPU):
Accuracy Tests
warmup (
warmup POST /v1/systemone-> 500,the joint schema head did not score this request);this PR serves
cuda_graph={prefill=51.61, decode=0.00}and 3/3/v1/systemonecalls return 200with bit-identical decisions across warm runs. Without the flag the log still reports
cuda_graph={prefill=0.00, decode=0.00}and the eager path answers 200 as before.rows, same edition): no row flips; scores stay within the noise band (+0.11 pp on clef-flash-int8,
-0.11 pp on the 27B). Those runs used our full local stacked configuration, not this patch alone.
Speed Tests and Profiling
Measured with a small stdlib probe (not
bench_serving), A100-80GB, clef-flash-int8, 306-tokenprompt, C=1: 70.5 ms eager -> 46.9 ms with the prefill graph. Profiling of the eager path shows
the gap comes from the host launch train between ~1000 short kernels, not from GPU work.
Note on the default: we keep the phase off unless it is requested explicitly. On the 27B (BF16)
checkpoint the same phase is neutral on short/1k prompts and regresses ~10k-token single requests
by 3-6% (bucket padding dominates the saved launch overhead), so deployments opt in per model size.
Checklist
ruff format,ruff --select F401,F821,F823,UP037,isort --check-only(pinned 7.0.0) andcodespellall pass on the three touched files.is the reproduction.
--cuda-graph-backend-prefillis already documented.CI States
Latest PR Test (Base): ❌ Run #43826180370
Latest PR Test (Extra): ❌ Run #43826180388
Latest PR Test (AMD ROCm 10): ❌ Run #43826180144
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: