Skip to content

[Feat] clef: Honor an explicit prefill CUDA graph backend - #45692

Open
Woody-Hu wants to merge 2 commits into
sgl-project:mainfrom
Woody-Hu:feat/clef-prefill-cuda-graph
Open

Woody-Hu wants to merge 2 commits into
sgl-project:mainfrom
Woody-Hu:feat/clef-prefill-cuda-graph

Conversation

@Woody-Hu

@Woody-Hu Woody-Hu commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

A Clef checkpoint serves one whole prompt as a single extend and never decodes, so the joint-head
hook declares both CUDA graph phases disabled. A declared resolution outranks raw CLI input, so
--cuda-graph-backend-prefill is silently overridden for these checkpoints and the prefill graph is
unreachable. /v1/systemone therefore runs fully eager: on a short prompt the launch train between
~1000 tiny kernels costs more wall time than the GPU work (A100, clef-flash-int8: 70.9 ms per
request, GPU busy 30.7%).

The prefill body is graph-eligible: BCG captures the transformer stack and replays the outer tail
eagerly, and the EmbeddingGemma branch in the same file already uses BCG prefill in embedding mode,
so this is not a new contract. Opting the phase in also exposed a bug in the breakable runner, fixed
by the first commit of this PR.

Modifications

  1. prefill_cuda_graph_runner.py (load_batch): the static ForwardBatch is rebuilt field by field
    and the tail then runs eagerly on the replayed hidden states, so the prefill-only readout metadata
    it consumes must survive the rebuild. Forward decision_layouts (a joint schema head raises
    the joint schema head did not score this request -> HTTP 500 without it) and
    token_indices_to_pool (setwise readout; forwarding it keeps the rebuild faithful rather than
    silently falling back to standard pooling if the runner's guard is ever relaxed -- today that
    guard keeps setwise batches off the graph).
  2. model_hook.py: keep disabling the decode phase (nothing to capture), but stop overriding a
    prefill backend the caller selected explicitly -- the same _cuda_graph_config_locked check the
    DeepSeek-V3 and EmbeddingGemma paths use. BCG is CUDA-only, so non-CUDA platforms stay eager.
    Without an explicit selection the behaviour is unchanged: both phases stay disabled. (Note:
    --cuda-graph-backend-prefill full remains unusable for this stack -- the hybrid GDN backend
    crashes while capturing -- so breakable is the working mode.)
  3. test/manual/systemone/test_decision_layouts_prefill_graph.py: reproduction (see below). A CI
    unit test is not practical here -- the bug needs CUDA graph capture plus a joint-head checkpoint,
    which the CPU CI cannot construct.

Reproduction (needs a Clef checkpoint and a GPU):

python -m sglang.launch_server --model-path <clef-flash> \
    --mem-fraction-static 0.85 --cuda-graph-backend-prefill breakable
python3 test/manual/systemone/test_decision_layouts_prefill_graph.py \
    --base-url http://127.0.0.1:31510 --model <clef-flash>
# add --expect-status 500 on the parent commit

Accuracy Tests

  • With the prefill backend selected explicitly, the parent commit fails the server's own startup
    warmup (warmup POST /v1/systemone -> 500, the joint schema head did not score this request);
    this PR serves cuda_graph={prefill=51.61, decode=0.00} and 3/3 /v1/systemone calls return 200
    with bit-identical decisions across warm runs. Without the flag the log still reports
    cuda_graph={prefill=0.00, decode=0.00} and the eager path answers 200 as before.
  • The graph-on configuration was A/B'd against the eager path on the Decision Index 0.3 suite (2638
    rows, same edition): no row flips; scores stay within the noise band (+0.11 pp on clef-flash-int8,
    -0.11 pp on the 27B). Those runs used our full local stacked configuration, not this patch alone.

Speed Tests and Profiling

Measured with a small stdlib probe (not bench_serving), A100-80GB, clef-flash-int8, 306-token
prompt, C=1: 70.5 ms eager -> 46.9 ms with the prefill graph. Profiling of the eager path shows
the gap comes from the host launch train between ~1000 short kernels, not from GPU work.

Note on the default: we keep the phase off unless it is requested explicitly. On the 27B (BF16)
checkpoint the same phase is neutral on short/1k prompts and regresses ~10k-token single requests
by 3-6%
(bucket padding dominates the saved launch overhead), so deployments opt in per model size.

Checklist

  • Format the code with pre-commit: ruff format, ruff --select F401,F821,F823,UP037,
    isort --check-only (pinned 7.0.0) and codespell all pass on the three touched files.
  • Unit tests: a CPU CI test is not constructible (see Modifications Add install with pip #3); the manual script above
    is the reproduction.
  • Documentation: no new flag or env var, --cuda-graph-backend-prefill is already documented.
  • Accuracy and speed results are in the sections above.
  • Code style guidance followed.

CI States

Latest PR Test (Base): ❌ Run #43826180370
Latest PR Test (Extra): ❌ Run #43826180388
Latest PR Test (AMD ROCm 10): ❌ Run #43826180144

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…able static ForwardBatch

The breakable prefill graph rebuilds its static ForwardBatch field by field
and runs the outer tail eagerly on the replayed hidden states. Two fields the
tail consumes were not carried over, so the eager readout saw None:

- decision_layouts: a joint schema head (Clef) scores each request from its
  layout and raised "the joint schema head did not score this request"
  (HTTP 500).
- token_indices_to_pool: setwise readout pools the score head at caller
  positions; dropping it silently falls back to standard pooling, i.e. wrong
  scores instead of an error.

Both are only populated for prefill-only batches, so the decode runner is
unaffected. The other prefill-only readout fields (dimensions,
return_pooled_hidden_states, multi_item_delimiter_indices) are already
forwarded.
The Clef joint-head hook declares both CUDA graph phases DISABLED, and a
declared resolution outranks raw CLI input (resolution_result), so
--cuda-graph-backend-prefill was silently overridden for these
checkpoints and the breakable prefill graph stayed unreachable.

Keep the default unchanged -- decode still has nothing to capture, and
BCG is not the mode this encoder-style prefill was validated with -- but
stop overriding a prefill backend the caller picked explicitly, the same
way the DeepSeek-V3 and EmbeddingGemma paths consult
_cuda_graph_config_locked. A Clef request is one complete prefill, so the
graph captures the transformer stack and the joint schema head keeps
running eagerly on the replayed hidden states -- which is only correct
with the prefill-only readout fix in the previous commit.

The manual repro therefore serves with --cuda-graph-backend-prefill
breakable; the default path still disables both phases.
@Woody-Hu
Woody-Hu force-pushed the feat/clef-prefill-cuda-graph branch from 3f3f004 to 6a21e57 Compare November 16, 2026 07:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant