Skip to content

[Perf] Avoid dtype conversion in decode page count - #43818

Open
kyoungbinkim wants to merge 1 commit into
sgl-project:mainfrom
kyoungbinkim:perf/remove-decode-page-count-cast
Open

kyoungbinkim wants to merge 1 commit into
sgl-project:mainfrom
kyoungbinkim:perf/remove-decode-page-count-cast

Conversation

@kyoungbinkim

@kyoungbinkim kyoungbinkim commented Oct 12, 2026 •

Copy link
Copy Markdown

Motivation

The decode allocation path calls get_num_new_pages on CPU tensors for every scheduling step. The existing .int().sum() sequence copies the boolean mask to int32 before reducing it. This removes that unnecessary dtype conversion from the host-side hot path.

Related to #24922.

Modifications

  • Replace the decode-path .int().sum() with torch.count_nonzero.
  • Correct the helper return annotation to int.
  • Add a regression test that verifies the page count and absence of aten._to_copy.
  • Add a focused host-side microbenchmark comparing the legacy and current implementations.

Accuracy Tests

This does not change model outputs. The unit test covers page-boundary counting and passes:

python test/registered/unit/mem_cache/test_num_new_pages.py

Speed Tests and Profiling

Command:

python benchmark/scheduler/bench_num_new_pages.py --iterations 5000 --repeats 5

WSL2, PyTorch 2.10.0+xpu, CPU host path:

Batch Legacy (us) Current (us) Helper speedup
1 4.670 3.094 1.51x
8 4.728 3.195 1.48x
32 4.935 3.315 1.49x
128 4.985 3.545 1.41x

The local Intel XPU runtime currently reports zero devices, so these results only measure the CPU helper. End-to-end GPU performance has not been measured, and this PR does not claim a GPU performance improvement.

Checklist

  • Format checks pass for all changed files.
  • Added a registered CPU/XPU unit test.
  • Documentation is not needed for this internal helper change.
  • Included the available focused benchmark and its GPU-validation limitation.
  • Followed the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #38231355869
Latest PR Test (Extra): ❌ Run #38231355300
Latest PR Test (AMD ROCm 10): ❌ Run #38231355777

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant