Skip to content

[Refactor] [Ascend] dsa: Align index K cache and scale layout with GPU - #47640

Open
dqh1900013087-tech wants to merge 17 commits into
sgl-project:mainfrom
dqh1900013087-tech:qli_version1_no_cpu_copy
Open

dqh1900013087-tech wants to merge 17 commits into
sgl-project:mainfrom
dqh1900013087-tech:qli_version1_no_cpu_copy

Conversation

@dqh1900013087-tech

@dqh1900013087-tech dqh1900013087-tech commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Align the NPU DSA index K cache and K scale storage layout with the GPU implementation while preserving inference performance and model accuracy.

Previously, the NPU implementation stored quantized index keys and their scales in separate buffers. This PR adopts the same packed page layout as GPU, storing FP8 keys and FP32 scales in a single allocation and writing them together.

Modifications

  • Align the NPU index cache with the GPU packed page layout: each page stores the FP8 index keys followed by their FP32 scales.
  • Reuse the existing Triton cache writer to write already-quantized keys and scales together.
  • Update the NPU DSA indexer to consume the packed cache through the registered packed quantized lightning indexer operator.
  • Update HiCache host allocation and memory accounting to use the index buffer's actual dtype and element size.
  • Add Ascend correctness tests covering cache layout, page boundaries, overwrites, padding slots, NPU graph replay, and packed reader/indexer integration.

Accuracy Tests

  • All applicable pre-commit checks passed.
  • Added correctness tests:
    • test/manual/ascend/test_dsa_index_k_cache.py
    • test/manual/ascend/test_dsa_packed_integration.py
  • The tests include comparisons between the packed reader and the existing NPU quantized lightning indexer.
  • Ascend hardware test results and end-to-end model accuracy comparisons: pending.

Speed Tests and Profiling

The cache writer combines key and scale writes into one Triton kernel launch. The packed indexer is invoked through the registered Torch operator with device-side sequence metadata.

The goal is to align the cache layout without introducing a performance regression.

Baseline versus PR latency, throughput, and profiling results: pending.

Checklist

  • Format code with pre-commit.
  • Add focused Ascend correctness tests.
  • Update documentation as needed.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci.

CI States

Latest PR Test (Base): ⚠️ Run #41796358986
Latest PR Test (Extra): ⚠️ Run #41796358746
Latest PR Test (AMD ROCm 10): ⚠️ Run #41796358870

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Reuse the GPU Triton writer through an NPU wrapper, read packed index pages with QLI, and adapt host cache accounting. Include writer, graph replay, and packed reader integration tests.

Keep the original CPU backup and restore implementations. Packed index cache CPU backup and restore are outside this experiment.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang jit-kernel npu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant