openpangu: use fused indexer top_k with -fidx (CPU-only op)#2111
Merged
ikawrakow merged 1 commit intoJul 12, 2026
Conversation
Route DSA indexer selection through ggml_indexer_topk when -fidx is set: one op computes the weighted-relu head sum plus causal mask and returns top-k rows without materializing the [n_kv, n_ihead, T] score tensor. Off by default; the unfused chunked/full paths are unchanged. Composes with gathered DSA, deferred attention-chunk masks, and the set_rows mask path.
ikawrakow
approved these changes
Jul 12, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
openPangu's indexer scoring is exactly the shape #2098's
GGML_OP_INDEXER_TOPKcomputes, so this PR routes openPangu's DSA selection through the op when-fidxis set. One branch inbuild_openpangu.cpp, off by default, unfused paths unchanged. It composes with the gathered DSA decode path, the deferred attention-chunk masks, and theset_rowsmask path, and the loader now echoesdsa_idx_topkfor openPangu the way it does for GLM-DSA. The op is CPU-only, so on a CUDA rig the scheduler places it on CPU and its operands cross the backend boundary each eval.The point on a hybrid rig is the compute-buffer ceiling. Measured on a 12 GB RTX 4070 (the #2065 Q4_K_M,
-fa off -ngl 999 -ot exps=CPU -ub 2048, q8_0 caches):The paths diverge at 64K, and at 128K the unfused path fails to allocate while
-fidxholds the CUDA0 buffer at its 32K floor, because the score tensors never exist on the GPU. The cost is throughput: with the op on CPU and attention on CUDA, the warm A/B sweep (figure, right panels) runs S_PP -8% at N_KV 2048 growing to -44% at 32K, and S_TG -4% to -26%. A-ngl 0probe on this 8-core desktop showed S_TG at parity and S_PP 3 to 12 percent behind, so on this class of CPU the flag is not a throughput feature, and can be left off when the unfused path fits.Correctness is output-level: a 4.7K-token needle at temp 0 (past top_k 2048, so selection prunes) returns identically with
-fidxoff and on across the base graph, MTP speculative decode, the deferred-chunk regime, and all four accepted indexer cache types (12/12 arms; top-k tie-break identity is not asserted).The branch leaves op placement to the scheduler, so if a CUDA implementation of the op lands (the direction #2103 is exploring), openPangu picks it up here with no further arch changes, and the round-trip cost above disappears. And the op carries #2098's platform contract (AVX2 and ARM dotprod builds); other builds should leave
-fidxoff, same as GLM-DSA today.