fix: add missing TurboQuant FA template instances for HIP/ROCm build#35
Merged
TheTom merged 1 commit intoMar 31, 2026
Conversation
The HIP build was missing 9 turbo cross-type flash attention vec instantiations (turbo4 combos, turbo3/turbo2 cross-types) that were present in the CUDA CMakeLists but not mirrored to the HIP CMakeLists. Also guard the D>=576 tile kernel dispatch with #ifndef GGML_USE_HIP since those instance files are already excluded from the HIP build (they exceed HIP's 65536-byte local memory limit). Tested on: ROCm 6.4.4, gfx1151 (AMD Ryzen AI Max+ 395 / Strix Halo)
Owner
|
Metal sanity check passed. Built and tested on M2 Pro (macOS, Metal backend) to verify no regressions. Build: clean, no warnings. PPL test (Qwen2.5-1.5B Q4_K_M, q8_0-K/turbo4-V, 512 context, 4 chunks):
Identical. Changes are HIP-only as expected. CUDA is also unaffected (the Merging. |
Owner
|
Thanks for the fix Terry, and welcome as a contributor! Great to have Strix Halo coverage on the fork. |
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
…026-07-12) 4 parallele Subagents (Vulkan/AMD, CUDA/MoE, arXiv, Multi-GPU/Batching) konsolidiert zu 25 neuen Items in ROADMAP: Tier 1 Quick Wins (3 neu): - AtomicBot-ai#31 K-Quant A-Matrix Transpose CM1 (PR ggml-org#22970, +5-15% PP auf Mars) - AtomicBot-ai#32 Pascal L1 Cache Tuning (-Xptxas -dlcm=ca, Styx) - TheTom#33 Per-Quant MMVQ/MMQ Batch Threshold (AMD MFMA, Mars/Venus) Tier 2 (7 neu): - TheTom#34 UBBoost, TheTom#35 Row-Packing DMMV, TheTom#36 Auto Param Fitting TP - TheTom#37 LFRU Expert Caching, TheTom#38 Conf-KV, TheTom#39 Talon, TheTom#40 MoE Load Balancing Tier 3 (5 neu): - TheTom#41 GRKV, TheTom#42 CapKV, TheTom#43 SliderQuant, TheTom#44 Alloc-MoE, TheTom#45 CUDA Streams QKV Tier 4 (10 neu): - TheTom#46-55: SpecMD, QUICK, FluxMoE, STAR-KV, VQKV, CompilerKV, SliceMoE, MoBiE, DASH-Q, GOOSE 7 PRs als bereits im Fork identifiziert (nicht erneut vorschlagen): ggml-org#21472, ggml-org#23764, ggml-org#22299, ggml-org#21611, ggml-org#22423, ggml-org#18749, Warp Shuffle, Constant Memory
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
Pre-Solo Planung nach solo-session Skill Methodik: - Phase 1: E4B+MTP 8GB Crash untersuchen (~3-4h) - Phase 2: TheTom#35 Row-Packing DMMV implementieren (~4-6h) - Phase 3: Batch-Evaluation 8 Tier 2 Items (~3-4h) - Phase 4: TheTom#45 CUDA Streams QKV Prototyp (~2-3h) Entscheidungen: - Uranus ist online → E4B auf 16GB testbar - Services deaktivieren für Tests, später reaktivieren - Code-Review Pflicht bei nicht-trivialen Änderungen - Kein Abbruch zwischendrin
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
RDNA3 (Mars Radeon 760M) previously used default row counts (rm_stdq=1, rm_kq=2 → NUM_ROWS=2). GCN already used rm_stdq=2, rm_kq=4 (NUM_ROWS=4). RDNA3 has sufficient register file and shared memory for 4 rows/workgroup. Expected: +10-20% DMMV throughput on Q4_K and standard quants.
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
DMMV kernel is not the bottleneck for small MoE models on RDNA3. Row-packing was already partially implemented (NUM_ROWS=2 default). Increasing to NUM_ROWS=4 on RDNA3 gives marginal improvement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The HIP build was missing 9 turbo cross-type flash attention vec
instantiations (turbo4 combos, turbo3/turbo2 cross-types) that were
present in the CUDA CMakeLists but not mirrored to the HIP CMakeLists.
Also guard the D>=576 tile kernel dispatch with #ifndef GGML_USE_HIP
since those instance files are already excluded from the HIP build
(they exceed HIP's 65536-byte local memory limit).
Tested on: ROCm 6.4.4, gfx1151 (AMD Ryzen AI Max+ 395 / Strix Halo)
Overview
Additional information
Requirements