convert: support LoRA conversion for MLA kv_b_proj by marksverdhei · Pull Request #1 · heiervang-technologies/ht-llama.cpp

marksverdhei · 2026-03-03T07:27:46Z

Summary

Adds MLA kv_b_proj LoRA conversion support to convert_lora_to_gguf.py
The base model splits kv_b_proj into k_b_proj (transposed) and v_b_proj, but LoraTorchTensor can't handle the required split+transpose on 3D tensors
Decomposes the LoRA A/B matrices and applies the split/transpose directly, yielding the correct per-head lora_a/lora_b tensors for both k_b_proj and v_b_proj

Test plan

Verify LoRA conversion works for MLA-architecture models (e.g., DeepSeek-V2/V3) with kv_b_proj adapters
Confirm non-MLA LoRA conversions are unaffected (no regression)
Validate converted GGUF LoRA loads and produces correct inference output

🤖 Generated with Claude Code

MLA architectures (GLM-4.7-Flash, DeepSeek-V2) store k_b and v_b projections in a combined kv_b_proj weight. The base model conversion splits and transposes this, but LoraTorchTensor cannot handle these operations. Intercept kv_b_proj in the LoRA modify_tensors to decompose the raw A/B matrices directly: - Split lora_B along the output dim (matching base model split) - For k_b (transposed): swap A/B roles per (B@A)^T = A^T @ B^T - For v_b (not transposed): keep standard decomposition - Shared dims use ne[2]=1 for ggml broadcast (no data duplication) Tested with GLM-4.7-Flash LoRA adapter (heiertech/GLM-4.7-Flash-NVFP4-convhist-lora).

marksverdhei · 2026-03-03T07:45:38Z

Squash-merged manually into ht branch.

- Remove duplicated cancel logic from ModelsSelector and ModelsSelectorSheet by deriving loading/cancelling state from the store (issue #1) - Fix race condition: no longer set isLoadingModel=false before cancel completes, preventing brief UI flash (issue #2) - Add MAX_CANCEL_POLL_ATTEMPTS (60) timeout to cancel polling loop to prevent infinite polling if server never transitions (issue #3) - Replace div cancel buttons with proper <button> elements for keyboard accessibility and screen reader support (issue #4) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* webui: add cancel button for in-progress model loading Allow users to cancel a model that is stuck loading or taking too long in the router mode model selector. The cancel button appears next to the loading spinner in both the model selector dropdown/sheet trigger and within individual model option rows. Uses the existing /models/unload endpoint which already supports unloading models in LOADING state. The frontend polling loop is interrupted via AbortController to prevent stale error toasts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * webui: add cancelling state indicator and fix cancel polling - Show orange "Cancelling" indicator with spinner while cancel is in progress - Poll until server confirms model is no longer in LOADING state before clearing the cancelling indicator - Guard against redundant unload calls on already-unloaded models - Keep loadingModelId alive during cancel so selector trigger shows the cancelling state correctly Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(webui): color-coded spinners for model load/unload/cancel states - Loading: green spinner, clockwise - Unloading: red spinner, reverse direction with "Unloading" label - Cancelling: orange spinner, reverse direction - Track unloading state separately in models store Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(webui): address PR review feedback for cancel model loading - Remove duplicated cancel logic from ModelsSelector and ModelsSelectorSheet by deriving loading/cancelling state from the store (issue #1) - Fix race condition: no longer set isLoadingModel=false before cancel completes, preventing brief UI flash (issue #2) - Add MAX_CANCEL_POLL_ATTEMPTS (60) timeout to cancel polling loop to prevent infinite polling if server never transitions (issue #3) - Replace div cancel buttons with proper <button> elements for keyboard accessibility and screen reader support (issue #4) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Sycl tp stage1 (#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths: * Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel chained via depends_on(memcpy_event). 4 SYCL submissions/call. * Large (nelem >= 32768): BF16-compressed. Each device compresses FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's inbox (HALF the PCIe bytes), then decompresses + adds into the local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved -- wins for any tensor where PCIe dominates kernel time. Threshold and BF16 path pattern mirror the CUDA NCCL allreduce. Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes (matches both path layouts: FP32 nelem floats; BF16 outbox+inbox = 2 * nelem uint16_t each). Single alloc+free per device keeps the SYCL pool's strict-LIFO invariant trivial. Initial impl handles N=2 FP32 contiguous tensors. Other cases return false, causing the meta-backend to use its generic butterfly fallback. Per-call sync is intentionally omitted. SYCL in-order queue semantics ensure that the meta-backend's next compute on the same per-device queue waits for our final ADD, and the next allreduce's first op on the same persistent buffer waits via the same queue. Only comm_free does an explicit final wait. OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process in communicator_impl.hpp:47 (condition devices.size() == 1), which is incompatible with llama.cpp's single-process multi-GPU model. Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 + DPC++ nightly): Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16: pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%) tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%) Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE): pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%) tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%) Qwen3-4B Q4_K_M: pp64 = 984.51 t/s, tg16 = 49.29 t/s Llama-3.3-70B in SYCL TP now comfortably beats production layer mode on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on both — the BF16 path is what unlocks the many-medium-allreduces prefill pattern. Build/CMake: no changes. No new dependencies. ~210 lines added across ggml-sycl.h and ggml-sycl.cpp. * Fix comments * documentation update to address PR feedback * Bring over my device-to-device memcpy chagnes * move the dev2dev_memcpy calls to the upstream 7-parameter variety * Fix a typo and remove a trailing whitespace

marksverdhei force-pushed the ht branch from d7717fe to e52bd66 Compare March 3, 2026 07:45

marksverdhei closed this Mar 3, 2026

This was referenced May 12, 2026

Runbook: rebase ht onto upstream/master (parked 2026-05-12) #38

Closed

ci+server: aioc HT-compat probe + /v1/reranking Phase A & B coverage #39

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

convert: support LoRA conversion for MLA kv_b_proj#1

convert: support LoRA conversion for MLA kv_b_proj#1
marksverdhei wants to merge 1 commit into
heiervang-technologies:htfrom
marksverdhei:fix/lora-mla-tensor-ops

marksverdhei commented Mar 3, 2026

Uh oh!

marksverdhei commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

marksverdhei commented Mar 3, 2026

Summary

Test plan

Uh oh!

marksverdhei commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant