feat: cherry-pick upstream DFlash spec-decode (#22105 + #25110 + companions)#201
Merged
Conversation
github-actions
Bot
added
documentation
…xtracted from upstream ggml-org#24707 minus the unrelated eagle3 fix)
* spec: add DFlash v2 support * dflash: support sliding window attention per layer_types * docs: add dflash section --------- Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
* dflash: refactor draft model conversion * apply fix for eagle3 convert
…gml-org#25215) llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null, but the tensor's buffer can be unallocated (NULL) when a graph only stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer and aborts with GGML_ASSERT(buffer). Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the adjacent kq_mask inputs already use in these two functions. When the buffer is unallocated there is no data to upload, so skipping is correct. Fixes ggml-org#25191 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> (cherry picked from commit a410713)
* spec: support spec-draft-p-min in DFlash * dflash: add n_min guard * dflash: guard both n_min and n_max (cherry picked from commit 152d337)
TheTom
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks upstream DFlash speculative decoding onto the fork.
Commits (dependency order)
llama: add n_embd_inp_enc encoder-input hparams— minimal DFlash dependency, extracted from upstream spec: fix segfault error on long prompts for eagle3 ggml-org/llama.cpp#24707 without its unrelated eagle3 fix (eagle3.cppbyte-identical to base). Backward-compatible fallback ton_embd_inp().d1b34251)fa72bc68)a4107133a) — required companion: the abort it guards is triggered by DFlash's KV-injection pass, and the fork has the vulnerableset_input_k_rotpattern152d337fa)Conflict resolution (3 small files)
common/common.hneed_n_rs_seq()upstream superset;common/speculative.cppkept fork'shas_mtp+ addedhas_draft_dflash(dropped unused rename,-Werror); docs took upstream.Validation
CUDA
llama-clicompiles + links clean;test-llama-archspasses (DFlash arch registration). Turbo-heavy files auto-merged clean.