Skip to content

Commit 6b8966b

Browse files
author
MrTrent
committed
f16 FA mask: allocate as F16 when flash_attn enabled
From upstream PR #23764. Avoids ggml_cast in graph builder. Our kernel already reads masks as half — no changes needed.
1 parent 8337928 commit 6b8966b

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

src/llama-graph.cpp

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,10 @@ static ggml_tensor * build_attn_inp_kq_mask(
3030
const auto n_tokens = ubatch.n_tokens;
3131
const auto n_stream = cparams.kv_unified ? 1 : ubatch.n_seqs_unq;
3232

33-
ggml_tensor * res = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, n_kv, n_tokens/n_stream, 1, n_stream);
33+
// flash attention requires an f16 mask
34+
const auto type = cparams.flash_attn ? GGML_TYPE_F16 : GGML_TYPE_F32;
35+
36+
ggml_tensor * res = ggml_new_tensor_4d(ctx, type, n_kv, n_tokens/n_stream, 1, n_stream);
3437
ggml_set_input(res);
3538
ggml_set_name(res, "attn_inp_kq_mask");
3639

0 commit comments

Comments
 (0)