[CUDA] FA+MTP crash: ggml_cuda_flash_attn_ext_mma_f16 fatal error in common_context_can_seq_rm on SM87 (Jetson Orin)

[rivascd]

Summary

Using --flash-attn on with --spec-type draft-mtp causes a fatal crash in
ggml_cuda_flash_attn_ext_mma_f16 on SM 8.7 (Jetson Orin Nano). The crash
occurs during common_context_can_seq_rm after the MTP draft context
initializes. Workaround: --flash-attn off — MTP works correctly without FA.

Environment

• Hardware: Jetson Orin Nano Developer Kit
• CUDA SM: 8.7
• JetPack: 7.2 / CUDA 13.2
• llama.cpp: b9592
• Build flags: GGML_CUDA=ON GGML_CUDA_F16=ON GGML_CPU_AARCH64=ON CMAKE_CUDA_ARCHITECTURES=87
• Model: Gemma 4 E4B (unsloth GGUF, gemma4 architecture)
• Draft model: MTP head GGUF (mtp-gemma-4-E4B-it.gguf)

Reproduction

llama-server \
  -m gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
  --model-draft mtp-gemma-4-E4B-it.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --n-gpu-layers 999 \
  --ctx-size 16384 \
  --parallel 1 \
  --no-mmap \
  --batch-size 128 \
  --ubatch-size 128 \
  --flash-attn on \
  --cache-type-k q4_0 \
  --fit off \
  --no-warmup \
  --swa-full \
  --cache-prompt \
  --context-shift \
  --keep 256 \
  --cache-ram 512 \
  --host 0.0.0.0 \
  --port 8081

Observed behavior

Fatal abort in fattn.cu:110 during startup. MTP draft context loads and logs
shared KV cache layers (see below), then common_context_can_seq_rm triggers
a decode that hits the MMA F16 flash attention kernel and crashes.

Notable pre-crash warnings:

W llama_kv_cache: layer 3: sharing with layer 41.
W llama_kv_cache: layer 0: sharing with layer 40. ← same pointer
W llama_kv_cache: layer 1: sharing with layer 40. ← same pointer
W llama_kv_cache: layer 2: sharing with layer 40. ← same pointer

Stack trace

#3 ggml_cuda_flash_attn_ext_mma_f16(ggml_backend_cuda_context&, ggml_tensor*)
#4 ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*)
#5 ggml_backend_sched_graph_compute_async
#6 llama_context::graph_compute(ggml_cgraph*, bool)
#7 llama_context::process_ubatch(...)
#8 llama_context::decode(llama_batch const&)
#9 llama_decode
#10 common_context_can_seq_rm(llama_context*)
#11 server_context_impl::load_model(common_params&)

Expected behavior

FA should work alongside MTP, or fail gracefully rather than aborting.

Workaround

--flash-attn off — MTP initializes and runs correctly. Confirmed ~1.4×
speedup (36–42% draft acceptance, ~24 t/s effective vs ~17 t/s baseline)
on this hardware despite FA being disabled.

Notes

The crash is reproducible 100% of the time with FA on + MTP. --fit off and
--no-warmup do not prevent it — the crash occurs in common_context_can_seq_rm
which runs unconditionally after draft context init.

[rivascd]

Full log

Details

`0.00.101.532 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg) 0.00.101.544 I device_info: 0.00.219.357 I - CUDA0 : Orin (7485 MiB, 6152 MiB free) 0.00.219.393 I - CPU : CPU (7485 MiB, 7485 MiB free) 0.00.219.689 I system_info: n_threads = 6 (n_threads_batch = 6) / 6 | CUDA : ARCHS = 870 | FORCE_MMQ = 1 | NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 0.00.221.061 I srv init: using 5 threads for HTTP server 0.00.221.555 I srv start: binding port with default address family 0.00.222.972 I srv llama_server: loading model 0.00.223.013 I srv load_model: loading model '/home/UserName/.cache/huggingface/hub/models--unsloth--gemma-4-E4B-it-qat-GGUF/snapshots/ce17dde5087147ba8dddf882a152cc70cb913e32/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf' 0.01.678.812 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden 0.01.680.283 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden 0.01.790.495 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list 0.06.929.393 W llama_context: n_ctx_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized 0.06.930.690 W llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com//pull/13194#issuecomment-2868343055) 0.07.188.768 I srv load_model: loading draft model '/home/UserName/.cache/huggingface/hub/models--unsloth--gemma-4-E4B-it-qat-GGUF/snapshots/5c3b4e2a0de02a5e5d572a9cad205bf7b3e5ab5b/mtp-gemma-4-E4B-it.gguf' 0.08.583.631 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden 0.08.584.834 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden 0.08.690.314 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list 0.08.909.502 W llama_context: n_ctx_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized 0.08.909.716 W llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com//pull/13194#issuecomment-2868343055) 0.08.909.923 W llama_kv_cache: layer 3: sharing with layer 41. k = 0x1a5b21000, v = 0x1a6421000 0.08.910.293 W llama_kv_cache: layer 0: sharing with layer 40. k = 0x1c09a1000, v = 0x1c0e21000 0.08.910.300 W llama_kv_cache: layer 1: sharing with layer 40. k = 0x1c09a1000, v = 0x1c0e21000 0.08.910.301 W llama_kv_cache: layer 2: sharing with layer 40. k = 0x1c09a1000, v = 0x1c0e21000 0.08.930.270 I srv load_model: initializing slots, n_slots = 1 0.09.132.983 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp' 0.09.133.033 I common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=2560, backend_sampling=1 0.09.133.047 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default] /home/UserName/llama.cpp/ggml/src/ggml-cuda/fattn.cu:110: fatal error [New LWP 20303] [New LWP 20302] [New LWP 20295] [New LWP 20294] [New LWP 20293] [New LWP 20292] [New LWP 20291] [New LWP 20290] [New LWP 20287] [New LWP 20286]

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffff84a67b74 in __GI___wait4 (pid=20304, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0 0x0000ffff84a67b74 in __GI___wait4 (pid=20304, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x0000ffff84078974 in ggml_print_backtrace () from /home/UserName/llama.cpp/build-b9592/bin/libggml-base.so.0
#2 0x0000ffff84078b2c in ggml_abort () from /home/UserName/llama.cpp/build-b9592/bin/libggml-base.so.0
#3 0x0000ffff7f959f68 in ggml_cuda_flash_attn_ext_mma_f16(ggml_backend_cuda_context&, ggml_tensor*) () from /home/UserName/llama.cpp/build-b9592/bin/libggml-cuda.so.0
#4 0x0000ffff7f99edbc in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/UserName/llama.cpp/build-b9592/bin/libggml-cuda.so.0
#5 0x0000ffff84094c9c in ggml_backend_sched_graph_compute_async () from /home/UserName/llama.cpp/build-b9592/bin/libggml-base.so.0
#6 0x0000ffff84219ec4 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/UserName/llama.cpp/build-b9592/bin/libllama.so.0
#7 0x0000ffff8421c9c0 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/UserName/llama.cpp/build-b9592/bin/libllama.so.0
#8 0x0000ffff8422416c in llama_context::decode(llama_batch const&) () from /home/UserName/llama.cpp/build-b9592/bin/libllama.so.0
#9 0x0000ffff84225d5c in llama_decode () from /home/UserName/llama.cpp/build-b9592/bin/libllama.so.0
#10 0x0000ffff8472df50 in common_context_can_seq_rm(llama_context*) () from /home/UserName/llama.cpp/build-b9592/bin/libllama-common.so.0
#11 0x0000ffff84f6a8a4 in server_context_impl::load_model(common_params&) () from /home/UserName/llama.cpp/build-b9592/bin/libllama-server-impl.so
#12 0x0000ffff84ec40f8 in llama_server(int, char**) () from /home/UserName/llama.cpp/build-b9592/bin/libllama-server-impl.so
#13 0x0000ffff849b84c4 in __libc_start_call_main (main=main@entry=0xaaaad1541140

, argc=argc@entry=38, argv=argv@entry=0xffffd1b62888) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#14 0x0000ffff849b8598 in __libc_start_main_impl (main=0xaaaad1541140 , argc=38, argv=0xffffd1b62888, init=, fini=, rtld_fini=, stack_end=) at ../csu/libc-start.c:360
warning: 360 ../csu/libc-start.c: No such file or directory
#15 0x0000aaaad15411b0 in _start ()
[Inferior 1 (process 20284) detached]
Aborted (core dumped)`