perf(hip): enable -funsafe-math-optimizations for the ROCm backend

[RapidMark]

Overview

The HIP/ROCm backend is denied the fast math CUDA gets, so it runs without that speedup. We added -funsafe-math-optimizations to let AMD use faster math while staying away from NaNs.

Additional information

Validated on RDNA4 (gfx1201, ROCm 7.0.2): builds clean, test-backend-ops 12508/12508 (including EXPM1), ~+8–15% on FLASH_ATTN.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - I used AI to write the initial comments, then went back and rewrote them myself.

[ggml-gh-bot]

Hi @RapidMark, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[RapidMark]

It's a one line change in the ggml/src/ggml-hip/CMakeLists.txt specifically fot HIP/ROCm... not AI written... just wrote the notes... thought it sounds better than me writting it... I re-wrote the notes and pushed...

[omerguzelelectronicguy]

@RapidMark thanks for the solution. I thought I have same problem I am getting error when I don't gave --flash-attn off flag. I made the change in your PR but it doesn't solve in my case.

(base) omerg@fedora:/GAME/github/llama.cpp$ ./build/bin/llama-bench -m ~/Downloads/Qwen3.6-27B-UD-IQ2_M.gguf 
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 12272 MiB):
  Device 0: AMD Radeon RX 6750 XT, gfx1031 (0x1031), VMM: no, Wave Size: 32, VRAM: 12272 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
/GAME/github/llama.cpp/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1110: GGML_ASSERT(max_blocks_per_sm > 0) failed
[New LWP 81681]
[New LWP 81674]

This GDB supports auto-downloading debuginfo from the following URLs:
  <ima:enforcing>
  <https://debuginfod.fedoraproject.org/>
  <ima:ignore>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fdcbb682412 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007fdcbb682412 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007fdcbb67662c in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007fdcbb676674 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007fdcbb6e624f in wait4 () from /lib64/libc.so.6
#4  0x00007fdcbff2301b in ggml_print_backtrace () from /GAME/github/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007fdcbff2318d in ggml_abort () from /GAME/github/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007fdcbed02a54 in void launch_fattn<256, 16, 2>(ggml_backend_cuda_context&, ggml_tensor*, void (*)(char const*, char const*, char const*, char const*, char const*, int const*, float*, HIP_vector_type<float, 2u>*, float, float, float, float, unsigned int, float, int, HIP_vector_type<unsigned int, 3u>, int, int, int, int, int, int, int, int, int, int, int, long, int, int, long, int, int, int, int, int, long), int, unsigned long, int, bool, bool, bool, int) () from /GAME/github/llama.cpp/build/bin/libggml-hip.so.0
#7  0x00007fdcbecf2eb5 in void ggml_cuda_flash_attn_ext_tile_case<256, 256>(ggml_backend_cuda_context&, ggml_tensor*) () from /GAME/github/llama.cpp/build/bin/libggml-hip.so.0
#8  0x00007fdcbeb4bef3 in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /GAME/github/llama.cpp/build/bin/libggml-hip.so.0
#9  0x00007fdcbeb487ac in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /GAME/github/llama.cpp/build/bin/libggml-hip.so.0
#10 0x00007fdcbff3f3bb in ggml_backend_sched_graph_compute_async () from /GAME/github/llama.cpp/build/bin/libggml-base.so.0
#11 0x00007fdcbf645500 in llama_context::graph_compute(ggml_cgraph*, bool) () from /GAME/github/llama.cpp/build/bin/libllama.so.0
#12 0x00007fdcbf64991b in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /GAME/github/llama.cpp/build/bin/libllama.so.0
#13 0x00007fdcbf64d7bb in llama_context::decode(llama_batch const&) () from /GAME/github/llama.cpp/build/bin/libllama.so.0
#14 0x00007fdcbf64f81e in llama_decode () from /GAME/github/llama.cpp/build/bin/libllama.so.0
#15 0x00007fdcc000102b in test_prompt(llama_context*, int, int, int) () from /GAME/github/llama.cpp/build/bin/libllama-bench-impl.so
#16 0x00007fdcc00154df in llama_bench(int, char**) () from /GAME/github/llama.cpp/build/bin/libllama-bench-impl.so
#17 0x00007fdcbb60a681 in __libc_start_call_main () from /lib64/libc.so.6
#18 0x00007fdcbb60a798 in __libc_start_main_impl () from /lib64/libc.so.6
#19 0x00000000004003b5 in _start ()
[Inferior 1 (process 81673) detached]
Aborted                    (core dumped) ./build/bin/llama-bench -m ~/Downloads/Qwen3.6-27B-UD-IQ2_M.gguf
(base) omerg@fedora:/GAME/github/llama.cpp$ ./build/bin/llama-bench -m ~/Downloads/Qwen3.6-27B-UD-IQ2_M.gguf --flash-attn off
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 12272 MiB):
  Device 0: AMD Radeon RX 6750 XT, gfx1031 (0x1031), VMM: no, Wave Size: 32, VRAM: 12272 MiB
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B IQ2_M - 2.7 bpw     |  10.26 GiB |    27.32 B | ROCm       |  -1 |   0 |           pp512 |        281.85 ± 0.58 |
| qwen35 27B IQ2_M - 2.7 bpw     |  10.26 GiB |    27.32 B | ROCm       |  -1 |   0 |           tg128 |         17.88 ± 1.04 |

build: e36a602ba (9659)

[RapidMark]

@omerguzelelectronicguy This PR is only a build-time math flag (-funsafe-math-optimizations), it doesn't touch flash attention, so it won't change that crash either way.

The assert you're hitting is in the FA launchers occupancy calc coming back as 0 (from what I can see).

This looks like a launch-config bug on gfx1031, probably worth its own issue with that log attached. For now --flash-attn off is the right workaround.

I do have a 6700XT, so I can try to see if I can find the issue you're having and will reply to you directly... but it's not in the scope of this PR.

[RapidMark]

@omerguzelelectronicguy I dug into your crash... and the good news is you can get it running with --flash-attn off (you found this — it's the clean workaround).

The bad news is... its an upstream bug, which I cannot look into right now because my 6700 XT is in a windows box and I don't have space in any linux box to test and run it (HIP is linux only, we use Vulkan on Windows).

If you want FA, use the (linux) Vulkan build, instead of the HIP/ROCm as Vulkan has it's own FA that does work on RDNA2.

[omerguzelelectronicguy]

@RapidMark Thank you for kindly helps and suggestions.

[jasonriedy]

Please never make this the default. -ffast-math enables a hodge-podge of different possible code manipulations that also lead to different hodge-podge behaviors / errors.

If you're looking at faster special functions, those should be called specifically. See the intrinsics at https://rocm.docs.amd.com/projects/HIP/en/latest/reference/math_api.html . If you're looking for optimizations that ignore NaNs, please don't. They are no end of pain to debug. The sign of zero mostly matters in complex arithmetic so long as you can tolerate a projective rather than affine arithmetic in the reals.

These are aspects that can be tuned and evaluated separately. And to me they should be handled separately. But then I'm on the IEEE 754 and P3109 committees, so I may be a tad biased.

[cb88]

Please never make this the default. -ffast-math enables a hodge-podge of different possible code manipulations that also lead to different hodge-podge behaviors / errors.

If you're looking at faster special functions, those should be called specifically. See the intrinsics at https://rocm.docs.amd.com/projects/HIP/en/latest/reference/math_api.html . If you're looking for optimizations that ignore NaNs, please don't. They are no end of pain to debug. The sign of zero mostly matters in complex arithmetic so long as you can tolerate a projective rather than affine arithmetic in the reals.

These are aspects that can be tuned and evaluated separately. And to me they should be handled separately. But then I'm on the IEEE 754 and P3109 committees, so I may be a tad biased.

CUDA backend is already using it from what he says... precise math is often slower than what the hardware can do in the case of LLMs are fairly tolerant of imprecise math... if can be made to work there is no reason not to its not scientific compute I'd totally agree with you on that side of things.

[Diablo-D3]

I agree with the above assessment. Either remove it from the CUDA path, or add it to the HIP path, it makes no sense to have one but not the other.

[jasonriedy]

As a fuddy-duddy, this compiler option lumps in a bunch of compiler- and architecture-dependent aspects. That doesn't matter much on the side that uses only one compiler but may matter on the sides that can use multiple compilers (at the moment).

Breaking this into multiple options likely is a good choice moving forward. Signed zeros and infinities may not exist in some situations. Signalling NaNs may not exist in other situations (we can hope). And in some systems NaNs do not exist at all. Or Infs. It depends on the HW.

Think of this as a form of quantization. Models that are trained along with a target's quantization methods potentially can perform better. Similarly, models trained with a target's available floating-point arithmetic potentially can perform better. I've seen some of those papers cross my desk.

I have no data here and only experience debugging issues. I would suggest providing methods to enable/disable different floating-point optimizations without the -ffast-math option. Platforms and models then could supply their presets.

I also have no time to make this happen. And the decisions are made by those doing the work.

[jasonriedy]

Oh, and background, I don't have any CUDA-supporting hardware. So I haven't paid attention to that side.

[RapidMark]

Thanks @jasonriedy, but worth being precise about scope: this is -funsafe-math-optimizations, not -ffast-math, and the difference maps onto your concerns:

1. NaNs/Infs: -funsafe-math does not enable -ffinite-math-only, so NaN/Inf semantics are preserved. I avoided -ffast-math for exactly that reason, its -ffinite-math-only mis-compiles expm1f overflow to -nan and won't even compile here (INFINITY in the softmax mask trips -Wnan-infinity-disabled).

2. Signed zeros: it does imply -fno-signed-zeros, but these kernels are real-valued matmul/attention/elementwise there's no complex arithmetic where ±0 matters.

It also mirrors the CUDA backend's nvcc -use_fast_math (also inf-safe), so this is parity between the two paths, validated on RDNA4 with test-backend-ops 12508/12508 passing including EXPM1.

Finally... CUDA has flags which make it inherently faster, but HIP/ROCm does not... this just ads parity... and if you want to debug... you turn off optimizations anyway...