Be able to set FA offset via command line argument

[ikawrakow]

This PR is a follow up of #1196. Instead of having the CUDA flash attention offset be a compile time constant, it can now be set via a command-line argument

-cuda fa-offset=value

value must be a valid floating point number in the interval [0...3] (this is checked and if the supplied value is outside this interval it is ignored).

By the default the offset is zero. If you find that a model works up to a given context length but then starts producing gibberish/incoherent output/endless repetitions, it is very likely it is due to f16 overflow in the FA calculation, and using this command line option is likely to solve it.

[ghost]

Is there a way to verify precisely if and to what extent the f16 overflow problem is happening?

[ikawrakow]

Is there a way to verify precisely if and to what extent the f16 overflow problem is happening?

I encountered it for the first time yesterday when computing PPL for MiniMax-M2 for a context of 64k tokens. The initial mainline PR that adds a hard-coded offset of ln(2) = 0.6931 (as I did in #1196) is from Dec 5 2025, modified to an offset of ln(8) because of a Granite model 3 weeks ago. Which tells us that the problem is very rare, else it would have been encountered and fixed much earlier. On the other hand, I vaguely remember some people complaining about gibberish (or NaNs, which manifest as endlessly repeating GGGG or DDDD) encountered at very long context, so it may as well be that fp16 precision is inadequate, at least for some models, when computing FA for very long contexts, but it is so rare that some users actually go there.