ggml : add support for CPU f16->f16 GGML_OP_SET_ROWS

[fairydreaming]

Overview

This PR adds support for CPU GGML_OP_SET_ROWS where src0 type is f16 and dst type is also f16. This will come in handy in models based on DeepSeek Sparse Attention (DeepSeek V3.2, V4, GLM 5.0, 5.1, 5.2) where we have to "sparsify" KQ mask based on lightning indexer indices. With f16 support in GGML_OP_SET_ROWS it will be possible to use f16 FA KQ masks without any extra casts, this will reduce memory usage and simplify code.

Additional information

This PR adds only CPU support with ggml_compute_forward_set_rows_f16() implementation based on existing ggml_compute_forward_set_rows_f32() with from_float calls replaced with a simple memcpy().

Backend OPs tests for ggml_set_rows() were also updated to include extra src type parameter and f16->f16 test cases.

CUDA implementation will follow next.

CC @am17an

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[am17an]

You can use the same function as ggml_compute_forward_set_rows_f16, just pass in the template param as type

[fairydreaming]

Sorry, no idea what do you mean, could you elaborate?

[am17an]

I mean use the same function as the f32 version, just use a template instead of adding a new function

[fairydreaming]

I mean use the same function as the f32 version, just use a template instead of adding a new function

You mean to add src0 type template argument in ggml_compute_forward_set_rows_f32() (suffix would be extra confusing, so I'd have to rename it to ggml_compute_forward_set_rows_impl()) and switch logic inside with if constexpr instead of using two different implementations?

Good idea, I don't like touching existing working code but will try.

[am17an]

Yes, it's okay to change because the bodies are identical except the from_float vs memcpy for the copy

[am17an]

These should be GGML_ASSERT, I think otherwise they will not fire in release builds

[fairydreaming]

Shall I update them in ggml_compute_forward_set_rows_f32() too?

[am17an]

If you combining the two functions then yes

[ggerganov]

To remind, the logic for when to use assert and GGML_ASSERT is that the latter is typically used when we have to assert some user-provided values and the former is used to clarify the assumptions that would be always true unless there is a bug in the internal logic.

Technically, at this point we should have already GGML_ASSERTed (typically in the ggml.c implementation of ggml_set_rows) the conditions required for the op to make sense and here we just need to assert. GGML_ASSERT here would be needed only for things that are specific to the CPU implementation. So if I am reading correctly:

    assert(ne0  == nc);
    assert(ne2  == ne02);
    assert(ne3  == ne03);
    GGML_ASSERT(src0->type == GGML_TYPE_F16);
    GGML_ASSERT(dst->type  == GGML_TYPE_F16);
    assert(ne02 % ne11 == 0);
    assert(ne03 % ne12 == 0);

[am17an]

Why not just use GGML ASSERT everywhere? If the performance is the same then I would prefer that as it's easier to remember.

[ggerganov]

The asserts are pre-processed away in release builds, so the performance should be better as we do fewer checks. Though likely it's something that's hard to measure in practice.

We are not very strict about this convention, so if you prefer you can use GGML_ASSERT. It's something that can be refined in refactoring passes.

[am17an]

Yes a potential footgun with assert is exactly what you said about the release builds. The debug build path is not exercised and something like this which should be ideally crash the process as it invalidates an invariant just goes through.

llama.cpp/src/llama-kv-cache.cpp

Lines 2547 to 2548 in e1a7a5a

	assert(status == LLAMA_MEMORY_STATUS_SUCCESS);

If a branch is never taken except I would expect these GGML_ASSERTs to be 0-cost on modern processors which execute instructions speculatively

[am17an]

We also have the check the dst type so that it's f16. Or perhaps check src->type == dst->type

[fairydreaming]

Looks like the Vulkan backend needs some fixes in ggml_backend_vk_device_supports_op().