New op: ggml_sum_rows_ext

[ikawrakow]

In the models that use HC and sparse attention there is often the need to sum over the rows of a tensor. ggml does not provide this functionality, so on needs to permute the zeroth and 1st dimension, make the resulting tensor contiguous using ggml_cont (because the ggml_sum_rows implementations do not support non-contiguous tensors), and only then use ggml_sum_rows to sum over the elements of the row. This is inefficient and may result in extra RAM/VRAM usage.

This PR adds ggml_sum_rows_ext, which takes as an additional argument the dimension over which the summation should be done. With that, the above simplifies to ggml_sum_rows_ext(ctx, tensor, 1).

As a test of the impact, in this PR I have used the new ggml_sum_rows_ext functionality in the openPangu model. The result is quite striking: ~60% better PP and ~15% better TG for short context. For long context the inference time is dominated by the indexer/attention computations, so the impact there is lower. Nevertheless, the average performance improvement for contexts between 0 and 64k tokens is 32% (PP) and 6.5% (TG). This is on a 2x3090+Ryzen-3995WX system with 20 out of 48 MoE layers left in RAM.

Running CPU-only the performance gain is less striking, but I still see ~10% better PP and ~6% better TG.

I guess, the main reason the effect is so large on CUDA is that the rows of the permuted tensor only have 4 elements, so in the CUDA implementation only 4 out of 32 threads in a WARP do actual work.