cpu: de-duplicate some of the operators and refactor

[cmdr2]

Note: This PR only deletes lines from ggml-cpu.c, it does not modify any functions in it. I'm not sure why git diff tries to combine them as changes. Standard diff shows the correct diff of ggml-cpu.c: https://gist.github.com/cmdr2/a76df5af311417619788e8330b1908b3

This PR de-duplicates some of the easily-templatized functions in ggml-cpu.c. It takes inspiration from binbcast.cu and common.cuh.

• Binary: add, sub, mul, div
• Unary: abs, sgn, neg, step, tanh, elu, relu, sigmoid, hardsigmoid, exp, hardswish, sqr, sqrt, sin, cos, log

This removes the op implementation functions from ggml-cpu.c (around 2000 lines). As a side-effect, all the functions now support bf16 as well as non-contiguous src1.

The performance is the same as the current implementation. It also passes all the runners on ggml-ci, which tested non-contiguous inputs (in SAM) and vDSP on Mac.

[ggerganov]

Hopefully all C++ compilers support structured bindings today. If we encounter issues, we might have to return simple struct of integers.

[ggerganov]

Wait for @slaren's review before merging. Thanks!

[cmdr2]

Wait for slaren's review before merging. Thanks!

@ggerganov Yes, will do :)

Thanks for the comments, fixed them in the latest commit. The CI runner also passes.

[slaren]

I will review when I have a chance, but I may not be able to do it for a few days.

[cmdr2]

Thanks @slaren No hurry :)

[cmdr2]

@slaren Gentle ping, thanks :)

[slaren]

Sorry for taking for so long to get to this, other than the minor suggestions it looks good to me.

[cmdr2]

Thanks @slaren , no problem! I've fixed the comments. Shall I go ahead and squash-merge?

[ggerganov]

Yup, go ahead. Thanks!

[LostRuins]

Hi all, apologies for the necropost.

I was trying to port some old code in TTS.cpp that was using an earlier version of the ggml library to a newer version of ggml, and I'm running into a new assert after this pull.

Specifically, there is now a constraint for ggml_are_same_shape(src0, src1) at

ggml/src/ggml-cpu/binary-ops.cpp

Line 65 in 2621d5a

GGML_ASSERT(ggml_are_same_shape(src0, src1));

that constraint was not there before, for example in ggml_compute_forward_mul_f32

ggml/src/ggml-cpu/ggml-cpu.c

Line 5432 in 4b344ba

static void ggml_compute_forward_mul_f32(

May I check what should I change in order to avoid this assert? Is there a reason for this new limitation?

Thanks!