Performance issues with cuBLAS and a bug

[cmp-nct]

Performance with cuBLAS isn't there yet, it is more a burden than a speedup with llama eval in my tests.
In a simple benchmark case it is absolutely amazing, getting 10 million elements multiplied in F32 goes from 1+ seconds down to 20 milliseconds. So the improvement is a blast!

But in the llama case the overhead seems to be enormous, when enabling it generically the average computation time shoots from 300ms up to 1500ms using cuBLAS.
I feel like the memory should have been prepared beforehand and I don't think the thousands of CUDA cores are used.
Is that loop really the best way to do it ?

      for (int64_t i03 = 0; i03 < ne03; i03++) {
            for (int64_t i02 = 0; i02 < ne02; i02++) {
                const float * x = (float *) ((char *) src0->data + i02*nb02 + i03*nb03);
                const float * y = (float *) ((char *) src1->data + i02*nb12 + i03*nb13);
                float * d = (float *) ((char *) dst->data + i02*nb2 + i03*nb3);
                CUDA_CHECK(cudaMemcpyAsync(d_X, x, sizeof(float) * x_ne, cudaMemcpyHostToDevice, g_cudaStream));
                CUDA_CHECK(cudaMemcpyAsync(d_Y, y, sizeof(float) * y_ne, cudaMemcpyHostToDevice, g_cudaStream));
                // compute
                CUBLAS_CHECK(
                    cublasSgemm(g_cublasH, CUBLAS_OP_T, CUBLAS_OP_N,
                            ne01, ne11, ne10,
                            &alpha, d_X, ne00,
                                    d_Y, ne10,
                            &beta,  d_D, ne01));

ggml_compute_forward_mul_mat_use_blas() is evaluated too often, I think it would be better to make a generic "check" function that uses the node Operation type as input.
I'm experimenting with another approach in that case, not finished yet but I think we need to calculate the flops required.
Then the function is called too often (init, compute, finalize), I'm just modifying it to set a flag in the tensor instead but that's just experimental atm.

There is a bug in ggml.c which causes the matrix multiplication being executed once for all threads in CUDA when we have a 32 bit input and output.

else if (node->src0->type == GGML_TYPE_F32 && node->src1->type == GGML_TYPE_F32) {
cur = 0;
}

I think it should rather be like this:

else if (node->src0->type == GGML_TYPE_F32 && node->src1->type == GGML_TYPE_F32) {
                            cur = 0;
                            if (ggml_compute_forward_mul_mat_use_blas(node->src0, node->src1, node)) {
                                node->n_tasks = 1;
                                cur = GGML_TYPE_SIZE[GGML_TYPE_F32]*(node->src0->ne[0]*node->src0->ne[1]);
                            }

If n_tasks is not reduced to 1 then threads are associated with the cuda one it will currently not stop each thread from using cuda (and surprisingly no crashes)

Finally: the computation with flops taking into consideration the memory time using a quick local benchmark at startup might be a good generic solution to decide if cuda should be enabled. Generically.
But for our use case we know beforehand if we need it, every layer is known and sizes are known.
So that flag in the tensor struct should be set manually (I currently use -1,1 and 0: -1 = generic, 1 = force cuda if possible, 0 = disable cuda)
There should be a way to use the full cuda cores, I have no experience in it (yet) but I'm sure we can get the multiplication through a lot faster.
It looks like the 30+attention heads are calculated in sequence, cuda can do that in one shot.

Also when cuda is enabled we need to make sure that the big matrix tensors are proper in memory for cuda, or they will be skipped. I didn't get to that point yet, still fighting ;)

[slaren]

I don't think we should expect to be able to do everything with cuBLAS, if the user has enough VRAM for that they should be using a GPU implementation of llama instead. We could investigate the possibility of always keeping the largest matrices in the GPU, up to the available VRAM. I am not sure if that is going to be enough to use cuBLAS for generation, but would improve the performance with batch processing.

Most of the time the GPU is sitting idle waiting for data, that is expected for the way this is done. We could likely improve utilization by allowing ggml to work on multiple mat muls simultaneously. There was some discussion about this in #1094, but I am not familiar with the graph processing code in ggml and I am not sure what would take to do that.

The loop does nothing in llama.cpp, ne03 and ne02 are always 1. Otherwise we could improve performance by working on these matrices at the same time here, but that's not the case.

[slaren]

I have tried implementing a cache for the quantized matrices (ie. the weights), and even in the best case where the entire model fits in VRAM, for me this is only ~30% faster in the perplexity test. Considering how much more VRAM this requires, I am not sure that it is worth it

Anyway, if anybody wants to give it a try, it is at slaren@5a606d5.

[SlyEcho]

It's true that for F32 x F32 it doesn't change the thread number but the function ggml_compute_forward_mul_mat_f32() will return return early in the BLAS case from threads other than 0;

static void ggml_compute_forward_mul_mat_f32(
        const struct ggml_compute_params * params,
        const struct ggml_tensor * src0,
        const struct ggml_tensor * src1,
              struct ggml_tensor * dst) {
// ...
#if defined(GGML_USE_ACCELERATE) || defined(GGML_USE_OPENBLAS) || defined(GGML_USE_CUBLAS)
    if (ggml_compute_forward_mul_mat_use_blas(src0, src1, dst)) {
        if (params->ith != 0) {
            return;
        }
// ...

The variable cur is used to calculate the temporary work buffer size, F32 x F32 doesn't need any temp workspace becaus it doesn't need to dequantize the matrixes, so it is 0.

The GEMM computation is not a very big part of the time taken, it really depends on the hardware but slow memory, slow video memory, slow PCI bus can totally ruin the performance.

@slaren, 30% improvement is pretty big in my opinion! On slower hardware it could be even more significant.

[slaren]

If there is interest in that, I can implement it properly and open a PR, however do it in a safe way in ggml we would need #1093 to be able to tag the matrices that are constant and can be cached in the GPU safely. Currently, I am just assuming that when multiplying q x f32, the quantized matrix is the weights and therefore constant and safe to cache, but that may not always be the case for other uses of ggml.

[cmp-nct]

I think that a smart mix of GPU and CPU can be a very interesting library extension. Any improvements in that area will be awesome.

The 1093 suggestions was kind of approved by GG: that is to extend the tensor_node with more information.
Such as "flag_enable_cuda=-1,0,1" or a "HR name", I am also tagging it to be of "high" or "low" performance, to enable a better decision making for hybrid core scheduling (Intel 12,13+ gen).
One more such flag would be if the tensor is constant, another one is if the tensor could be precached in VRAM (or on demand).
In my own source I use a struct but GG was probably right to just use constant lenghts and non structs, separate them by prefix.
Basically there is no downside to add anything into it, it's tiny in length.
To add something two steps are needed:

1. add the new variables
2. initialize them in the tensor_impl function

Strange Performance:
The current performance disadvantage when enabling cuBLAS for all MUL operations is huge, it's 4-5 times slower than on CPU.
I've not benchmarked it yet, something must be increddibly slow. the GPU can hardly be slower in calculation, so the only thing that comes to my mind is the async memory copy being slow ?

Of course a tiny multiplication is not worth to be moved to GPU if the graph is mostly CPU based, but it shouldn't take 4-5 times longer when doing it that way. Maybe someone has insight into that already ?
I've had MUL timings of 1500ms per layer, where CPU is at below 300ms. That's odd.

[slaren]

so the only thing that comes to my mind is the async memory copy being slow ?

I think that's all there is to it. The cuBLAS matrix multiplication is very fast, but the latency is high due to having to move the data between main RAM and the GPU. At some point, the matrix is big enough that it takes longer to multiply it in the CPU than to copy the data to and from the GPU. But that only happens with big enough matrices, in our case is when processing in large batches. @SlyEcho showed that here. It is the same issue with OpenBLAS, it is only worth using it if the cost of dequantizing the matrices is lower than the time gained from the faster matrix multiplication. 4-5 times slower doesn't seem wrong, that's why the minimum batch size to use BLAS is 32.

[cmp-nct]

Though it's really a huge delay. The PCIE bus is faster than RAM was a couple years ago. I'll try benchmark that area (though the async part makes it a bit harder).

One function I was working on (which needs the flags mentioned earlier to be useable) is an actual smart decision maker.
I wanted to benchmark the system on startup with a quick memory, gpu and cpu test.
Then calculate the flops required and the memory movement required for GPU and for CPU work.
This can be used to make a smart decision on actual expected performance difference (better than >32 dimensions*3)

[gjmulder]

@cmp-nct @slaren

EDIT2: Even though I had the performance power governor enabled the PCIe buses were powersaving by switching to 2.5 GT/sec. All results below are with 8GT/sec lanes, across 4 old GTX 1080Ti GPUs with varying numbers of PCIe lanes.

Here's some data points which may be of interest with a 65B q4_0 model, and for the first 406 lines of wiki.test.raw .

CPU	GPU	PCIe Gen3 Lanes	Batch Size	GPU Memory Util % (*)	Perplexity per pass (secs)
AMD 1650 16 Core	NVidia 1080Ti	4	8	0	233
AMD 1650 16 Core	NVidia 1080Ti	4	64	7	206
AMD 1650 16 Core	NVidia 1080Ti	4	128	8	124
AMD 1650 16 Core	NVidia 1080Ti	4	256	6	65
AMD 1650 16 Core	NVidia 1080Ti	4	512	7	47
AMD 1650 16 Core	NVidia 1080Ti	8	512		36
AMD 1650 16 Core	NVidia 1080Ti	16	512		32
AMD 1650 16 Core	NVidia 1080Ti	16	512	10	31

EDIT1:
(*) Percent of time over the past sample period during which global (GPU device) memory was being read or written.

$ git log | head -1
commit 54bb60e26858be251a0eb3cb70f80322aff804a0

[SlyEcho]

It may be useful to see some kind of profile of what the GPU is doing in llama.cpp. For AMD, I can use rocprof, I tried to see if something similar is available on Nvidia, and there was just some confusing info.

[dfyz]

For AMD, I can use rocprof, I tried to see if something similar is available on Nvidia, and there was just some confusing info

For high-level tracing of what's going on in the system, NSight Systems is the way to go. It will give you a trace very similar to what rocprof provides, with CPU and GPU activity at microsecond-level granularity.

With lots of kernels, looking at the trace can get overwhelming, so you might want to add some markers/ranges to pinpoint the problematic MUL operations.

[slaren]

I found some inefficiencies in the current master, there was a copy that could be made in parallel with the dequantize.

In branch cuda-cache, using pinned memory, an additional stream to copy and dequantize at the same time, and caching the weights, it is currently like this:

Overall this brings perplexity time from 40m to 25m, so it's a nice improvement.

[SlyEcho]

@slaren I tried your branch on HIP, it seems to be running the dequant and right after that the GEMM (Cijk_...) VRAM is also used more.

Speed is only a little better, from 59 minutes (5.49 s/p) to 44 (4.10 s/p)

EDIT: I think there might be something to optimize in the kernel launch arguments, you know with group and local sizes, etc.

[slaren]

I think there might be something to optimize in the kernel launch arguments, you know with group and local sizes, etc.

I tried that when I made the initial implementation of dequantization on the GPU using cudaOccupancyMaxPotentialBlockSize, but it made no difference, but maybe now that everything is faster it is worth giving it another try.

Edit: Tested it again and still couldn't notice a difference.