CUDA: add FP32 FlashAttention vector kernel by JohannesGaessler · Pull Request #7188 · ggml-org/llama.cpp

JohannesGaessler · 2024-05-09T22:08:40Z

This PR adds an FP32 FlashAttention kernel that is very similar to the FP16 kernel. It enables using FlashAttention on NVIDIA GPUs without fast FP16 and without tensor cores. It should also provide a speedup on more recent NVIDIA GPUs for batch size 1 and FP32 precision. I have moved the FP16 and FP32 FlashAttention vector kernels to separate files in order to speed up compilation. I also added a function ggml_backend_cuda_get_device_cc to ggml-cuda.h in order to avoid breaking tests/test-backend-ops on NVIDIA GPUs without tensor cores. Unlike with the FP16 kernel there are no weird issues with arrays of size 1 vs. regular variables.

Performance on 1x P40:

model	backend	ngl	n_batch	fa	test	t/s
llama 8B Q4_0	CUDA	99	1	0	pp 4096	40.17 ± 0.00
llama 8B Q4_0	CUDA	99	1	1	pp 4096	52.69 ± 0.00
llama 8B Q4_0	CUDA	99	2	0	pp 4096	42.57 ± 0.00
llama 8B Q4_0	CUDA	99	2	1	pp 4096	97.77 ± 0.00
llama 8B Q4_0	CUDA	99	4	0	pp 4096	71.13 ± 0.00
llama 8B Q4_0	CUDA	99	4	1	pp 4096	117.34 ± 0.00
llama 8B Q4_0	CUDA	99	8	0	pp 4096	83.96 ± 0.00
llama 8B Q4_0	CUDA	99	8	1	pp 4096	143.52 ± 0.00
llama 8B Q4_0	CUDA	99	16	0	pp 4096	110.77 ± 0.00
llama 8B Q4_0	CUDA	99	16	1	pp 4096	125.14 ± 0.00
llama 8B Q4_0	CUDA	99	32	0	pp 4096	210.72 ± 0.00
llama 8B Q4_0	CUDA	99	32	1	pp 4096	209.30 ± 0.00
llama 8B Q4_0	CUDA	99	64	0	pp 4096	387.33 ± 0.00
llama 8B Q4_0	CUDA	99	64	1	pp 4096	315.04 ± 0.00
llama 8B Q4_0	CUDA	99	128	0	pp 4096	532.72 ± 0.00
llama 8B Q4_0	CUDA	99	128	1	pp 4096	357.28 ± 0.00
llama 8B Q4_0	CUDA	99	256	0	pp 4096	664.68 ± 0.00
llama 8B Q4_0	CUDA	99	256	1	pp 4096	374.81 ± 0.00
llama 8B Q4_0	CUDA	99	512	0	pp 4096	748.74 ± 0.00
llama 8B Q4_0	CUDA	99	512	1	pp 4096	375.69 ± 0.00
llama 8B Q4_0	CUDA	99	1024	0	pp 4096	749.28 ± 0.00
llama 8B Q4_0	CUDA	99	1024	1	pp 4096	375.78 ± 0.00
llama 8B Q4_0	CUDA	99	2048	0	pp 4096	749.41 ± 0.00
llama 8B Q4_0	CUDA	99	2048	1	pp 4096	375.89 ± 0.00
llama 8B Q4_0	CUDA	99	4096	0	pp 4096	749.52 ± 0.00
llama 8B Q4_0	CUDA	99	4096	1	pp 4096	375.90 ± 0.00

JohannesGaessler · 2024-05-09T22:12:23Z

Fixes #7055 .

slaren · 2024-05-09T22:27:29Z

This happens regularly, but it's never going to be ok to add backend-specific functions to test-backend-ops. Instead, add the necessary checks to the supports_op function in ggml-cuda.

sorasoras · 2024-05-10T15:21:24Z

  Device 0: AMD Radeon RX 7900 XTX, compute capability 11.0, VMM: no
| model                          |       size |     params | backend    | ngl | sm         |         fa | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------- | ---------: | ---------- | ---------------: |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          1 | pp 512     |   687.77 ± 12.71 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          1 | tg 128     |     34.70 ± 0.30 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          0 | pp 512     |    767.92 ± 1.57 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          0 | tg 128     |     34.36 ± 0.14 |
| qwen2 13B Q5_K - Small         |   9.33 GiB |    14.17 B | ROCm       |  99 | none       |          1 | pp 512     |   1511.97 ± 9.18 |
| qwen2 13B Q5_K - Small         |   9.33 GiB |    14.17 B | ROCm       |  99 | none       |          1 | tg 128     |     57.05 ± 0.02 |
| qwen2 13B Q5_K - Small         |   9.33 GiB |    14.17 B | ROCm       |  99 | none       |          0 | pp 512     |   1773.69 ± 5.63 |
| qwen2 13B Q5_K - Small         |   9.33 GiB |    14.17 B | ROCm       |  99 | none       |          0 | tg 128     |     56.31 ± 0.72 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          1 | pp 1024    |    650.39 ± 8.36 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          1 | pp 2048    |    574.70 ± 3.05 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          1 | pp 4096    |    465.18 ± 3.77 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          1 | tg 128     |     35.06 ± 0.06 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          0 | pp 1024    |   760.63 ± 11.81 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          0 | pp 2048    |    726.50 ± 7.10 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          0 | pp 4096    |    669.02 ± 2.67 |
| qwen2 ?B IQ4_XS - 4.25 bpw     |  16.51 GiB |    32.51 B | ROCm       |  99 | none       |          0 | tg 128     |     33.90 ± 0.30 |
build: de85f908 (2834)

  Device 0: Tesla P40, compute capability 6.1, VMM: yes
| model                          |       size |     params | backend    | ngl | sm         |         fa | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------- | ---------: | ---------- | ---------------: |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          1 | pp 512     |    180.33 ± 0.31 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          1 | tg 128     |     11.35 ± 0.01 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          0 | pp 512     |    201.03 ± 0.31 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          0 | tg 128     |      9.31 ± 0.02 |
| qwen2 13B Q5_K - Small           |   9.33 GiB |    14.17 B | CUDA        |  99 |          1 | pp 512     |    376.66 ± 0.05 |
| qwen2 13B Q5_K - Small           |   9.33 GiB |    14.17 B | CUDA        |  99 |          1 | tg 128     |     22.66 ± 0.03 |
| qwen2 13B Q5_K - Small           |   9.33 GiB |    14.17 B | CUDA        |  99 |          0 | pp 512     |    435.80 ± 0.16 |
| qwen2 13B Q5_K - Small           |   9.33 GiB |    14.17 B | CUDA        |  99 |          0 | tg 128     |     17.88 ± 0.02 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          1 | pp 1024    |    166.17 ± 0.07 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          1 | pp 2048    |    143.93 ± 0.05 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          1 | pp 4096    |    113.58 ± 0.09 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          1 | tg 128     |     11.30 ± 0.00 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          0 | pp 1024    |    196.44 ± 0.25 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          0 | pp 2048    |    189.21 ± 0.23 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          0 | pp 4096    |    177.17 ± 0.21 |
| qwen2 ?B Q4_K - Medium         |  18.34 GiB |    32.51 B | CUDA       |  99 |          0 | tg 128     |      9.31 ± 0.01 |
build: de85f908 (2834)

TG speed up is significant but PP is slower quite a bit, I don't know why.

JohannesGaessler · 2024-05-10T15:26:17Z

There simply isn't yet a kernel optimized for large batch sizes.

slaren · 2024-05-10T16:12:51Z

I don't think it is necessary to check every device here, instead get the context and check only the device for this context. Something like this:

ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; if (ggml_cuda_info().devices[ctx->device].cc < CC_VOLTA) { return false; }

scottmudge · 2024-05-10T23:48:04Z

Just adding a small data point, with KoboldCPP compiled with this, with a Q8_K 11b model on 2 x 1080 Ti (Pascal) setup, I get:

~20.2 T/s avg (proc + gen) with FP32 FA enabled.
~13.4 T/s avg (proc + gen) with FP32 FA disabled.

So a significant improvement in my case. Whereas with FP16 FA, I saw a decrease. So it definitely has utility for a subset of users.

github-actions · 2024-05-11T11:03:09Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 541 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8664.65ms p(95)=20209.22ms fails=, finish reason: stop=481 truncated=60
Prompt processing (pp): avg=106.85tk/s p(95)=491.78tk/s
Token generation (tg): avg=32.65tk/s p(95)=46.81tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=cuda-fa-no-tc-11 commit=aa9cbd76608e8aacf5e02e9568d935e9c4e9fbfe

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1715463597 --> 1715464221
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 558.36, 558.36, 558.36, 558.36, 558.36, 610.59, 610.59, 610.59, 610.59, 610.59, 629.47, 629.47, 629.47, 629.47, 629.47, 666.6, 666.6, 666.6, 666.6, 666.6, 695.05, 695.05, 695.05, 695.05, 695.05, 697.12, 697.12, 697.12, 697.12, 697.12, 701.22, 701.22, 701.22, 701.22, 701.22, 735.1, 735.1, 735.1, 735.1, 735.1, 737.46, 737.46, 737.46, 737.46, 737.46, 752.09, 752.09, 752.09, 752.09, 752.09, 781.28, 781.28, 781.28, 781.28, 781.28, 813.34, 813.34, 813.34, 813.34, 813.34, 837.77, 837.77, 837.77, 837.77, 837.77, 835.99, 835.99, 835.99, 835.99, 835.99, 824.19, 824.19, 824.19, 824.19, 824.19, 834.31, 834.31, 834.31, 834.31, 834.31, 841.64, 841.64, 841.64, 841.64, 841.64, 854.48, 854.48, 854.48, 854.48, 854.48, 852.15, 852.15, 852.15, 852.15, 852.15, 852.24, 852.24, 852.24, 852.24, 852.24, 858.53, 858.53, 858.53, 858.53, 858.53, 859.4, 859.4, 859.4, 859.4, 859.4, 859.81, 859.81, 859.81, 859.81, 859.81, 858.76, 858.76, 858.76, 858.76, 858.76, 860.3, 860.3, 860.3, 860.3, 860.3, 856.9, 856.9, 856.9, 856.9, 856.9, 870.98, 870.98, 870.98, 870.98, 870.98, 866.95, 866.95, 866.95, 866.95, 866.95, 866.21, 866.21, 866.21, 866.21, 866.21, 866.35, 866.35, 866.35, 866.35, 866.35, 871.11, 871.11, 871.11, 871.11, 871.11, 870.16, 870.16, 870.16, 870.16, 870.16, 870.69, 870.69, 870.69, 870.69, 870.69, 871.67, 871.67, 871.67, 871.67, 871.67, 881.92, 881.92, 881.92, 881.92, 881.92, 887.42, 887.42, 887.42, 887.42, 887.42, 849.43, 849.43, 849.43, 849.43, 849.43, 846.9, 846.9, 846.9, 846.9, 846.9, 846.49, 846.49, 846.49, 846.49, 846.49, 851.96, 851.96, 851.96, 851.96, 851.96, 852.52, 852.52, 852.52, 852.52, 852.52, 832.68, 832.68, 832.68, 832.68, 832.68, 818.88, 818.88, 818.88, 818.88, 818.88, 813.54, 813.54, 813.54, 813.54, 813.54, 812.92, 812.92, 812.92, 812.92, 812.92, 810.82, 810.82, 810.82, 810.82, 810.82, 812.09, 812.09, 812.09, 812.09, 812.09, 815.63, 815.63, 815.63, 815.63, 815.63, 814.74, 814.74, 814.74, 814.74, 814.74, 814.73, 814.73, 814.73, 814.73, 814.73, 815.09, 815.09, 815.09, 815.09, 815.09, 817.5, 817.5, 817.5, 817.5, 817.5, 821.17, 821.17, 821.17, 821.17, 821.17, 820.54, 820.54, 820.54, 820.54, 820.54, 818.59, 818.59, 818.59, 818.59, 818.59, 820.36, 820.36, 820.36, 820.36, 820.36, 821.02, 821.02, 821.02, 821.02, 821.02, 821.66, 821.66, 821.66, 821.66, 821.66, 821.42, 821.42, 821.42, 821.42, 821.42, 822.64, 822.64, 822.64, 822.64, 822.64, 823.45, 823.45, 823.45]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1715463597 --> 1715464221
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 45.07, 45.07, 45.07, 45.07, 45.07, 38.87, 38.87, 38.87, 38.87, 38.87, 31.59, 31.59, 31.59, 31.59, 31.59, 30.53, 30.53, 30.53, 30.53, 30.53, 31.39, 31.39, 31.39, 31.39, 31.39, 32.2, 32.2, 32.2, 32.2, 32.2, 33.29, 33.29, 33.29, 33.29, 33.29, 34.13, 34.13, 34.13, 34.13, 34.13, 34.16, 34.16, 34.16, 34.16, 34.16, 33.84, 33.84, 33.84, 33.84, 33.84, 33.94, 33.94, 33.94, 33.94, 33.94, 34.08, 34.08, 34.08, 34.08, 34.08, 33.56, 33.56, 33.56, 33.56, 33.56, 33.28, 33.28, 33.28, 33.28, 33.28, 31.96, 31.96, 31.96, 31.96, 31.96, 32.21, 32.21, 32.21, 32.21, 32.21, 32.56, 32.56, 32.56, 32.56, 32.56, 32.34, 32.34, 32.34, 32.34, 32.34, 31.89, 31.89, 31.89, 31.89, 31.89, 31.61, 31.61, 31.61, 31.61, 31.61, 31.44, 31.44, 31.44, 31.44, 31.44, 31.41, 31.41, 31.41, 31.41, 31.41, 31.56, 31.56, 31.56, 31.56, 31.56, 31.26, 31.26, 31.26, 31.26, 31.26, 31.49, 31.49, 31.49, 31.49, 31.49, 31.63, 31.63, 31.63, 31.63, 31.63, 31.53, 31.53, 31.53, 31.53, 31.53, 30.77, 30.77, 30.77, 30.77, 30.77, 30.74, 30.74, 30.74, 30.74, 30.74, 31.01, 31.01, 31.01, 31.01, 31.01, 31.11, 31.11, 31.11, 31.11, 31.11, 31.19, 31.19, 31.19, 31.19, 31.19, 31.32, 31.32, 31.32, 31.32, 31.32, 31.47, 31.47, 31.47, 31.47, 31.47, 31.31, 31.31, 31.31, 31.31, 31.31, 31.3, 31.3, 31.3, 31.3, 31.3, 31.24, 31.24, 31.24, 31.24, 31.24, 31.16, 31.16, 31.16, 31.16, 31.16, 31.37, 31.37, 31.37, 31.37, 31.37, 31.5, 31.5, 31.5, 31.5, 31.5, 31.62, 31.62, 31.62, 31.62, 31.62, 31.73, 31.73, 31.73, 31.73, 31.73, 31.54, 31.54, 31.54, 31.54, 31.54, 31.23, 31.23, 31.23, 31.23, 31.23, 31.19, 31.19, 31.19, 31.19, 31.19, 29.95, 29.95, 29.95, 29.95, 29.95, 29.96, 29.96, 29.96, 29.96, 29.96, 29.96, 29.96, 29.96, 29.96, 29.96, 30.1, 30.1, 30.1, 30.1, 30.1, 30.19, 30.19, 30.19, 30.19, 30.19, 30.26, 30.26, 30.26, 30.26, 30.26, 30.26, 30.26, 30.26, 30.26, 30.26, 30.21, 30.21, 30.21, 30.21, 30.21, 30.12, 30.12, 30.12, 30.12, 30.12, 30.06, 30.06, 30.06, 30.06, 30.06, 30.07, 30.07, 30.07, 30.07, 30.07, 30.18, 30.18, 30.18, 30.18, 30.18, 30.3, 30.3, 30.3, 30.3, 30.3, 30.42, 30.42, 30.42, 30.42, 30.42, 30.47, 30.47, 30.47, 30.47, 30.47, 30.52, 30.52, 30.52]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1715463597 --> 1715464221
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.18, 0.18, 0.18, 0.18, 0.18, 0.35, 0.35, 0.35, 0.35, 0.35, 0.3, 0.3, 0.3, 0.3, 0.3, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.26, 0.26, 0.26, 0.26, 0.26, 0.17, 0.17, 0.17, 0.17, 0.17, 0.32, 0.32, 0.32, 0.32, 0.32, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.27, 0.27, 0.27, 0.27, 0.27, 0.28, 0.28, 0.28, 0.28, 0.28, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.29, 0.29, 0.29, 0.29, 0.29, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.35, 0.35, 0.35, 0.35, 0.35, 0.26, 0.26, 0.26, 0.26, 0.26, 0.1, 0.1, 0.1, 0.1, 0.1, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.27, 0.27, 0.27, 0.27, 0.27, 0.17, 0.17, 0.17, 0.17, 0.17, 0.08, 0.08, 0.08, 0.08, 0.08, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.35, 0.35, 0.35, 0.35, 0.35, 0.49, 0.49, 0.49, 0.49, 0.49, 0.49, 0.49, 0.49, 0.49, 0.49, 0.53, 0.53, 0.53, 0.53, 0.53, 0.09, 0.09, 0.09, 0.09, 0.09, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.22, 0.22, 0.22, 0.22, 0.22, 0.31, 0.31, 0.31, 0.31, 0.31, 0.19, 0.19, 0.19, 0.19, 0.19, 0.21, 0.21, 0.21, 0.21, 0.21, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.18, 0.18, 0.18]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1715463597 --> 1715464221
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0]

JohannesGaessler · 2024-05-11T13:38:29Z

I don't have any ALiBi models set up for testing but according to tests/test-backend-ops the implementation works correctly.

gilbrotheraway · 2024-11-04T22:12:02Z

hi, i get an error when trying to run with -fa on my p100 is support dropped?

JohannesGaessler · 2024-11-04T22:27:23Z

Pascal is still supported, make an issue.

* CUDA: add FP32 FlashAttention vector kernel * fixup! CUDA: add FP32 FlashAttention vector kernel * fixup! fixup! CUDA: add FP32 FlashAttention vector kernel * fixup! fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

* scripts(pascal): P5200 build notes + bench harness + Vulkan baseline Working notes for getting ht-llama.cpp running on the Quadro P5200 (Pascal sm_61, 16 GB). Toolkit wall: CUDA 13 dropped sm_61, so CUDA backend requires aur/cuda-pascal 12.9.1 + gcc14. Driver 580 still runs sm_61 binaries fine. Vulkan baseline (Llama-3.1-8B Q4_K_M, ngl=99, fa=0, build f6feddb): pp128 269 t/s pp512 278 t/s pp2048 251 t/s tg32 35 t/s tg128 35 t/s CUDA results pending cuda-pascal install (gcc14 source build dominates). Untracked primer (quadro-p5200-llamacpp-primer.md) referenced as the source for the FP16-1/64-FP32, INT8 DP4A, and ggml-org#7188 FA-fix facts. * scripts(pascal): CUDA backend bench results + complete install recipe CUDA 12.9 toolkit built and benched on the Quadro P5200 (Pascal sm_61). Five obstacles climbed on stock Arch: 1) CUDA 13 dropped Pascal → installed 12.9 from runfile --extract 2) Runfile libxml2.so.2 missing → bypassed installer with --extract 3) gcc-15/16 too new for nvcc → gcc-14 from archlinux-archive 4) gcc14 AUR source-build slow → 51MB binary pkg.tar.zst (30s install) 5) glibc 2.43 cospi/sinpi clash → noexcept(true) patch on CUDA math.h+hpp Full recipe in scripts/build-pascal-p5200.md. Bench summary (Llama-2-7B Q4_0, ngl=99, build 5159fee, P5200 sm_61): CUDA fa=1: pp512=795, tg128=45.8 t/s Vulkan: pp512=418, tg128=43.0 t/s CUDA wins pp ~1.9x, tg within 7% (bandwidth-bound). ggml_cuda_init confirms: compute capability 6.1, VMM yes, GGML_CUDA_FORCE_MMQ baked in (visible in nvcc cmdline). CC6.1 + MMQ-only + no cuBLAS fallback = the INT8 dp4a path is what is running. JSON artifacts committed alongside for replay/comparison. * scripts(pascal): packaging recipe — rpath-clean runtime tarball for Omarchy ISO Adds §7 Packaging covering the cmake-install + patchelf + symlink-chain + stage-and-tar pipeline that produces pascal-cuda-artifacts.tar.zst (the runtime fast-path consumed by hai-os-dev's autoinstall). Also drops the stale "TODO — fill in once build-cuda completes" placeholder and moves Sources to the true end of the doc. Recipe reproduces the verified-clean tarball: rpath stripped on all installed targets, libllama/libllama-common copied + patchelfed, symlinks recreated, members rooted at opt/ for `tar -C / -xf` extraction, ld.so.conf.d snippet documented so no LD_LIBRARY_PATH is needed at runtime. * scripts(pascal): correct §7 tarball size + add reference sha256 Was: prose-estimate "~810 MB before zstd, ~470 MB after" — actual is ~816 MB unpacked, 512 MB compressed (110 members). Adds the reference sha256 from the verified crystal build for hai-os-dev to byte-check against. Notes zstd non-determinism so re-runs are expected to differ. * scripts(pascal): field primer + Omarchy autoinstall handoff guide Round out the Pascal/P5200 enablement bundle (PR #99) with the two human-facing companions to scripts/build-pascal-p5200.md: - quadro-p5200-llamacpp-primer.md: Pascal/GP104 + llama.cpp field guide (the two facts that drive every decision, CUDA vs Vulkan, measured 1080-parity numbers, 16 GB VRAM sizing, optimization checklist). - quadro-p5200-omarchy-autoinstall.md: 7-question handoff guide for hai-os-dev — extra packages (no AUR), CUDA-12.9 runfile pin, build flags, the five obstacles + fixes (glibc 2.43 noexcept patch incl.), pre-build at image time, HaiOS integration points, verified 512 MB / sha256 0efed65... reference tarball, measured baseline. Both docs reference the canonical recipe at scripts/build-pascal-p5200.md and the verified tarball cached at crystal:/home/me/pascal-cuda-artifacts.tar.zst. * scripts(pascal): v2 build flags (server+router) + Gemma4 MTP bench JSONs Recipe update: add -DLLAMA_BUILD_SERVER=ON + -DLLAMA_BUILD_TESTS=OFF to the CUDA configure step. Required for the llama-app unified router (bin/llama) to link — without server-on, libllama-server-impl.so is not built and llama-app link fails with `cannot find -lllama-server-impl`. Also required for Gemma4 MTP: ctx_other wiring for the Gemma4Assistant draft class lives only in tools/server/server-context.cpp; the standalone llama-speculative-simple binary segfaults with "Gemma4Assistant requires ctx_other to be set". Rationale block also captures the spec-decode footgun: --spec-type defaults to `none`, so -md <draft> alone is silently ignored. Must pass --spec-type draft-mtp to engage. The /props default_generation_settings.params["speculative.types"] field is per-REQUEST sampler default, NOT the server engine state — the canonical engagement read is server stderr (draft acceptance line + statistics draft-mtp: ... summary). Bench JSONs (crystal Pascal P5200, Gemma4 12B QAT Q4_K_XL, sm_61 CUDA FORCE_MMQ, -fa on, -ngl 99, ctx 4096, greedy temp=0/top_k=1): baseline (no MTP, llama-bench): pp128=465.71 t/s, pp512=456.37 t/s tg32=25.54 t/s, tg128=25.54 t/s (flat — bandwidth-bound) MTP A/B via `llama serve` /completion (degenerate "0"×128 output): A baseline (--spec-type none): 25.26 t/s B MTP (--spec-type draft-mtp): 103.72 t/s ← 4.11× CEILING draft acceptance: 1.00 (118/118) — trivially predictable, not deployment MTP A/B via /v1/chat/completions (non-degenerate, 256 tokens): A baseline: 25.18 t/s B MTP: 76.06 t/s ← 3.02× REPRESENTATIVE greedy speedup draft acceptance: 0.7627 (225/295) bit-identical content sha A vs B (greedy lossless property) All three regimes labeled in the JSON so 4.11× isn't quoted as the deployment number — the representative ~3× greedy or the memory-recorded titan 1.66× (default sampling) are the honest reads. * scripts(pascal): v2 server/MTP docs — §6/§7 scope flip + Gemma4 MTP numbers Follow-on to 3662be4 (v2 build flags). Lands the doc side of the LLAMA_BUILD_SERVER=ON v2 build into the two human-facing companions. omarchy autoinstall guide §6: - v1/v2 tarball table: v2 = pascal-cuda-artifacts-v2-server.tar.zst, sha 2528d952..., 515.5 MB, 121 members, server+router scope. v1 (0efed65..., untouched) stays valid for non-serving bakes; v2 is the additive serving-capable successor, not a recall. - serving footgun: --spec-type defaults to `none` (-md silently ignored); engagement proof is server stderr, not /props. - Gemma4 MTP results, three clearly-labeled regimes (lossless A/B): 4.11x degenerate ceiling / 3.02x representative greedy (headline) / 1.66x sampling deployment ref. build-pascal §7 packaging: - version the tarball filename; never overwrite a live pull source. - v1/v2 size+sha table. - reconcile the stale "router not in this tarball" section to v2 reality: member-delta (+11), single-.so impls, lib64 prune, extraction-validate. - note that bin/llama-server / bin/llama-cli are separate targets, not in llama-app's dep closure (reproducing v2 needs them in --target). Also folds in a one-line build-target fix (line 80: add llama-server + llama-cli to --target) that landed in the shared tree from the fork-manager session concurrently — verified correct, kept so the recipe reproduces v2. * scripts(pascal): #100 bullets 1-3+6 bench evidence — server, router, gpu-only, vision+MTP Closes 4 of 10 issue #100 bullets on Pascal P5200 (v2 server-capable build): - bullet 1 (llama-server): standalone /opt/ht-llama-cuda/bin/llama-server → ready in 4s, /health 200, /completion 40 tok @ 25.84 t/s - bullet 2 (llama-server router): unified `bin/llama serve` shim → ready in 4s, /health 200, /completion 40 tok @ 25.82 t/s - bullet 3 (gpu only works): both runs above use -ngl 99 -fa on - bullet 6 (gemma4 12b qat mtp all modalities): combined mmproj + draft-mtp + spec engine → A. coexistence: /v1/chat with image_url + --spec-type draft-mtp engaged → predicted=96, stderr draft acceptance = 0.66102 (78/118) → B. grounding (decoupled to mtmd-cli, avoids Gemma4 chat-template empty-content quirk): all 3 ground-truth features matched (PASCAL, P5200, red rectangle); requires --jinja (otherwise std::runtime_error custom-template-not-supported abort). Methodology: - regression band ±3% pinned vs committed baseline 25.54 tg / 76.05 MTP; both server-router runs in band (24.77-26.31 t/s window). - engagement read on stderr (draft acceptance / draft-mtp stats), NOT /props (per --spec-type footgun memory). - chat-content quirk explicitly noted in JSON so empty content does not read as fail or regression. Bullets 4 (gpu+cpu offload) + 7-10 (qwen 27B/35B-MoE / gemma 26B/31B) land in subsequent commits once the lithium IQ3-class + titan 31B IQ4_XS transfers complete on crystal. * scripts(pascal): #100 regression rerun + nit fold-ins (slug form, cross-harness note) Regression bench (task #15): re-ran the gemma4-12B-QAT bench from regime-2 on the v2 server-capable build to lock the no-regression gate against the committed baseline (25.54 t/s) and MTP reference (76.05 t/s). - Baseline /v1/chat greedy: mean 24.96 t/s across 3 reps (-2.31% vs 25.54), in band. - MTP /v1/chat greedy: mean 75.07 t/s across 3 reps (-1.29% vs 76.05), in band. - Draft acceptance: 0.76271 — bit-identical to committed regime-2 (225/295 accepted/generated). Strong determinism proof. Fold-in nits on the 2 already-committed JSONs (no dedicated fix commit per crystal-assist's review): - Memory slug citations switched from hyphen-form to underscore-form to match the actual slug names (feedback_spec_type_footgun, reference_mtmd_cli_jinja_required) — resolves to exact-match in tooling. - bench-pascal-server-router-smoke.json: added cross_harness_note clarifying the +1.2% server-endpoint vs llama-bench tg128 agreement STRENGTHENS the no-regression claim (different harnesses, in band). * scripts(pascal): #100 bullets 8, 4, 10 bench evidence + regression nit fold-ins Three model bench JSONs from the v2 server-capable Pascal build: bench-pascal-qwen3.6-27b-iq3-xxs.json (bullet 8): Qwen3.6-27B at UD-IQ3_XXS (11.17 GiB), -ngl 99 -fa on -c 4096 → mode=full-gpu, 65/0/65 layers, gpu_residency_pct=95.45%. /completion 11.27 t/s, /v1/chat 10.44 t/s, gpu free 4 GiB after load. Content reply: "The capital of France is Paris." (qwen3.6 thinking mode active). bench-pascal-gemma4-31b-iq4-xs-offload-{ngl40,ngl48}.json (bullets 4 + 10): ngl=40 phase-1 → ngl=48 phase-2-verify accelerator (crystal-assist's recipe): per_layer_combined = (gpu.model + gpu.context) / layers_gpu at ngl=40 = 309 MiB; ngl_max = 40 + floor((2954 - 400) / 309) = 48. Phase-2 verify PASS at -ngl=48: 49/62 layers GPU, 13/62 layers CPU, 4.95 t/s /completion, 4.98 t/s /v1/chat. Dense-layer partial-offload, host_model=3967 MiB, host_context=768 MiB. Card 96% utilized. gemma-4-31B-IQ4_XS is the smallest 31B quant available anywhere on titan or lithium (sweep done by crystal-assist) — confirms 31B = the documented offload-demo model, closes bullets 4 AND 10 in one bench. Regression rerun JSON: minor wording fix — the bit-identical content sha 01ba4719c80b6fe9 is sha256(b"\n") (single newline), not empty string or null. Banks the harness blind-spot that hashing `jq -r .content` output cannot distinguish JSON-null vs "" vs literal "null" vs "\n". A==B determinism conclusion stands (per crystal-assist review). * scripts(pascal): #100 bullets 7 + 9 bench evidence — qwen 35B MoE + gemma4 26B MoE Closes the last two model bullets: bench-pascal-qwen3.6-35b-a3b-iq3-xxs.json (bullet 7): Qwen3.6-35B-A3B (MoE, 3B active) at UD-IQ3_XXS (12.30 GiB), -ngl 99 -fa on -c 4096 → mode=full-gpu, 41/0/41 layers, gpu_residency_pct=96.85%. /completion 44.45 t/s, /v1/chat 40.96 t/s — fastest of any tested model (3B active keeps per-token compute light). Content reply: "The capital of France is Paris." VRAM 13003/16384 MiB after load (3 GiB headroom). bench-pascal-gemma4-26b-a4b-iq4-xs.json (bullet 9): Gemma4-26B-A4B (MoE, 128 experts / 8 active per token) at UD-IQ4_XS (12.66 GiB), -ngl 99 -fa on -c 4096 → 31/0/31 layers on GPU, /completion 42.10 t/s, /v1/chat 42.36 t/s. Content reply: "### Answer: The capital of France is Paris." VRAM 14345/16384 MiB after load. Classifier note (banked in JSON): the 26B host_model=748 MiB tripped the harness's 600-MiB expert-MoE threshold. Inspection of the gemma4 config (vocab=262144, hidden=5120, IQ4_XS bytes/weight) confirms 748 is the embedding tensor + boundary buffers (≈671 MiB pure embedding), NOT expert offload — all 128 experts are in gpu.model_mib=12952. PRIMARY layer-count signal (31/0/31) correctly reads full-GPU. The 600 MiB threshold was calibrated to 12B embeddings (~540 MiB) and under-scales for larger vocab×hidden_dim products. Mode patched to full-gpu with classifier_note explaining the misfire + suggested remediation (host_model_pct_of_total < 10-15% = embedding-pattern; ≥ that = real expert offload). All 10 issue #100 bullets now have committed bench evidence. --------- Co-authored-by: marksverdhei <marksverd@gmail.com> Co-authored-by: marksverdhei <mark.sverdhei@gmail.com>

* CUDA: add FP32 FlashAttention vector kernel * fixup! CUDA: add FP32 FlashAttention vector kernel * fixup! fixup! CUDA: add FP32 FlashAttention vector kernel * fixup! fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

JohannesGaessler mentioned this pull request May 9, 2024

Flash Attention not working with NVIDIA Quadro P3200 Pascal Architecture GPU #7055

Closed

JohannesGaessler linked an issue May 9, 2024 that may be closed by this pull request

Flash Attention not working with NVIDIA Quadro P3200 Pascal Architecture GPU #7055

Closed

JohannesGaessler added performance Speed related topics Nvidia GPU Issues specific to Nvidia GPUs Review Complexity : High Generally require indepth knowledge of LLMs or GPUs labels May 9, 2024

JohannesGaessler force-pushed the cuda-fa-no-tc-11 branch from 29e01c3 to de85f90 Compare May 10, 2024 07:02

slaren reviewed May 10, 2024

View reviewed changes

JohannesGaessler force-pushed the cuda-fa-no-tc-11 branch from de85f90 to e0d1184 Compare May 10, 2024 16:32

JohannesGaessler mentioned this pull request May 11, 2024

ggml : full ALiBi support #7192

Merged

8 tasks

JohannesGaessler added 2 commits May 11, 2024 14:49

CUDA: add FP32 FlashAttention vector kernel

bbeb952

fixup! CUDA: add FP32 FlashAttention vector kernel

41f5f3a

JohannesGaessler force-pushed the cuda-fa-no-tc-11 branch from e0d1184 to 41f5f3a Compare May 11, 2024 13:15

JohannesGaessler added 2 commits May 11, 2024 15:32

fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

f3c3eaf

fixup! fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

aa9cbd7

slaren approved these changes May 12, 2024

View reviewed changes

JohannesGaessler merged commit dc685be into ggml-org:master May 12, 2024

ss4elby mentioned this pull request May 12, 2024

Flash Attention LostRuins/koboldcpp#844

Closed

AncientMystic mentioned this pull request Jun 9, 2024

Any way to increase performance? And switch to F32? ollama/ollama#4506

Closed

ghts mentioned this pull request Jun 16, 2024

RuntimeError: FlashAttention only supports Ampere GPUs or newer. oobabooga/textgen#5985

Open

1 task

apollo-mg mentioned this pull request Jul 12, 2026

Eval bug: SM_60 Quality Loss, FP32 math silently done in FP16, fix merged in two forks. #25593

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CUDA: add FP32 FlashAttention vector kernel#7188

CUDA: add FP32 FlashAttention vector kernel#7188
JohannesGaessler merged 4 commits into
ggml-org:masterfrom
JohannesGaessler:cuda-fa-no-tc-11

JohannesGaessler commented May 9, 2024

Uh oh!

JohannesGaessler commented May 9, 2024

Uh oh!

slaren commented May 9, 2024

Uh oh!

sorasoras commented May 10, 2024 •

edited

Loading

Uh oh!

JohannesGaessler commented May 10, 2024

Uh oh!

slaren May 10, 2024

Uh oh!

scottmudge commented May 10, 2024

Uh oh!

github-actions Bot commented May 11, 2024 •

edited

Loading

Uh oh!

JohannesGaessler commented May 11, 2024

Uh oh!

gilbrotheraway commented Nov 4, 2024

Uh oh!

JohannesGaessler commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

JohannesGaessler commented May 9, 2024

Uh oh!

JohannesGaessler commented May 9, 2024

Uh oh!

slaren commented May 9, 2024

Uh oh!

sorasoras commented May 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesGaessler commented May 10, 2024

Uh oh!

slaren May 10, 2024

Choose a reason for hiding this comment

Uh oh!

scottmudge commented May 10, 2024

Uh oh!

github-actions Bot commented May 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesGaessler commented May 11, 2024

Uh oh!

gilbrotheraway commented Nov 4, 2024

Uh oh!

JohannesGaessler commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sorasoras commented May 10, 2024 •

edited

Loading

github-actions Bot commented May 11, 2024 •

edited

Loading