GB10 / SM 121 (DGX Spark) Benchmark Results - TurboQuant slower than f16 on unified memory

[dentity007]

Summary

Ran Madreag's release/cuda-optimized branch (fork of this repo) on an NVIDIA DGX Spark (GB10, SM 12.1, 128GB unified LPDDR5X memory). I believe these are the first SM 121 results for turbo3/turbo4.

Key finding: Both turbo3 and turbo4 are consistently slower than f16 on this hardware, with degradation increasing at deeper context. Up to -23.6% at 32K context for token generation.

This appears to be a unified memory architecture effect - the GB10's 128GB pool eliminates the VRAM pressure that makes KV cache quantization beneficial on discrete GPUs.

Hardware

Spec	Value
GPU	NVIDIA GB10 (Blackwell)
Compute Capability	SM 12.1
Memory	124,610 MiB unified (LPDDR5X, ~273 GB/s)
CUDA	13.0, Driver 580.95.05
OS	Ubuntu 24.04 aarch64 (ARM Grace)

Build

git clone -b release/cuda-optimized https://github.com/Madreag/turbo3-cuda.git
cd turbo3-cuda
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121
cmake --build build -j$(nproc)

Commit: 1766c9133 (build 8793). Compiled clean with no issues on SM 121.

Model

Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf (21.26 GiB, 31.58B params MoE)
Config: -ngl 99 -fa 1 -t 20 -r 3

Results: Token Generation (tg32)

Depth	f16	turbo4	turbo3	turbo4 vs f16	turbo3 vs f16
0	45.21	44.06	43.66	-2.5%	-3.4%
4096	43.29	41.58	41.68	-3.9%	-3.7%
8192	43.37	39.49	40.60	-8.9%	-6.4%
16384	43.29	36.21	36.54	-16.4%	-15.6%
32768	41.61	31.81	32.09	-23.6%	-22.9%

Results: Prompt Processing (pp2048)

Depth	f16	turbo4	turbo3	turbo4 vs f16	turbo3 vs f16
0	809.55	805.17	805.06	-0.5%	-0.6%
4096	794.71	788.86	788.90	-0.7%	-0.7%
8192	780.74	776.05	776.85	-0.6%	-0.5%
16384	763.60	758.19	757.55	-0.7%	-0.8%
32768	718.57	711.26	712.34	-1.0%	-0.9%

Analysis

The pattern is clear: dequantization overhead dominates on unified memory. The GB10 has abundant memory (~125GB available) so there's no bandwidth payoff from smaller KV cache entries. Meanwhile the ~273 GB/s LPDDR5X bandwidth is ~6x lower than GDDR7, making the compute-to-bandwidth ratio unfavorable for quantized approaches.

This is consistent with prior findings showing standard q4_0 KV cache also degrades on GB10 unified memory (92.5% throughput collapse at 64K context, and q4_0 actually using more memory than f16).

Suggestion

Would it be useful to add a "Unified Memory / DGX Spark" note to documentation? These results suggest turbo types don't benefit this architecture class. Happy to contribute a PR with the data.

Appreciate the great work on TurboQuant - the kernels compiled and ran perfectly on SM 121, the performance characteristics just don't transfer to unified memory.

[sz-xuejin]

I also met the performance issue on Jetson T5000. Here is my testing result.

It seems turboquant decrease the performance comparing to original llama.cpp，longer prompt will result in more severe the performance degradation.

[dentity007]

Thanks @sz-xuejin , and apologies for the slow reply.

Jetson T5000 is a second independent data point for the same pattern: unified-memory ARM platforms show performance degradation with symmetric turbo configurations that is not present on discrete-VRAM GPUs. That narrows the hypothesis from "might be GB10-specific" to "unified memory vs discrete VRAM is the dividing line."

A few updates since I filed this issue on April 1:

I cross-validated the GB10 results on two independent RTX 4090 RunPod pods (sm_89, Iowa and Romania). The 4090s agreed within 7%, which ruled out hardware oddity and pointed at the architecture dimension.
In the process, I hit a separate but related dispatch bug on F16-K + TURBO-V that caused silent CPU fallback on both sm_89 and sm_121. Filed as issue #83, fixed in PR #84 (merged April 20).
On the asymmetric KV cache side, q8_0-K + f16-V and f16-K + q8_0-V both work cleanly on unified memory with minimal throughput impact. For your Jetson setup, that pairing might be worth testing before committing to the symmetric turbo path.
Your unified-memory architecture pushback is consistent with the broader picture: TurboQuant's benefit curve leans on the VRAM pressure that unified memory architecturally does not have. The savings equation is different.

If you have wikitext-2 perplexity numbers or a longer context sweep from your T5000 runs, I'd love to see them. Would help pin down whether the degradation curve shape matches between GB10 and T5000, or if the Jetson has its own profile.

[salty-flower]

Another datapoint from the opposite end of the hardware spectrum: consumer Blackwell + small discrete VRAM.

System:

• RTX 5070 Ti Laptop GPU, 12 GB VRAM (sm_120)
• 32 GB RAM
• Linux / NixOS

Model / config:

• Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
• stock llama.cpp vs this turboquant fork
• --flash-attn on
• turboquant run with --cache-type-k turbo3 --cache-type-v turbo3
• for the long-context runs I also disabled prompt cache / ctx checkpoints (--cache-ram 0 --ctx-checkpoints 0)

What I saw:

1. Usable context window

• stock llama.cpp: stable exact-match retrieval up to 52736, first fail at 52992
• turboquant: passes all the way to the model’s full 262144 context window

So on this box turboquant increases the fully usable context window by about 4.97x.

2. Same-context performance comparison (ctx = 52736)

• stock sync tok/s: 0.4822

• turbo sync tok/s: 0.4716

• delta: about -2.2%

• stock sync latency: 22.10s

• turbo sync latency: 22.59s

• delta: about +2.2%

• stock stream TTFT: 21.94s

• turbo stream TTFT: 22.32s

• delta: about +1.7%

• stock GPU process memory: 8748 MiB

• turbo GPU process memory: 7622 MiB

• delta: -1126 MiB (-12.9%)

3. Quality gate

• both variants passed the same exact-match retrieval tasks at the shared benchmark context
• turboquant also passed a fuller binary search with multiple exact retrieval tasks, not just a single passkey prompt, all the way to 262144

So at least on a VRAM-constrained discrete GPU, the tradeoff looks very good: much larger usable context, slightly lower VRAM, and only a small performance penalty at the stock-safe context.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.