CUDA: re-capture the graph when a CPY node's read-source address changes

[joelfarthing]

I came across this while experimenting with moving openPangu attention off its bespoke path onto the shared DSA gather (#2109) and indexer-mask (#2119) machinery. The fix is general to any CPY whose read source moves under CUDA graphs and stands on its own.

In ggml_graph_node_has_matching_properties, a GGML_OP_CPY is exempt from re-capture when its source address changes, for both operands. That is meant for KV-cache writes: their destination (src[1]) advances each token but is repaired at replay through the cpy_dest_ptrs indirection table. src[0] is the read source and has no such indirection, so a CPY whose source moves while its subgraph stays shape-stable never re-captures and replays a stale pointer. The fix keeps src[1] exempt and compares src[0] like any other source:

-            node->op != GGML_OP_CPY &&
-            node->op != GGML_OP_VIEW
+            node->op != GGML_OP_VIEW &&
+            !(node->op == GGML_OP_CPY && i == 1)

Where it applies: openPangu-2.0-Flash, q8_0 latent cache, -fa off, greedy. Past ~600 tokens the sliding-window latent-K read (a Q8_0-to-F32 cast whose q8_0 view steps to a new block) replays the previous block's bytes, so attention reads stale K and output collapses into a repetition loop; with the fix it tracks the window and stays coherent.

The narrowing is safe: every other CPY reads from a stable address, and a KV write keeps src[0] at a stable compute-buffer address, so this adds no re-captures for them. On Qwen2.5-0.5B (no moving-source CPY) the graph counters are identical patched vs unpatched (33 captures, 0 disabled; balanced 3-rep A/B) and decode throughput matches within noise.

Correctness, greedy --temp 0 -n 1000, unpatched main vs PR patched by swapping only libggml.so on the posted base: unpatched runs to the token cap in a 23x-repeated loop, patched stays coherent and stops on its own. The two arms emit token-identical code until they diverge. At the window step, the stale K read makes attention re-emit the previous line instead of advancing.

Current main collapses:

    # Test the in-place version
    print("Sorted array (basic):", sorted_array)

    # Test the in-place version
    print("Sorted array (basic):", sorted_array)

    # Test the in-place version
    print("Sorted array (basic):", sorted_array)

    # (repeats to the -n cap)

With this PR, generation continues:

    # Test the in-place version
    test_array2 = [64, 34, 25, 12, 22, 11, 90]
    quicksort_inplace(test_array2)
    print("Sorted array (in-place):", test_array2)

Throughput, warm A-B-A, openPangu Q4_K_M, -ngl 999 -ot exps=CPU -ub 2048 -fa off, swapping only libggml.so. Decode (S_TG) is the affected path, patched vs unpatched:

N_KV	This PR	main
0	15.62	15.64
2048	9.20	9.15
4096	10.36	10.36
6144	12.01	12.05

Post-onset decode delta averages +0.06%, prefill +0.02%, both within noise. Instrumented counters over a 915-token decode: 1413 captures, peak 3 consecutive updates, 0 graphs disabled, so graphs stay engaged. (The fallback disables a graph at 4 consecutive updates; the q8 window steps periodically and stays under it, so correctness holds and the added re-captures cost nothing measurable.)

Reproducer: openPangu-2.0-Flash Q4_K_M, RTX 4070, CUDA 13.3, Release. Prompt "Write a quicksort implementation in python", greedy:

llama-cli -m openPangu-2.0-Flash-Q4_K_M.gguf -ngl 999 -ot exps=CPU -fa off \
  -ctk q8_0 -ictk q8_0 -c 8192 -n 1000 --temp 0 -s 0 -f prompt.txt -sp

Separately, I did spot a real problem worth its own fix: the Q8_0-to-Q8_0 transpose used by MLA mla=2 q8 caches is admitted into CUDA graphs but writes its captured destination directly and ignores the indirection table, so a moving destination there would replay stale. I'm calling it out of scope for this PR; openPangu never hits it (it dequants each q8 boundary to F32 first, tripwire-confirmed), and I would rather keep this change to two lines and address that path on its own.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ikawrakow]

I need to be away for a few days and don't like the idea of merging this PR just before that. Will merge when I'm back.