Skip to content

nvidia_agent_tasks: qwen38_gdn KDA solution - #196

Open
Waterpine wants to merge 1 commit into
BBuf:mainfrom
Waterpine:song-qwen38-gdn
Open

Waterpine wants to merge 1 commit into
BBuf:mainfrom
Waterpine:song-qwen38-gdn

Conversation

@Waterpine

Copy link
Copy Markdown
Contributor

Hi BBuf,

Here is the KDA solution for qwen38_nvfp4__gdn_sigmoid_gating_verify. Could you please take a look and let me know if you have any feedback? Thanks!

@BBuf

BBuf commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Thanks for the contribution. I tested this PR on an NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120), and also tried to adapt the three ideas to the real Qwen3.8-27B NVFP4 + DSpark target-verify path on top of sglang PR #36865 (dc04b615c132036a4aca103142c78a63cc86538b).

Conclusion

I do not think the current solution is production-integrable. The QKVZBA copy is a valid isolated optimization, but its end-to-end gain is below run-to-run noise. The conv and GDN paths fail the speculative-decoding correctness gate. More importantly, the packaged KDA task does not currently describe the production target-verify contracts for conv or GDN, so passing/scoring well on this task is not sufficient evidence for an SGLang integration.

PR-as-submitted test result

I ran the submitted test_solution.py unchanged on the SM120 machine. It did not pass:

qwen38_gdn_gating_update#t9:
  assert_close rtol=0.02 atol=0.02 FAILED (worst elementwise 9.42)

qwen38_qkvzba_split#t9:
  qkvzba_split() missing num_heads_qk/num_heads_v/head_qk/head_v

qwen38_conv1d_update#t9:
  ValueError: unsupported x shape (9, 16384)

The run warned that the installed SGLang GDN/conv sources differ from the copies pinned in the task. Therefore, the exact GDN error magnitude above is environment-specific and should not be treated as a cross-version result. The QKV and conv failures are independent of that drift: bench_harness.py falls back to the baseline RECONSTRUCT for the candidate, while test_solution.py does not, so the two official gates feed different APIs to the same solution.

Production-layout kernel measurements

The submitted files cannot be dropped into production directly because the recorded conv/GDN signatures are incomplete or wrong. I adapted the kernel ideas to the actual SGLang call contracts, then measured them with CUDA events and isolated NCU replays:

Operation CUDA-event baseline Candidate Speedup NCU baseline Candidate Speedup Result
QKVZBA split 39.360 us 31.520 us 1.249x 6.496 us 2.624 us 2.48x bit-exact
causal conv verify 53.760 us 28.352 us 1.896x 13.152 us 5.920 us 2.22x isolated output/state/snapshots bit-exact after changing the arithmetic and API
GDN recurrence 75.568 us 71.104 us 1.063x 26.176 us 13.024 us 2.01x tolerance-only; rejected by E2E

The NCU and CUDA-event absolute times differ because NCU replays only the device kernel, while the event timing includes the normal launch path. The isolated speedups are real, but they do not imply an end-to-end win.

For the QKV kernel, the final NCU launch was 192 blocks x 96 threads, 22 registers/thread, no shared memory, and 2.624 us. Global load/store instruction counts fell from 2,016/2,016 to 580/580. This is the strongest part of the PR.

For conv, I had to replace the submitted packed BF16-FMA accumulation with the baseline arithmetic order (BF16-rounded products followed by ordered FP32 additions), add the actual SiLU path, support the real state/index strides, and write all nine intermediate rollback snapshots. Only after those changes did standalone comparisons become bit-exact.

End-to-end results

Workload for every completed run: Qwen3.8-27B NVFP4 target + DSpark draft, 20 random requests, 8192 input tokens, 1024 output tokens, concurrency 1, seed 1234, temperature 0, cache flushed.

The sglang #36865 baseline was stable across three runs:

Variant Output tok/s Accept length Retokenized output Mean E2E
#36865 baseline run 1 129.65 3.30 20,421 7896.16 ms
#36865 baseline run 2 129.75 3.30 20,421 7890.06 ms
#36865 baseline run 3 129.80 3.30 20,421 7887.20 ms

Ablations:

Variant Output tok/s Accept length Retokenized output Conclusion
PR-style QKV only (one combined output storage) 130.16 3.30 20,421 numerically correct, only +0.32% vs baseline median
QKV with the baseline's four independent output storages 128.74 3.30 20,421 numerically correct, -0.78%; QKV E2E effect is noise-sized
QKV + early non-baseline conv arithmetic 127.81 3.21 19,912 rejected
QKV + GDN candidate about 44 while running 1.00 n/a aborted; speculative acceptance collapsed
QKV + standalone-bit-exact conv 118.86 2.96 19,029 rejected
same exact conv with CUDA PDL wait/trigger + launch attribute 117.53 2.96 19,029 rejected; PDL was required but not the whole issue

The important correctness indicators are accept length and retokenized output, not just generated-token throughput. A variant can appear fast while changing the token stream or destroying speculative acceptance.

The existing Triton conv uses Programmatic Dependent Launch (gdc_wait, gdc_launch_dependents, launch_pdl=True). The submitted CUDA conv uses a normal <<<...>>> launch and has no equivalent synchronization. I added cudaGridDependencySynchronize, cudaTriggerProgrammaticLaunchCompletion, and cudaLaunchAttributeProgrammaticStreamSerialization; the E2E result still remained at 19,029 / 2.96. Therefore the missing PDL semantics are a real integration bug, but not the only missing contract.

Problems in the task definition

1. The conv workload is not the production conv call

The task records/reconstructs:

x          [1, 16384, 9]
conv_state [1, 16384, 4]
weight     [16384, 4]
no activation, no indices, no intermediate snapshots

The actual Qwen3.8 target-verify call I captured is:

x                          [1, 10240, 9], stride (92160, 1, 10240), BF16
conv_state                 [slots, 10240, 3], row stride (3, 1), BF16
weight                     [10240, 4], BF16
activation                 silu
conv_state_indices         one live slot index (production capture used int32)
intermediate_conv_window   [slots, 9, 10240, 3], overlapping/as-strided snapshots
intermediate_state_indices one live snapshot slot index

10240 = Q + K + V; 16384 is the pre-split QKVZ width. The current task is feeding the conv the wrong tensor family. The state stores kernel_width - 1 = 3 history values, not a four-column state with an unused column.

The omitted intermediate windows are semantic outputs used for speculative rollback/tree traversal. Modeling a "fully accepted window" is not equivalent: it may execute similar arithmetic, but it does not validate the state that SGLang later commits after partial acceptance.

Fix: recapture the call after QKVZBA split from the current production backend, including activation, index tensors, real strides, intermediate snapshots, and both chain/tree modes. Remove the shape-guessing RECONSTRUCT once the complete signature is captured.

2. The GDN workload is decode-like, not target-verify semantics

The real target-verify path uses disable_state_update=True and writes every step to intermediate_states_buffer using intermediate_state_indices; retrieve_parent_token may alter the recurrence topology. The task omits all of these arguments. The submitted candidate accepts them through **_unused, ignores them, and writes initial_state_source in place.

There is another performance-contract mismatch: the fast closed-form GDN path is selected only when initial_state_source.shape[0] == 1. A real server state pool has multiple slots, so production falls into the other serial kernel. The task's single-slot compact pool makes the advertised fast path look reachable when it is not.

Fix: capture a realistic multi-slot pool (or an envelope-strided compact view that preserves the production slot pitch), nonzero live indices, disable_state_update, all nine intermediate snapshots, and parent-token topology. Benchmark the branch that production will actually take.

3. In-place state is not compared at all

gates.compare() sees only the returned tensor. Neither baseline entry declares OUTPUT_ARGS, and the task ships no state chain. Thus changes to initial_state_source, conv_state, or intermediate rollback buffers are invisible to the current per-row correctness gate.

The prompt says there is a state-chain check, but the repository currently prints:

this task ships no state chain

The prompt/config also reference docs/capture_gaps.md, but that file is absent from this task.

Fix: declare and compare every mutated destination, and ship captured chains that exercise verify -> accept-N -> rollback/commit for several consecutive steps. For target verify, compare final source-state invariance plus every written intermediate snapshot. Make absence of a declared chain a contract failure for stateful ops.

4. The tolerances are not appropriate for speculative verification

The GDN tolerance is copied from a decode-oriented test, and the conv tolerance is copied from a generic causal-conv test. They do not protect the discrete speculative-acceptance decision. The closed-form GDN candidate intentionally uses approximate FP16 sigmoid/tensor-core products and a truncated Neumann inverse; it can pass a per-call numeric envelope yet change accepted tokens and recursively alter later state. In the production ablation, acceptance fell to 1.00.

Fix:

  • QKVZBA: bit-exact values and output layout/storage contract.
  • Conv: bit-exact output, final state, and all rollback snapshots for this deterministic BF16 fixed shape.
  • GDN: at minimum, strict snapshot/state-chain validation on captured production tensors plus accepted-token identity. If the task permits tolerance-only output, an E2E DSpark acceptance/output-identity gate must be mandatory.

5. QKV changes the alias/storage contract

The baseline allocates four independent outputs (mixed_qkv, z, b, a). The submitted solution allocates one large storage and returns four views into it. Values are bit-exact, but the alias graph is different and the task does not check it. The apparent allocation/graph advantage is therefore partly obtained under a different API contract.

Fix: require shape, stride, storage offset, and non-aliasing to match the baseline, or explicitly change and validate the downstream API before scoring the merged-storage variant.

6. Zero real-tensor coverage permits overfitting to synthetic data

The task explicitly reports 0/11 data args real (0%). It also repairs synthetic inputs (for example, taking abs() of dt_bias) and clamps synthetic indices. That is not enough for a stateful, acceptance-sensitive recurrence. The submitted approximation was tuned to the packaged random envelope, not the model's state trajectory.

Fix: do not accept a production claim until all three rows have real payloads or privacy-safe statistically faithful captures, including state-before/state-after chains and real index/stride envelopes. Include adversarial seeds only as additional coverage, not as a replacement for production tensors.

7. The two official runners enforce different contracts

bench_harness.py applies the baseline RECONSTRUCT to a candidate that has no hook. test_solution.py applies only the candidate's hook, and silently skips missing ops. This explains two of the direct test failures and makes "benchmark passes" and "test passes" non-equivalent statements.

Fix: build one normalized kwargs object with a shared task-level reconstruction, feed clones of it to both arms in both runners, and require the candidate to implement every declared op. Ideally, complete captures should eliminate task-specific reconstruction.

8. Baseline drift should fail closed

The loader emits a warning when installed SGLang differs from the copied source, then continues. For these stateful kernels, that can turn a correctness result into a version-comparison artifact.

Fix: make hash drift a hard failure by default, with an explicit override that marks the report non-authoritative.

9. Three distinct APIs should not share one aggregate score

QKV split, causal conv with rollback side effects, and GDN recurrence have very different correctness contracts. A geometric mean across them can hide one unsafe path behind another fast path.

Fix: split them into separate tasks, or require every op to pass its own hard correctness + production-contract gate before computing any aggregate score.

Recommended next step

Please fix/recapture the task before optimizing this PR further. I would keep the QKV 128-bit copy idea, preserve the four-output storage contract, and treat it as a standalone task. Conv should be regenerated from the real 10240-channel call with exact snapshots and CUDA-graph/PDL semantics. GDN should not use the truncated closed-form approximation unless a captured multi-step rollback chain and E2E accepted-token identity both pass.

Until those task changes land, the current benchmark can rank kernels for a synthetic microproblem, but it cannot certify a Qwen3.8 DSpark target-verify optimization.

@Waterpine

Copy link
Copy Markdown
Contributor Author

Hi BBuf,

Thanks for your comments. I think the main reason this kernel is not yet ready for production integration is that the current test coverage is not sufficiently comprehensive.

Would it be possible to add more test cases under
https://github.com/BBuf/KDA-Pilot/tree/main/nvidia_agent_tasks/qwen38_nvfp4__gdn_sigmoid_gating_verify/tests
to better validate the kernel’s behavior and production readiness?

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants