Repository navigation
Conversation
3 of 4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow the existing optimized BF16 gate/up GEMM to serve M=256, 1024, 2048 and 4095, in addition to 4096, for K=1536 and N=12288/24576. Other shapes retain their previous dispatch. This changes the capability/dispatch predicate and tests; the kernel body, tile layout and BF16 weights are unchanged.
Depends on #559. The incremental change is two files at
06bd1b41f5d3fc3c35a50fd932fa21faf633f848; incremental diff. This draft targets main because the parent branch is in the fork; its upstream diff includes the parent until #559 lands.Validation on Intel Arc Pro B70:
Companion SGLang PR #44936 queries capability per row count and preserves older-wheel behavior. Its matched serving A/B used the same native binaries in both arms: mean first-token request latency improved 4.25% at M256, 4.69% at M1024, 5.32% at M2048 and 4.84% at M4095. Decode throughput changed +0.18% / -0.07% / +0.95% at request concurrency 1/8/32; treat those small changes as roughly flat.
Serving recipe: BF16 Gemma4 E2B + BF16 assistant, B70 TP1, identical full decode graphs/NEXTN settings, prefix cache disabled, 24 first-token samples per row size; decode blocks 128/640/1536 requests with 128 output tokens each (about 71/71/86 seconds). Single A/B, no statistical significance claim. All 216 first-token outputs match. Longer response/logprob checks are not universally exact; an identical-baseline rerun also differs in all four M512 cases. No new accuracy score or universal numerical parity is claimed. The 525 exact checks concern this operator and their tested inputs.
The target was built and loaded alongside the existing featureful package through an isolated sidecar; this is not full-wheel installation validation. Pre-commit passes. Draft pending parent integration and review.
Evidence archive SHA256:
539e5d8081610493ff860981ec80dce50214116dd4eea522706fecba7667c5eb(raw serving records, source/binary hashes, tests, baseline rerun and retirement records).