Repository navigation
[Fix] deterministic: Disable the hybrid SWA memory pool with the radix cache - #44868
Open
aarnetalman wants to merge 3 commits into
Open
aarnetalman wants to merge 3 commits into
aarnetalman wants to merge 3 commits into
Conversation
…x cache With the radix cache on, the hybrid SWA memory pool reuses a cached prefix only as far as its sliding-window KV has not been evicted, so the reused length, and with it a request's result, depends on earlier requests. Deterministic mode now disables the pool, with a warning, for hybrid SWA models when the radix cache is on. DeepSeek V4 and Inkling, whose memory pools rely on it, are left as they are.
Turning the hybrid SWA memory pool off is only safe for models that run without it. On MI350X, Gemma 4 (E4B, 26B-A4B, 31B) crashes during CUDA graph capture and Laguna-XS.2 fails in Triton decode once the pool is off, while both pass the deterministic checks with it on. gpt-oss and Mellum2 differ with the pool on and pass with it off. Deterministic mode now turns the pool off only for those verified architectures.
rkarhila-amd
approved these changes
Oct 23, 2026
rkarhila-amd
left a comment
There was a problem hiding this comment.
I don't see issues. Ready to open for real code review.
| logger = logging.getLogger(__name__) | ||
|
|
||
| # Hybrid SWA models verified to need the pool off in deterministic mode, and to run without it. Others are left | ||
| # alone: Gemma 4 and Laguna, for example, fail to start without the pool. |
There was a problem hiding this comment.
I had to read this comment a few times and compare to the PR description before my brain could parse it.
Contributor
Author
There was a problem hiding this comment.
I'll update the comment
Contributor
Author
There was a problem hiding this comment.
Reworded it in 564c264: it now says what the list is for, and that a model is listed only when tests show both that the pool breaks its determinism and that the model runs without it. I also reworded the comment at the check below it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Deterministic inference promises the same output for a request whatever else the server has served, but on hybrid sliding-window models (for example gpt-oss) the radix cache breaks that. The hybrid SWA memory pool keeps only the sliding window of KV for the sliding-window layers and evicts the rest. A cached prefix can then be reused only as far as its sliding-window KV is still there, so how much of a prompt is served from the cache, and so how it is prefilled, depends on which earlier requests evicted what. The result of the same request changes with the server's history.
With gpt-oss-20b in deterministic mode on MI350X,
test_deterministic --test-mode prefix(the same prompts sent repeatedly, sharing prefixes) gave up to 8 different outputs for one prompt. It passes with--disable-radix-cache, and it passes with--disable-hybrid-swa-memorywith the radix cache on.Modifications
arg_groups/attention_hook.py(handle_deterministic_inference): in deterministic mode with the radix cache on, gpt-oss and Mellum getdisable_hybrid_swa_memory=Truewith a warning, as deterministic mode already does for--enable-aiter-allreduce-fusion. Their sliding-window layers then keep the full KV like the other layers.memory_pool.py; Laguna in Triton decode).test_model_overrides.py: deterministic mode resolvesdisable_hybrid_swa_memorytoTruefor gpt-oss; it staysFalsewith--disable-radix-cache, without deterministic mode, for a model without sliding-window layers, and for a hybrid SWA model outside the list (Laguna).--disable-radix-cache; every other hybrid SWA model.Accuracy Tests
MI350X,
rocm/sgl-dev:v0.5.21-rocm724-mi35x-20261013,SGLANG_USE_AITER=1, gpt-oss-20b, TP1 and TP2, Triton attention, deterministic inference, radix cache on. The MoE layer runs the deterministic MXFP4 MoE from ROCm/aiter#6583 (without it, the MoE itself is not batch-invariant), with the open fixes #43788 and #34543 applied (deterministic mode on MI350X needs them).python3 -m sglang.launch_server --model-path openai/gpt-oss-20b --tp {1,2} \ --attention-backend triton --enable-deterministic-inferencepython3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50(4 prompts with prefix lengths 1, 511, 2,048 and 4,097, about 300 samples each), run 3-4 times in a row on one server; numbers are distinct outputs per prompt:--disable-radix-cacheAfter, on the same servers:
single1 output;prefix --return-logprobidentical logprobs;radix_cachepass; a bitwise batch-size check (the same prompt in batches of 1 to 50, 1,275 responses; token IDs and logprobs compared exactly with the batch-size-1 response) 1,275 / 1,275, at TP1 and TP2. The same set also passes for gpt-oss-20b at TP4 and TP8 and for gpt-oss-120b at TP1, TP2, TP4 and TP8.GSM8K (sgl-eval 0.1.2,
--temperature 0 --max-tokens 2048, TP1): gpt-oss-20b 94.77% before and after; gpt-oss-120b 95.98% after.Unit tests,
python3 -m pytest test/registered/unit/test_model_overrides.py -k "deterministic or hybrid_swa or gpt_oss or step3p"(same image): before, the new test fails (1 failed, 10 passed); after, 11 passed.Mellum2-12B-A2.5B,
rocm/sgl-dev:v0.5.22-rocm724-mi35x-20261018, Triton attention, deterministic, radix cache on,prefixthree times: with the pool 1/1/2/3, 1/1/3/5, 1/1/3/3 distinct outputs; with this change 1/1/1/1 three times, and every other check passes (bitwise 1,275 / 1,275).Speed Tests and Profiling
No change without deterministic mode, with
--disable-radix-cache, or for models without sliding-window layers. For hybrid SWA models in deterministic mode, the sliding-window layers now keep the full KV, so the KV cache holds fewer tokens. gpt-oss-20b TP1,python3 -m sglang.benchmark.serving --backend sglang --dataset-name random --random-range-ratio 1.0 --request-rate inf, 3 rounds each, medians, same session (output tok/s):Checklist
CI States
Latest PR Test (Base): ❌ Run #39956038460
Latest PR Test (Extra): ❌ Run #39956038458
Latest PR Test (AMD ROCm 10): ❌ Run #39956038406