Skip to content

[Fix] deterministic: Disable the hybrid SWA memory pool with the radix cache - #44868

Open
aarnetalman wants to merge 3 commits into
sgl-project:mainfrom
aarnetalman:deterministic-disable-hybrid-swa-memory
Open

aarnetalman wants to merge 3 commits into
sgl-project:mainfrom
aarnetalman:deterministic-disable-hybrid-swa-memory

Conversation

@aarnetalman

@aarnetalman aarnetalman commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Deterministic inference promises the same output for a request whatever else the server has served, but on hybrid sliding-window models (for example gpt-oss) the radix cache breaks that. The hybrid SWA memory pool keeps only the sliding window of KV for the sliding-window layers and evicts the rest. A cached prefix can then be reused only as far as its sliding-window KV is still there, so how much of a prompt is served from the cache, and so how it is prefilled, depends on which earlier requests evicted what. The result of the same request changes with the server's history.

With gpt-oss-20b in deterministic mode on MI350X, test_deterministic --test-mode prefix (the same prompts sent repeatedly, sharing prefixes) gave up to 8 different outputs for one prompt. It passes with --disable-radix-cache, and it passes with --disable-hybrid-swa-memory with the radix cache on.

Modifications

  • arg_groups/attention_hook.py (handle_deterministic_inference): in deterministic mode with the radix cache on, gpt-oss and Mellum get disable_hybrid_swa_memory=True with a warning, as deterministic mode already does for --enable-aiter-allreduce-fusion. Their sliding-window layers then keep the full KV like the other layers.
  • Only architectures verified to need it and to run without the pool are listed. Other hybrid SWA models are left unchanged: Gemma 4 (E4B, 26B-A4B, 31B) and Laguna-XS.2 pass the deterministic checks with the pool on, and fail to start without it (Gemma 4 during CUDA graph capture in memory_pool.py; Laguna in Triton decode).
  • test_model_overrides.py: deterministic mode resolves disable_hybrid_swa_memory to True for gpt-oss; it stays False with --disable-radix-cache, without deterministic mode, for a model without sliding-window layers, and for a hybrid SWA model outside the list (Laguna).
  • Unchanged: deterministic mode off; deterministic mode with --disable-radix-cache; every other hybrid SWA model.

Accuracy Tests

MI350X, rocm/sgl-dev:v0.5.21-rocm724-mi35x-20261013, SGLANG_USE_AITER=1, gpt-oss-20b, TP1 and TP2, Triton attention, deterministic inference, radix cache on. The MoE layer runs the deterministic MXFP4 MoE from ROCm/aiter#6583 (without it, the MoE itself is not batch-invariant), with the open fixes #43788 and #34543 applied (deterministic mode on MI350X needs them).

python3 -m sglang.launch_server --model-path openai/gpt-oss-20b --tp {1,2} \
  --attention-backend triton --enable-deterministic-inference

python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50 (4 prompts with prefix lengths 1, 511, 2,048 and 4,097, about 300 samples each), run 3-4 times in a row on one server; numbers are distinct outputs per prompt:

TP1 TP2
before 4 of 8 runs differ: 1 1 1 1, 1 1 3 3, 1 1 3 4, 1 1 1 1, 1 1 1 1, 1 2 4 5, 1 1 1 2, 1 1 1 1 5 of 5 runs differ: 1 2 2 3, 1 4 1 8, 1 2 1 2, 1 4 1 3, 1 3 3 3
before, --disable-radix-cache 1 1 1 1 in 4 of 4 runs 1 1 1 1 in 3 of 3 runs
after (server logs "Disable the hybrid SWA memory pool because deterministic inference is enabled with the radix cache.") 1 1 1 1 in 8 of 8 runs 1 1 1 1 in 4 of 4 runs

After, on the same servers: single 1 output; prefix --return-logprob identical logprobs; radix_cache pass; a bitwise batch-size check (the same prompt in batches of 1 to 50, 1,275 responses; token IDs and logprobs compared exactly with the batch-size-1 response) 1,275 / 1,275, at TP1 and TP2. The same set also passes for gpt-oss-20b at TP4 and TP8 and for gpt-oss-120b at TP1, TP2, TP4 and TP8.

GSM8K (sgl-eval 0.1.2, --temperature 0 --max-tokens 2048, TP1): gpt-oss-20b 94.77% before and after; gpt-oss-120b 95.98% after.

Unit tests, python3 -m pytest test/registered/unit/test_model_overrides.py -k "deterministic or hybrid_swa or gpt_oss or step3p" (same image): before, the new test fails (1 failed, 10 passed); after, 11 passed.

Mellum2-12B-A2.5B, rocm/sgl-dev:v0.5.22-rocm724-mi35x-20261018, Triton attention, deterministic, radix cache on, prefix three times: with the pool 1/1/2/3, 1/1/3/5, 1/1/3/3 distinct outputs; with this change 1/1/1/1 three times, and every other check passes (bitwise 1,275 / 1,275).

Speed Tests and Profiling

No change without deterministic mode, with --disable-radix-cache, or for models without sliding-window layers. For hybrid SWA models in deterministic mode, the sliding-window layers now keep the full KV, so the KV cache holds fewer tokens. gpt-oss-20b TP1, python3 -m sglang.benchmark.serving --backend sglang --dataset-name random --random-range-ratio 1.0 --request-rate inf, 3 rounds each, medians, same session (output tok/s):

1,024 in / 1,024 out, 256 prompts, concurrency 256 8,192 in / 1,024 out, 256 prompts, concurrency 128
deterministic, before (pool on) 5,087 1,726
deterministic, after (pool off) 5,062 (-0.5%) 1,719 (-0.4%)

Checklist


CI States

Latest PR Test (Base): ❌ Run #39956038460
Latest PR Test (Extra): ❌ Run #39956038458
Latest PR Test (AMD ROCm 10): ❌ Run #39956038406

…x cache

With the radix cache on, the hybrid SWA memory pool reuses a cached prefix only
as far as its sliding-window KV has not been evicted, so the reused length, and
with it a request's result, depends on earlier requests. Deterministic mode now
disables the pool, with a warning, for hybrid SWA models when the radix cache
is on. DeepSeek V4 and Inkling, whose memory pools rely on it, are left as
they are.
Turning the hybrid SWA memory pool off is only safe for models that run
without it. On MI350X, Gemma 4 (E4B, 26B-A4B, 31B) crashes during CUDA
graph capture and Laguna-XS.2 fails in Triton decode once the pool is
off, while both pass the deterministic checks with it on. gpt-oss and
Mellum2 differ with the pool on and pass with it off. Deterministic mode
now turns the pool off only for those verified architectures.

@rkarhila-amd rkarhila-amd left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see issues. Ready to open for real code review.

logger = logging.getLogger(__name__)

# Hybrid SWA models verified to need the pool off in deterministic mode, and to run without it. Others are left
# alone: Gemma 4 and Laguna, for example, fail to start without the pool.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to read this comment a few times and compare to the PR description before my brain could parse it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update the comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded it in 564c264: it now says what the list is for, and that a model is listed only when tests show both that the pool breaks its determinism and that the model runs without it. I also reworded the comment at the check below it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants