Skip to content

[Fix] spectrum: reset Spectrum state per CFG branch - #42423

Open
fusheng-ji wants to merge 14 commits into
sgl-project:mainfrom
fusheng-ji:fix/diffusion-spectrum-cfg-reset
Open

fusheng-ji wants to merge 14 commits into
sgl-project:mainfrom
fusheng-ji:fix/diffusion-spectrum-cfg-reset

Conversation

@fusheng-ji

@fusheng-ji fusheng-ji commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #35053.

Spectrum now initializes each independent CFG branch at denoising step 0. Negative-only ranks therefore use the configured skip window and discard forecast history from previous requests.

Under CFG parallel, ranks of the remaining shared-state models also reset their local state. SpectrumParams.get_total_forward_steps counts one local forward per step via a new keyword-only cfg_parallel flag (default False). No current model reaches this path (FLUX now has per-branch state, and MiniMax-H3 does not support CFG parallel); it keeps any shared-state model correct once it enables CFG parallel. Serial shared-state models keep positive-branch reset ownership, and the public full-reset API remains available.

FLUX now gets per-branch Spectrum state, like Wan/Hunyuan/SD3. With serial true CFG, its cond and uncond forwards used to share one counter and one forecaster. As a result:

  • the forecaster mixed cond and uncond features;
  • CFG gating, which stops the uncond forward after the gate step, shifted the cond skip schedule;
  • the end-of-run wrap never fired.

Embedded-guidance FLUX.1-dev runs a single branch and is unaffected. Cache algorithms, thresholds, and defaults are unchanged.

Adapted the branch-local reset from @Oxygen56's unmerged PR #35140, with production-context regression tests using real tensor forecasts. The tests cover separate CFG ranks (Wan and FLUX), both serial branch orders, completed/interrupted requests with changed feature shapes, debug statistics, shared-state models, topology-aware forward counts, and FLUX under CFG gating (cond decisions and outputs match the ungated run).

Supersedes #42157 (branch renamed to follow the repository's branch naming).

Validation

Unit: the new lifecycle tests, run against upstream main, main with both fixes, and both fixes with FLUX moved back to one shared Spectrum state:

Test main main + both fixes both fixes, FLUX shares one Spectrum state
test_cfg_ranks_initialize_the_same_skip_schedule ❌ ✅ ✅
test_flux_cfg_gating_keeps_cond_schedule ❌ ✅ ❌
test_reused_model_matches_fresh_model_after_request_restart ❌ ✅ ✅
test_serial_cfg_keeps_branch_forecasts_independent ❌ ✅ ✅
test_shared_cfg_state_is_not_reset_twice ✅ ✅ ✅
test_total_forward_steps_count_local_forwards ❌ ✅ ✅

test_shared_cfg_state_is_not_reset_twice guards serial shared-state behaviour that main already had, so it passes everywhere.

Together with the other fix on top of main, the Spectrum/TeaCache lifecycle tests, existing Spectrum forecaster, MiniMax-H3 Spectrum, CFG gating/policy, DiT config boundary and sampling-parameter tests give 77 passed and 59 subtests passed. Repository pre-commit hooks pass.

End to end on B200 (Wan2.1-T2V-1.3B-Diffusers, 832x480, seed 42, guidance 3.0, Torch SDPA; compile/BCG/warmup off), comparing upstream main with main + both fixes. FLUX runs use yujiepan/FLUX.1-dev-tiny-random (FLUX.1-dev architecture, random weights) with true CFG, so they compare behaviour, not image quality.

Request isolation: a different-shape request runs first, then the target is compared with a fresh instance.

Model, topology main main + both fixes
Wan, 1 GPU serial CFG bit-identical to fresh bit-identical to fresh
Wan, 2 GPU CFG parallel negative rank fails Spectrum feature shape must remain constant (stale forecaster) → other rank hangs until timeout bit-identical to fresh (frames and latents)
FLUX, 1 GPU serial true CFG same Spectrum decisions as fresh same Spectrum decisions as fresh
FLUX, 2 GPU CFG parallel negative rank fails size of tensor a (256) must match … (400) (stale state) → other rank hangs until timeout same Spectrum decisions as fresh
image-1791056762374

The tiny FLUX checkpoint is not bit-reproducible even without a cache: two fresh instances differ (max abs pixel diff 11), and so do repeated requests on one instance. This is run-to-run non-determinism, not state carried across requests, so FLUX isolation is judged by the absence of crashes and by identical per-step Spectrum decisions.

FLUX with CFG gating: serial true CFG, 20 steps, gate at step 6. With per-branch state the cond decisions match the ungated run step for step; with FLUX on one shared state they do not.
image-1791056806963

Cost (Wan): 33 frames, 30 steps, 3 prompts × 3 requests each. Spectrum skips 63% of DiT forwards in both topologies, with mean PSNR vs no cache 26.6 dB (1 GPU) and 27.2 dB (2 GPUs, CFG parallel), worst prompt 20.6 dB. Median speedup was 2.14× on 1 GPU; on 2 GPUs our shared nodes made timing noisy (median 1.33×, individual requests 2.5–5.1 s vs 4.8 s without a cache). Repeated requests on the same instance were bit-identical.

image-1791056796240 image-1791056801169

The validation above was run locally and on our cluster. GPU CI needs the run-ci label from a maintainer.

Update (2026-10-14)

  • Merged current main. Kept force_spectrum_real on ForwardContext while taking [diffusion] Remove unused helpers and write-only state #43064's removal of the unused attention_backend_cls. The lifecycle test moved with the other diffusion unit tests to python/sglang/multimodal_gen/test/unit/ and, like them, drops per-file CI registration.
  • CFG residual refresh uses real forwards. When CFG gating refreshes its frozen cond-minus-uncond difference, both branches now run real forwards through the normal Spectrum hooks, so the difference never holds a forecast and both branch histories advance once. The forced-real scope is restored if a forward raises. On a small Wan2.1-T2V-1.3B CFG 5 regression (256x256, 17 frames, 20 steps, gate 0.3 plus Spectrum, two prompts), mean frame PSNR of the combined mode against the uncached run rose from 8.7 / 12.3 dB to 13.0 / 16.0 dB.
  • Docs. spectrum.mdx explains how Spectrum combines with CFG gating, measured on Wan2.1-T2V-1.3B at its defaults: the combination is slower than Spectrum alone and no more accurate.
  • The FLUX.1 true-CFG Spectrum schedule and the per-model gating caps are in the follow-up [Fix] spectrum: bound Spectrum under FLUX.1 true CFG and cap CFG gate reuse #44261.

Lifecycle and gating unit tests on the merged branch: 14 passed, 21 subtests passed.

Re-verification (2026-10-15)

On the current head: test_spectrum_lifecycle.py and test_cfg_gating.py 14 passed, 21 subtests passed, 0 skipped; pre-commit run --all-files passes (Rust hooks skipped: no cargo here, no Rust files touched). The Wan CFG-gating numbers in the docs come from a real Wan2.1-T2V-1.3B run at its defaults, not from mocks.

Update (2026-10-15)

Merged main again (73 commits, including the FLUX.3 and MOSS-Image models). One conflict, in forward_context.py: main added a TRT-LLM sequence-metadata cache field to ForwardContext next to the field this PR adds, so both are kept. Lifecycle and gating unit tests on the merged head: 14 passed, 21 subtests passed, 0 skipped. pre-commit run --all-files passes twice (Rust hooks skipped: no cargo here, no Rust files touched).


CI States

Latest PR Test (Base): ⊘ Skipped -- add the run-ci label to run CI tests.
Latest PR Test (Extra): ⊘ Skipped -- add the run-ci label to run CI tests.
Latest PR Test (AMD ROCm 10): ⊘ Skipped -- add the run-ci label to run CI tests.

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Initialize each separate CFG branch at its first denoising step so negative-only ranks use the configured window and discard previous request history. CFG-parallel ranks of shared-state models also reset their local state and count one local forward per denoising step. Preserve the serial shared-state path and the public full reset.

Give FLUX per-branch Spectrum state like Wan, Hunyuan, and SD3. With serial true CFG its cond and uncond forwards shared one counter and one forecaster, so CFG gating (which stops the uncond forward after the gate step) shifted the cond skip schedule and kept the end-of-run wrap from firing, and the forecaster mixed cond and uncond features. Embedded-guidance FLUX.1-dev runs one branch and is unaffected.

Adapted the branch-local reset from Oxygen56 in unmerged PR sgl-project#35140. Add real-tensor lifecycle regressions using the production forward context, including completed and interrupted requests, CFG-parallel shared-state ranks, and FLUX under CFG gating.

Fixes sgl-project#35053
@fusheng-ji
fusheng-ji force-pushed the fix/diffusion-spectrum-cfg-reset branch from 0530adb to 51cdc07 Compare October 4, 2026 00:17
Keep force_spectrum_real on ForwardContext while taking main's removal of the
unused attention_backend_cls state. main moved test/registered/unit/diffusion
into python/sglang/multimodal_gen/test/unit and dropped per-file CI
registration there; the Spectrum lifecycle test follows.
Before the gate both CFG branches run real forwards so the frozen
residual never holds a forecast, which removes Spectrum's skips there.
On Wan2.1-T2V-1.3B at its defaults the combination is slower than
Spectrum alone and no more accurate; add the measured settings.
# Conflicts:
#	python/sglang/multimodal_gen/runtime/managers/forward_context.py
@fusheng-ji fusheng-ji changed the title [diffusion] Reset Spectrum state per CFG branch [Fix] spectrum: reset Spectrum state per CFG branch Oct 16, 2026
@mintlify

mintlify Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit def7335.

Approve preview

@fusheng-ji

Copy link
Copy Markdown
Contributor Author

@mickqian @niehen6174 Could you take a look when you have a moment? This resets Spectrum state per CFG branch (fixes #35053); #44261 builds on it. The branch is up to date with main and Lint passes. Could one of you /tag-and-rerun-ci? Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Diffusion] Spectrum CFG parallel skips negative-branch initialization and can reuse forecaster state across requests

1 participant