Repository navigation
[Fix] spectrum: reset Spectrum state per CFG branch - #42423
Open
fusheng-ji wants to merge 14 commits into
Open
fusheng-ji wants to merge 14 commits into
fusheng-ji wants to merge 14 commits into
Conversation
fusheng-ji
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
kevin-mii,
mickqian,
niehen6174,
ping1jing2 and
yichiche
as code owners
October 3, 2026 19:47
fusheng-ji
force-pushed
the
fix/diffusion-spectrum-cfg-reset
branch
from
October 3, 2026 19:57
434d3a7 to
0530adb
Compare
Initialize each separate CFG branch at its first denoising step so negative-only ranks use the configured window and discard previous request history. CFG-parallel ranks of shared-state models also reset their local state and count one local forward per denoising step. Preserve the serial shared-state path and the public full reset. Give FLUX per-branch Spectrum state like Wan, Hunyuan, and SD3. With serial true CFG its cond and uncond forwards shared one counter and one forecaster, so CFG gating (which stops the uncond forward after the gate step) shifted the cond skip schedule and kept the end-of-run wrap from firing, and the forecaster mixed cond and uncond features. Embedded-guidance FLUX.1-dev runs one branch and is unaffected. Adapted the branch-local reset from Oxygen56 in unmerged PR sgl-project#35140. Add real-tensor lifecycle regressions using the production forward context, including completed and interrupted requests, CFG-parallel shared-state ranks, and FLUX under CFG gating. Fixes sgl-project#35053
fusheng-ji
force-pushed
the
fix/diffusion-spectrum-cfg-reset
branch
from
October 4, 2026 00:17
0530adb to
51cdc07
Compare
Keep force_spectrum_real on ForwardContext while taking main's removal of the unused attention_backend_cls state. main moved test/registered/unit/diffusion into python/sglang/multimodal_gen/test/unit and dropped per-file CI registration there; the Spectrum lifecycle test follows.
Before the gate both CFG branches run real forwards so the frozen residual never holds a forecast, which removes Spectrum's skips there. On Wan2.1-T2V-1.3B at its defaults the combination is slower than Spectrum alone and no more accurate; add the measured settings.
fusheng-ji
requested review from
JustinTong0323,
sogalin,
wisclmy0611 and
zijiexia
as code owners
October 14, 2026 09:30
5 tasks done
# Conflicts: # python/sglang/multimodal_gen/runtime/managers/forward_context.py
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
Keep force_spectrum_real alongside main's quality forward context.
Contributor
Author
|
@mickqian @niehen6174 Could you take a look when you have a moment? This resets Spectrum state per CFG branch (fixes #35053); #44261 builds on it. The branch is up to date with main and Lint passes. Could one of you |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #35053.
Spectrum now initializes each independent CFG branch at denoising step 0. Negative-only ranks therefore use the configured skip window and discard forecast history from previous requests.
Under CFG parallel, ranks of the remaining shared-state models also reset their local state.
SpectrumParams.get_total_forward_stepscounts one local forward per step via a new keyword-onlycfg_parallelflag (defaultFalse). No current model reaches this path (FLUX now has per-branch state, and MiniMax-H3 does not support CFG parallel); it keeps any shared-state model correct once it enables CFG parallel. Serial shared-state models keep positive-branch reset ownership, and the public full-reset API remains available.FLUX now gets per-branch Spectrum state, like Wan/Hunyuan/SD3. With serial true CFG, its cond and uncond forwards used to share one counter and one forecaster. As a result:
Embedded-guidance FLUX.1-dev runs a single branch and is unaffected. Cache algorithms, thresholds, and defaults are unchanged.
Adapted the branch-local reset from @Oxygen56's unmerged PR #35140, with production-context regression tests using real tensor forecasts. The tests cover separate CFG ranks (Wan and FLUX), both serial branch orders, completed/interrupted requests with changed feature shapes, debug statistics, shared-state models, topology-aware forward counts, and FLUX under CFG gating (cond decisions and outputs match the ungated run).
Supersedes #42157 (branch renamed to follow the repository's branch naming).
Validation
Unit: the new lifecycle tests, run against upstream
main,mainwith both fixes, and both fixes with FLUX moved back to one shared Spectrum state:mainmain+ both fixestest_cfg_ranks_initialize_the_same_skip_scheduletest_flux_cfg_gating_keeps_cond_scheduletest_reused_model_matches_fresh_model_after_request_restarttest_serial_cfg_keeps_branch_forecasts_independenttest_shared_cfg_state_is_not_reset_twicetest_total_forward_steps_count_local_forwardstest_shared_cfg_state_is_not_reset_twiceguards serial shared-state behaviour thatmainalready had, so it passes everywhere.Together with the other fix on top of
main, the Spectrum/TeaCache lifecycle tests, existing Spectrum forecaster, MiniMax-H3 Spectrum, CFG gating/policy, DiT config boundary and sampling-parameter tests give 77 passed and 59 subtests passed. Repository pre-commit hooks pass.End to end on B200 (Wan2.1-T2V-1.3B-Diffusers, 832x480, seed 42, guidance 3.0, Torch SDPA; compile/BCG/warmup off), comparing upstream
mainwithmain+ both fixes. FLUX runs useyujiepan/FLUX.1-dev-tiny-random(FLUX.1-dev architecture, random weights) with true CFG, so they compare behaviour, not image quality.Request isolation: a different-shape request runs first, then the target is compared with a fresh instance.
mainmain+ both fixesSpectrum feature shape must remain constant(stale forecaster) → other rank hangs until timeoutsize of tensor a (256) must match … (400)(stale state) → other rank hangs until timeoutThe tiny FLUX checkpoint is not bit-reproducible even without a cache: two fresh instances differ (max abs pixel diff 11), and so do repeated requests on one instance. This is run-to-run non-determinism, not state carried across requests, so FLUX isolation is judged by the absence of crashes and by identical per-step Spectrum decisions.
FLUX with CFG gating: serial true CFG, 20 steps, gate at step 6. With per-branch state the cond decisions match the ungated run step for step; with FLUX on one shared state they do not.

Cost (Wan): 33 frames, 30 steps, 3 prompts × 3 requests each. Spectrum skips 63% of DiT forwards in both topologies, with mean PSNR vs no cache 26.6 dB (1 GPU) and 27.2 dB (2 GPUs, CFG parallel), worst prompt 20.6 dB. Median speedup was 2.14× on 1 GPU; on 2 GPUs our shared nodes made timing noisy (median 1.33×, individual requests 2.5–5.1 s vs 4.8 s without a cache). Repeated requests on the same instance were bit-identical.
The validation above was run locally and on our cluster. GPU CI needs the
run-cilabel from a maintainer.Update (2026-10-14)
main. Keptforce_spectrum_realonForwardContextwhile taking [diffusion] Remove unused helpers and write-only state #43064's removal of the unusedattention_backend_cls. The lifecycle test moved with the other diffusion unit tests topython/sglang/multimodal_gen/test/unit/and, like them, drops per-file CI registration.spectrum.mdxexplains how Spectrum combines with CFG gating, measured on Wan2.1-T2V-1.3B at its defaults: the combination is slower than Spectrum alone and no more accurate.Lifecycle and gating unit tests on the merged branch: 14 passed, 21 subtests passed.
Re-verification (2026-10-15)
On the current head:
test_spectrum_lifecycle.pyandtest_cfg_gating.py14 passed, 21 subtests passed, 0 skipped;pre-commit run --all-filespasses (Rust hooks skipped: nocargohere, no Rust files touched). The Wan CFG-gating numbers in the docs come from a real Wan2.1-T2V-1.3B run at its defaults, not from mocks.Update (2026-10-15)
Merged
mainagain (73 commits, including the FLUX.3 and MOSS-Image models). One conflict, inforward_context.py:mainadded a TRT-LLM sequence-metadata cache field toForwardContextnext to the field this PR adds, so both are kept. Lifecycle and gating unit tests on the merged head: 14 passed, 21 subtests passed, 0 skipped.pre-commit run --all-filespasses twice (Rust hooks skipped: nocargohere, no Rust files touched).CI States
Latest PR Test (Base): ⊘ Skipped -- add the
run-cilabel to run CI tests.Latest PR Test (Extra): ⊘ Skipped -- add the
run-cilabel to run CI tests.Latest PR Test (AMD ROCm 10): ⊘ Skipped -- add the
run-cilabel to run CI tests.Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: