Skip to content

[Fix] Keep momentum delta rule chunk masks stable for small mu - #1583

Open
arcusbuilds wants to merge 1 commit into
fla-org:mainfrom
arcusbuilds:fix/momentum-chunk-small-mu
Open

arcusbuilds wants to merge 1 commit into
fla-org:mainfrom
arcusbuilds:fix/momentum-chunk-small-mu

Conversation

@arcusbuilds

@arcusbuilds arcusbuilds commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1578. Chunk-mode MomentumDeltaNet drifted from the recurrent definition once μ ≲ 0.8 (error 0.97 at μ = 0.5, NaN at the μ = e^-2 clamp): the chunk reference built its in-chunk mask by subtracting two cumulative sums, which cancel in fp32. It now builds the mask from within-chunk decay matrices, so every exponent is ≤ 0 and nothing is subtracted. Only the PyTorch reference changes; chunk_momentum_delta_rule runs autograd through it, so training forward and backward are both fixed.

Test plan

  • New test_full_momentum_chunk_ref_decaying_gates (μ = 0.8, the 0.135 clamp, per-token gates; checks o, final S/M and all input grads against recurrent_momentum_delta_rule_ref): fails 3/3 on main, passes here.
  • tests/ops/test_momentum_delta.py: 18/18 pass, plain and with FLA_CI_ENV=1.
  • Other dependent tests from find_dependent_tests.py: test_gdn2.py still running locally; test_dplr_tilelang.py not run locally (tilelang not installed). Neither imports the momentum ops.
  • RTX 2000 Ada Laptop (sm_89), torch 2.11.0+cu128, triton 3.6.0.

Benchmark / NCU

chunk_momentum_delta_rule fwd+bwd, B2 T2048 H4 D64 bf16, chunk 64, same GPU, 3 alternating rounds: main 132.6 / 136.7 / 134.5 ms, this PR 133.1 / 139.9 / 131.0 ms. Neutral. No varlen run (the op doesn't support cu_seqlens).

Breaking changes

None. Outputs change only where the old path was wrong. The β + 1e-6 offset is dropped, so β is used exactly, as in the recurrent reference.

Checklist

  • I have read CONTRIBUTING.md and follow its conventions (code style, docstrings, commit prefixes).
  • I have read AGENTS.md and, where my change matches its scope, the relevant skill under .agents/skills.
  • Dependent tests pass locally or in CI, and new behavior is covered by tests where applicable (tick as N/A for changes with no testable code, e.g. docs-only).
  • Kernel changes include same-hardware before/after benchmark numbers, dense + varlen where applicable (tick as N/A when no kernel code changed).
  • This PR is minor/cosmetic-only (typo, formatting, style-only tweaks) — tick only if it is, and justify below.
  • I understand this PR exceeds 500 changed lines and have explained its single purpose and why it should stay together below.

Signed-off-by: arcusbuilds <212402043+arcusbuilds@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] chunk_momentum_delta_rule diverges from the recurrent reference when log_mu is moderately negative

2 participants