Skip to content

Fix InternLM2/InterLN3 DynamicNTKScalingRoPE - #1606

Merged
zcbenz merged 2 commits into
mainfrom
fix/internlm-2-3
Aug 18, 2026
Merged

zcbenz merged 2 commits into
mainfrom
fix/internlm-2-3

Conversation

@michalk8

@michalk8 michalk8 commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

Fix DynamicNTKScalingRoPE; I've done parity checks for both InternLM2/InternLM3 in bf16/fp32

Also fixes typo for InternLM2Model -> InternLM3Model.

Per-layer diffs for InternLM3Model (fp32):

Details
stage              max abs    mean abs  Δmax vs prev
embeddings      0.0000e+00  0.0000e+00   +0.0000e+00
layer  0        2.1935e-05  2.1588e-08   +2.1935e-05
layer  1        2.3842e-05  3.7467e-08   +1.9073e-06
layer  2        2.4796e-05  5.3179e-08   +9.5367e-07
layer  3        5.4932e-04  1.3868e-07   +5.2452e-04
layer  4        5.4932e-04  1.6798e-07   +0.0000e+00
layer  5        5.4932e-04  2.0177e-07   +0.0000e+00
layer  6        5.4932e-04  2.3335e-07   +0.0000e+00
layer  7        5.4932e-04  2.6904e-07   +0.0000e+00
layer  8        5.4932e-04  3.1151e-07   +0.0000e+00
layer  9        5.4932e-04  3.5688e-07   +0.0000e+00
layer 10        5.4932e-04  4.1828e-07   +0.0000e+00
layer 11        5.4932e-04  4.6634e-07   +0.0000e+00
layer 12        5.4932e-04  5.4093e-07   +0.0000e+00
layer 13        5.4932e-04  6.1406e-07   +0.0000e+00
layer 14        5.4932e-04  6.8721e-07   +0.0000e+00
layer 15        5.4932e-04  7.8666e-07   +0.0000e+00
layer 16        5.4932e-04  8.5657e-07   +0.0000e+00
layer 17        5.4932e-04  9.2318e-07   +0.0000e+00
layer 18        5.4932e-04  9.9479e-07   +0.0000e+00
layer 19        4.8828e-04  1.1296e-06   -6.1035e-05
layer 20        4.8828e-04  1.2521e-06   +0.0000e+00
layer 21        4.8828e-04  1.3359e-06   +0.0000e+00
layer 22        4.8828e-04  1.4446e-06   +0.0000e+00
layer 23        4.8828e-04  1.5285e-06   +0.0000e+00
layer 24        4.8828e-04  1.6002e-06   +0.0000e+00
layer 25        4.8828e-04  1.6893e-06   +0.0000e+00
layer 26        4.2725e-04  1.7170e-06   -6.1035e-05
layer 27        3.6621e-04  1.7650e-06   -6.1035e-05
layer 28        3.6621e-04  1.7803e-06   +0.0000e+00
layer 29        3.6621e-04  1.8475e-06   +0.0000e+00
layer 30        3.6621e-04  1.9195e-06   +0.0000e+00
layer 31        3.6621e-04  2.0273e-06   +0.0000e+00
layer 32        3.6621e-04  2.1177e-06   +0.0000e+00
layer 33        3.6621e-04  2.3002e-06   +0.0000e+00
layer 34        3.6621e-04  2.4995e-06   +0.0000e+00
layer 35        4.2725e-04  2.7606e-06   +6.1035e-05
layer 36        4.2725e-04  3.0551e-06   +0.0000e+00
layer 37        4.8828e-04  3.4085e-06   +6.1035e-05
layer 38        4.8828e-04  3.7889e-06   +0.0000e+00
layer 39        4.8828e-04  4.1197e-06   +0.0000e+00
layer 40        4.8828e-04  4.4659e-06   +0.0000e+00
layer 41        4.8828e-04  4.8655e-06   +0.0000e+00
layer 42        4.8828e-04  5.2824e-06   +0.0000e+00
layer 43        4.8828e-04  5.8591e-06   +0.0000e+00
layer 44        4.8828e-04  6.4599e-06   +0.0000e+00
layer 45        5.4932e-04  7.1053e-06   +6.1035e-05
layer 46        5.4932e-04  8.0155e-06   +0.0000e+00
final (norm)    3.9101e-05  1.6672e-06   -5.1022e-04

closes #1545 #1553 #1562

@michalk8
michalk8 requested a review from nastya236 July 22, 2026 18:35
@michalk8 michalk8 self-assigned this Jul 22, 2026
@zcbenz
zcbenz merged commit 13425df into main Aug 18, 2026
2 checks passed
@zcbenz
zcbenz deleted the fix/internlm-2-3 branch August 18, 2026 04:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

internlm3: DynamicNTKScalingRoPE scales positions by a hard-coded 2.0 and never consumes rope_scaling.factor

2 participants