Skip to content

feat: support internlm2 - #636

Merged
Ying1123 merged 1 commit into
sgl-project:mainfrom
zhyncs:internlm2
Jul 17, 2024
Merged

Ying1123 merged 1 commit into
sgl-project:mainfrom
zhyncs:internlm2

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 17, 2024

Copy link
Copy Markdown
Contributor

cc @Ying1123 @merrymercy @hnyls2002

tested with internlm2-chat-7b

python3 playground/reference_hf.py --model internlm2-chat-7b
python3 -m sglang.bench_latency --model internlm2-chat-7b --correct --output-len 16 --trust-remote-code

transformers

prefill logits tensor([291.2500, 293.7500, 364.5000,  ..., 307.5000, 332.2500, 257.0000],
       device='cuda:0')
prefill logits tensor([314.7500, 320.5000, 398.0000,  ..., 333.0000, 359.5000, 280.0000],
       device='cuda:0')
prefill logits tensor([332.2500, 334.2500, 404.2500,  ..., 351.5000, 374.5000, 293.0000],
       device='cuda:0')

sglang

prefill logits (final) tensor([[288.7500, 291.2500, 361.5000,  ..., 305.0000, 329.5000, 254.8750],
        [313.7500, 319.7500, 396.2500,  ..., 331.7500, 358.7500, 279.0000],
        [331.5000, 333.5000, 403.2500,  ..., 350.7500, 373.7500, 292.5000]],
       device='cuda:0', dtype=torch.float16)

@Ying1123 Ying1123 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On my machine (H100):
SRT:

prefill logits (final) tensor([[262., 262., 342.,  ..., 272., 290., 240.],
        [272., 276., 348.,  ..., 284., 302., 247.],
        [360., 362., 450.,  ..., 378., 392., 326.]], device='cuda:0',
       dtype=torch.bfloat16)

HF:

prefill logits tensor([265.5000, 264.5000, 345.0000,  ..., 274.7500, 293.0000, 242.7500],
       device='cuda:0')
prefill logits tensor([279.2500, 281.7500, 355.2500,  ..., 290.0000, 310.0000, 252.6250],
       device='cuda:0')
prefill logits tensor([359.7500, 361.2500, 448.5000,  ..., 377.0000, 390.5000, 324.5000],
       device='cuda:0')

@Ying1123
Ying1123 merged commit a8552cb into sgl-project:main Jul 17, 2024
@zhyncs
zhyncs deleted the internlm2 branch July 17, 2024 05:42
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
….py (sgl-project#636)

* Create test_ascend_disaggregation_deepep_k8s_2p1d.py

* Create pr-test-npu-innersource.yml

* Add files via upload

* Add files via upload

* Delete test_ascend_tp1_bf16.py

* Delete test/srt/ascend_k8s/test_vlm_models_glm_4_1v_9b_thinking.py

* Create deepep.yaml

* Create run_ascend_ci.py

* Create run_ascend_deepep.sh

* Update run_suite.py

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update run_ascend_deepep.sh

* 111

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update pr-test-npu-innersource.yml

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

* Update run_ascend_ci.py

* Update test_ascend_disaggregation_deepep_k8s_2p1d.py

---------

Co-authored-by: Sugar920 <duzhangchen@h-partners.com>
michaelzhang-ai added a commit to michaelzhang-ai/sglang that referenced this pull request Mar 20, 2026
…ccuracy threshold

PR sgl-project#17784 (transformers 5.3.0 upgrade) changed grok.py to access
config.rope_parameters["rope_theta"] directly, but GitConfig (grok-2)
does not have this attribute, crashing the server on startup with
AttributeError: 'GitConfig' object has no attribute 'rope_parameters'.

Restore safe access via getattr with fallback, matching the pattern
used elsewhere in the codebase.

Also lower the MI325 Grok-2 GSM8K accuracy threshold from 0.915 to
0.90 to match the MI35x test, since nightly sgl-project#636 showed 0.910 which
is within normal run-to-run variance.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…project#636)

The handover onto a token-sharded decode pool is supported on exactly one
combination: page_size == 1, hisparse off, the mooncake transport, on the
uneven-TP replicated-KV layout. All four were already enforced -- and every
one of them only at RUNTIME. Three fire in disaggregation/decode.py
(:1109-1133) on the first send_metadata; the transport one fires from inside
the SENDER (nixl/conn.py:2511, mori/conn.py:1699), later still. So the server
boots healthy, reports 200, accepts traffic, and dies on the FIRST handover,
after the full weight load and in production.

That lateness is the defect, not the number of files: four loud failures are
fine, four loud failures that all arrive after the weight load are not. Every
input is decidable from ServerArgs, so the combination is decided at parse
time, reporting ALL violations at once rather than one reboot at a time.

PLACEMENT -- a correction to the recon, and the load-bearing part of this
commit. The suggested home was inside handle_pd_disaggregation, on the
grounds that all four flags are in scope there. Three are; dcp_size is not.
It is auto-set to tp_size inside _handle_uneven_tp, which __post_init__ calls
at :5957, AFTER _handle_pd_disaggregation at :5916 -- both verified in the
same function body. A gate in the hook would read the unresolved dcp_size,
conclude the token-shard path is not in force, and pass silently on exactly
the configuration it guards. That is worse than no gate, because it reads as
coverage.

The fork has been here before: sgl-project#108's _reject_unsupported_draft_kv_dcp sits
late for the same reason and records a MEASURED false refusal from reading
these inputs early ("rank_tp_ratio=auto-performance, dcp_size=1" rejecting a
correct boot). This gate follows that precedent and runs immediately after it,
which is also after the hook's own mooncake_tcp -> mooncake rewrite, so the
transport check sees the resolved name.

Inert unless it applies: returns early for a monolithic server, and for any
layout that is not dcp_size > 1 == tp_size. The decode-side runtime checks
stay as the backstop for anything arriving by another route.

Tests: test_pd_dcp_token_shard_contract_636.py, 8 hermetic + 6 subtests,
green. One contract test over the combination (each precondition refuses and
names itself; several violations report together; inert off the layout and off
PD), plus a mechanism-reach class that reads __post_init__'s source and pins
the ORDERING -- gate present, after _handle_uneven_tp, after the PD hook.
Two can-fail arms executed: moving the call up next to the PD hook turns
test_gate_runs_after_uneven_tp_resolves_dcp_size red, and dropping the
page_size precondition turns the contract test and its subtest red. The first
is the trap the placement correction exists for.

Base-vs-branch on test/registered/unit/server_args/ + unit/disaggregation/
with the same command: 762 passed, 0 failed on branch, so no regression is
possible. ruff clean on both changed files (server_args.py carries 355
pre-existing ruff findings, identical on base -- verified against
HEAD:server_args.py); codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…ct#642)

The draft KV pool rides the main transfer as extra layers addressed by the
SAME index array as the target pool (prefill.py:186-195, decode.py:439-448,
"The indices are always shared with a target model"), and mooncake/conn.py
contains no occurrence of "draft" at all, so the transport could not treat
them differently even in principle. On a DCP decode arm those indices are
compact owner-rule rows (decode.py:1119-1125). Whether that is the right
address depends entirely on --draft-kv-layout: 'dcp' gives the draft pool
dcp_compact_pool_rows, the target's own coordinate system; 'replicated', the
default, keeps "the full global context per rank"
(model_runner_kv_cache_mixin.py:2819-2823). Under the default, token L's draft
KV is written at row compact(L), which in a full-context pool is where a
different token lives -- sgl-project#345's right-token/wrong-slot class, silent, error
growing with slot id.

Two guards. L11, a parse-time refusal naming the combination, placed with the
sgl-project#636 gate because it also reads the resolved dcp_size. L10, draft_kv_layout
added to DraftKvCanonicalLayout's compared fields, since the two arms must
agree: disagreeing arms differ in both row count and per-row byte length while
sharing one index array.

L10 lives in the layout contract rather than in compute_model_identity_hash,
and the reason is preserved in the code and pinned by a test: that hash is
taken with include_parallel_vectors=False precisely so a PP prefill and a
TP+DCP decode on identical weights still pair (#631a guard 1).
draft_kv_layout is a parallelism decision, so putting it there would
reintroduce exactly the false refusal that flag exists to prevent.

Why the hazard is not reachable today, stated precisely because a latent guard
that becomes dead code is its own defect: #631a REFUSES a PD arm that asks for
speculation, and its SGLANG_PD_AUTO_DISABLE_SPEC escape hatch disables it
instead. Neither branch lets a draft worker exist, so the registration never
runs. The accurate reason is "no branch permits it", not "the refusal nulls
it" -- the branches could diverge, and someone hardening only the escape hatch
would not realise they were also holding a corruption path shut. This gate is
what makes lifting #631a safe, and it is in the tree BEFORE that lift rather
than bundled into it.

Tests: 12 hermetic + 2 subtests, green. HazardTest is the point of the file
and is guard-INDEPENDENT: it models the two code sites' own arithmetic and
demonstrates the corruption executably -- rank 0 of a 3-way split over a
12-token context owns slots 0/3/6/9, and writing them at compact rows into a
full-context pool puts token 3 in token 1's row, token 6 in token 2's, token 9
in token 3's, while rows 6 and 9 stay empty. 3 of 4 owned tokens land at a
wrong address and nothing raises. A guard whose test only checks that the
guard raises proves nothing about the hazard, so the hazard is shown first and
stays green in both can-fail arms.

Can-fail: removing the L11 refusal turns both GuardTest refusal cases red;
dropping draft_kv_layout from the compared tuple turns the L10 disagreement
test red. Base-vs-branch on unit/disaggregation + unit/server_args with the
same command: 785 passed, 0 failed on branch, so no regression is possible.
ruff and codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…contract, path B verdict

Seven commits from the sgl-project#625 Route A line, all desk-proven, none needing cards.

#631a guard 1 (57e13eb): a PD pair whose arms hold different weights is
refused, on model_identity_hash only with include_parallel_vectors=False so a
PP prefill and a TP+DCP decode on the same weights still pair. Surfaced and
fixed a latent TypeError: revision and quantization were the only identity
parts not str()-coerced, harmless while every caller ran late with a resolved
ServerArgs, reachable from the registration path.

sgl-project#636 (4f486ab): the four preconditions of the DCP token-sharded handover
become ONE boot-time contract instead of four runtime deaths on first
transfer. Placed after _reject_unsupported_draft_kv_dcp because dcp_size is
not resolved until _handle_uneven_tp (server_args.py:10319, called at :5957,
after the PD hook at :5916) -- a gate at the hook reads the unresolved value
and passes silently on the exact production configuration it guards. The
ordering is pinned by a test, not by a comment.

#631b contract (214433e) and spec (ec3a440): versioned canonical
full-head draft-KV layout with a geometry gate whose bound is the rejected
alternative's own cost rather than a constant in MB, plus the L1-L11 lift
criterion for the #631a refusal.

#631b composition verdict (66da8b3): --draft-kv-layout and
disaggregation_mode were never considered together. Under the default
'replicated' layout the draft pool is sized to the full global context while
the decode arm addresses every registered buffer with compact owner-rule rows
-- two coordinate systems, silent wrong-slot drift. That makes 'dcp' the only
layout on which the existing shared-index transfer is correct at all, not
merely the cheaper one.

sgl-project#642 (66ca5ee, a79fc3c): guards for that composition, with a hazard
test that stays green when the guard is removed so the arms distinguish a
broken guard from a broken hazard, and an identity-hash invariant pinned by
behaviour after the source-text pin was shown to miss the likely refactor.

The #631a refusal stays in force: the wiring slice needs cards and a
half-wired draft path that boots and silently decodes without speculation is
the failure this line exists to remove.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
… J's root cause

Supersedes the defect-G and acceptance-readiness sections of the original
sgl-project#656 handoff. Records: G fixed and proven on metal; the first
policy-driven flip in the feature's history committing at 01:37:12Z; the
self-contradictory quiescence predicate that was the real unblocker; and
defect J in three parts -- J.1 slot scope (proven, fixed), J.2 row extent
(measured, deliberately not cut, with the live-spec-reserve measurement
owed before it is), J.3 the cutover not carrying the resident decode set
(root cause, and the reason a flip under load is currently impossible).

Names the next build precisely -- resident-request carry across the
stack/topology swap in build_production_flip_cutover and the scheduler
topology snapshot, covering request objects, scheduler bookkeeping and
mamba/GDN state AND locks, not just KV cells -- and flags the standing
architecture context that design must compose with (sgl-project#635/sgl-project#636 PP dcp_size=1
to TP dcp_size=3 handover and its four silent preconditions; sgl-project#212 store
routes truncating GDN state).

Also records the two hypotheses of mine that died on the way, so they are
not re-derived from a symptom they both fit perfectly, and the
SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0 demotion that unmasked J.3.

Documentation only; no behaviour change. Suite unchanged at 379 passed.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
… precondition

P2 -- the uneven-TP replicated-KV layout -- sat in validate_pd_dcp_token_
shard_contract's APPLICABILITY test rather than among its violations, so an
arm that failed it was waved through instead of refused. That is the open leg
of sgl-project#636: a PD decode arm with stock head-sharded DCP passed every boot gate,
loaded its weights, reported 200, accepted traffic and raised
"Stock head-sharded DCP receive is not supported" (decode.py:1153-1159) on the
FIRST handover -- verbatim the lateness this gate exists to remove.

The predicate was also the wrong one. The runtime decides on
uneven_dcp_owner_bounds() (distributed/utils.py:421), non-None iff
uneven_dcp_kv_replicated(dcp_size) == `dcp_size > 1 and
get_tp_partition_ratios() is not None` (utils.py:346-354) -- there is no
equality between dcp_size and tp_size in it. The gate's `dcp == tp` test was
therefore both too narrow (a dcp != tp arm WITH a ratio plan does build
compact rows, and its page_size/hisparse/transport preconditions are live
there, but the gate skipped them) and blind to P2 itself. Mirror the runtime
predicate: dcp_size <= 1 is the only genuinely inert shape, and a missing
--rank-tp-ratio is now a named refusal.

Ordering is unchanged and still correct: the sgl-project#636 gate runs before #631b's,
so this refusal preempts the speculation admission that shares the old
`dcp == tp` spelling.

Also fixes a literal percent in --phase-flip-spill-depth's help text. `% o`
parses as the octal conversion with a space flag, so `--help` raised
"TypeError: %o format: an integer is required, not dict" for EVERY option.

Tests, red first:
- test_head_sharded_dcp_decode_arm_is_refused_at_boot and
  test_the_owner_rule_does_not_require_dcp_equal_tp both fail
  "ValueError not raised" against the pre-fix gate, pass after.
- test_inert_off_the_token_sharded_layout split: dcp_size=1 stays inert,
  the dcp=2/tp=3 case moved to the refusal it actually is.
- test/registered/unit/disaggregation/ + test/registered/unit/server_args/:
  821 passed, 0 failed (2 failed before the help-text fix).
- sgl-project#631 flip family: 1076 passed / 0 failed, inherited baseline unmoved.

Ship config is untouched: it carries no --disaggregation-mode, so the gate
returns at its first line.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…ws name a path this container cannot route to

sgl-project#636's open leg (P2) closed red-first and sgl-project#659 cut 1 landed as the first live
sgl-project#407 consumer. The shift's main finding is a refusal: the brief's preferred
second tier, Rig-2 RAM over the 40G link, does not exist from this process --
the fast NICs are on unroutable subnets, both reachable paths measure 75 MB/s,
and the far box has 8.6 GiB of swap-backed RAM against a 12.9 GB KV region.
Booked as C24 with the measurements, because rig1.json claims 2.83 GB/s
'measured' for that tier and nothing may size against it unre-measured.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants