Repository navigation
feat: support internlm2 - #636
Merged
Merged
Conversation
Ying1123
approved these changes
Jul 17, 2024
Contributor
There was a problem hiding this comment.
On my machine (H100):
SRT:
prefill logits (final) tensor([[262., 262., 342., ..., 272., 290., 240.],
[272., 276., 348., ..., 284., 302., 247.],
[360., 362., 450., ..., 378., 392., 326.]], device='cuda:0',
dtype=torch.bfloat16)
HF:
prefill logits tensor([265.5000, 264.5000, 345.0000, ..., 274.7500, 293.0000, 242.7500],
device='cuda:0')
prefill logits tensor([279.2500, 281.7500, 355.2500, ..., 290.0000, 310.0000, 252.6250],
device='cuda:0')
prefill logits tensor([359.7500, 361.2500, 448.5000, ..., 377.0000, 390.5000, 324.5000],
device='cuda:0')
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
cen121212
pushed a commit
to cen121212/sglang
that referenced
this pull request
Nov 10, 2025
….py (sgl-project#636) * Create test_ascend_disaggregation_deepep_k8s_2p1d.py * Create pr-test-npu-innersource.yml * Add files via upload * Add files via upload * Delete test_ascend_tp1_bf16.py * Delete test/srt/ascend_k8s/test_vlm_models_glm_4_1v_9b_thinking.py * Create deepep.yaml * Create run_ascend_ci.py * Create run_ascend_deepep.sh * Update run_suite.py * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update run_ascend_deepep.sh * 111 * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update pr-test-npu-innersource.yml * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py * Update run_ascend_ci.py * Update test_ascend_disaggregation_deepep_k8s_2p1d.py --------- Co-authored-by: Sugar920 <duzhangchen@h-partners.com>
michaelzhang-ai
added a commit
to michaelzhang-ai/sglang
that referenced
this pull request
Mar 20, 2026
…ccuracy threshold PR sgl-project#17784 (transformers 5.3.0 upgrade) changed grok.py to access config.rope_parameters["rope_theta"] directly, but GitConfig (grok-2) does not have this attribute, crashing the server on startup with AttributeError: 'GitConfig' object has no attribute 'rope_parameters'. Restore safe access via getattr with fallback, matching the pattern used elsewhere in the codebase. Also lower the MI325 Grok-2 GSM8K accuracy threshold from 0.915 to 0.90 to match the MI35x test, since nightly sgl-project#636 showed 0.910 which is within normal run-to-run variance.
Closed
1 of 3 tasks
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 7, 2026
…project#636) The handover onto a token-sharded decode pool is supported on exactly one combination: page_size == 1, hisparse off, the mooncake transport, on the uneven-TP replicated-KV layout. All four were already enforced -- and every one of them only at RUNTIME. Three fire in disaggregation/decode.py (:1109-1133) on the first send_metadata; the transport one fires from inside the SENDER (nixl/conn.py:2511, mori/conn.py:1699), later still. So the server boots healthy, reports 200, accepts traffic, and dies on the FIRST handover, after the full weight load and in production. That lateness is the defect, not the number of files: four loud failures are fine, four loud failures that all arrive after the weight load are not. Every input is decidable from ServerArgs, so the combination is decided at parse time, reporting ALL violations at once rather than one reboot at a time. PLACEMENT -- a correction to the recon, and the load-bearing part of this commit. The suggested home was inside handle_pd_disaggregation, on the grounds that all four flags are in scope there. Three are; dcp_size is not. It is auto-set to tp_size inside _handle_uneven_tp, which __post_init__ calls at :5957, AFTER _handle_pd_disaggregation at :5916 -- both verified in the same function body. A gate in the hook would read the unresolved dcp_size, conclude the token-shard path is not in force, and pass silently on exactly the configuration it guards. That is worse than no gate, because it reads as coverage. The fork has been here before: sgl-project#108's _reject_unsupported_draft_kv_dcp sits late for the same reason and records a MEASURED false refusal from reading these inputs early ("rank_tp_ratio=auto-performance, dcp_size=1" rejecting a correct boot). This gate follows that precedent and runs immediately after it, which is also after the hook's own mooncake_tcp -> mooncake rewrite, so the transport check sees the resolved name. Inert unless it applies: returns early for a monolithic server, and for any layout that is not dcp_size > 1 == tp_size. The decode-side runtime checks stay as the backstop for anything arriving by another route. Tests: test_pd_dcp_token_shard_contract_636.py, 8 hermetic + 6 subtests, green. One contract test over the combination (each precondition refuses and names itself; several violations report together; inert off the layout and off PD), plus a mechanism-reach class that reads __post_init__'s source and pins the ORDERING -- gate present, after _handle_uneven_tp, after the PD hook. Two can-fail arms executed: moving the call up next to the PD hook turns test_gate_runs_after_uneven_tp_resolves_dcp_size red, and dropping the page_size precondition turns the contract test and its subtest red. The first is the trap the placement correction exists for. Base-vs-branch on test/registered/unit/server_args/ + unit/disaggregation/ with the same command: 762 passed, 0 failed on branch, so no regression is possible. ruff clean on both changed files (server_args.py carries 355 pre-existing ruff findings, identical on base -- verified against HEAD:server_args.py); codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 7, 2026
…ct#642) The draft KV pool rides the main transfer as extra layers addressed by the SAME index array as the target pool (prefill.py:186-195, decode.py:439-448, "The indices are always shared with a target model"), and mooncake/conn.py contains no occurrence of "draft" at all, so the transport could not treat them differently even in principle. On a DCP decode arm those indices are compact owner-rule rows (decode.py:1119-1125). Whether that is the right address depends entirely on --draft-kv-layout: 'dcp' gives the draft pool dcp_compact_pool_rows, the target's own coordinate system; 'replicated', the default, keeps "the full global context per rank" (model_runner_kv_cache_mixin.py:2819-2823). Under the default, token L's draft KV is written at row compact(L), which in a full-context pool is where a different token lives -- sgl-project#345's right-token/wrong-slot class, silent, error growing with slot id. Two guards. L11, a parse-time refusal naming the combination, placed with the sgl-project#636 gate because it also reads the resolved dcp_size. L10, draft_kv_layout added to DraftKvCanonicalLayout's compared fields, since the two arms must agree: disagreeing arms differ in both row count and per-row byte length while sharing one index array. L10 lives in the layout contract rather than in compute_model_identity_hash, and the reason is preserved in the code and pinned by a test: that hash is taken with include_parallel_vectors=False precisely so a PP prefill and a TP+DCP decode on identical weights still pair (#631a guard 1). draft_kv_layout is a parallelism decision, so putting it there would reintroduce exactly the false refusal that flag exists to prevent. Why the hazard is not reachable today, stated precisely because a latent guard that becomes dead code is its own defect: #631a REFUSES a PD arm that asks for speculation, and its SGLANG_PD_AUTO_DISABLE_SPEC escape hatch disables it instead. Neither branch lets a draft worker exist, so the registration never runs. The accurate reason is "no branch permits it", not "the refusal nulls it" -- the branches could diverge, and someone hardening only the escape hatch would not realise they were also holding a corruption path shut. This gate is what makes lifting #631a safe, and it is in the tree BEFORE that lift rather than bundled into it. Tests: 12 hermetic + 2 subtests, green. HazardTest is the point of the file and is guard-INDEPENDENT: it models the two code sites' own arithmetic and demonstrates the corruption executably -- rank 0 of a 3-way split over a 12-token context owns slots 0/3/6/9, and writing them at compact rows into a full-context pool puts token 3 in token 1's row, token 6 in token 2's, token 9 in token 3's, while rows 6 and 9 stay empty. 3 of 4 owned tokens land at a wrong address and nothing raises. A guard whose test only checks that the guard raises proves nothing about the hazard, so the hazard is shown first and stays green in both can-fail arms. Can-fail: removing the L11 refusal turns both GuardTest refusal cases red; dropping draft_kv_layout from the compared tuple turns the L10 disagreement test red. Base-vs-branch on unit/disaggregation + unit/server_args with the same command: 785 passed, 0 failed on branch, so no regression is possible. ruff and codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 7, 2026
…contract, path B verdict Seven commits from the sgl-project#625 Route A line, all desk-proven, none needing cards. #631a guard 1 (57e13eb): a PD pair whose arms hold different weights is refused, on model_identity_hash only with include_parallel_vectors=False so a PP prefill and a TP+DCP decode on the same weights still pair. Surfaced and fixed a latent TypeError: revision and quantization were the only identity parts not str()-coerced, harmless while every caller ran late with a resolved ServerArgs, reachable from the registration path. sgl-project#636 (4f486ab): the four preconditions of the DCP token-sharded handover become ONE boot-time contract instead of four runtime deaths on first transfer. Placed after _reject_unsupported_draft_kv_dcp because dcp_size is not resolved until _handle_uneven_tp (server_args.py:10319, called at :5957, after the PD hook at :5916) -- a gate at the hook reads the unresolved value and passes silently on the exact production configuration it guards. The ordering is pinned by a test, not by a comment. #631b contract (214433e) and spec (ec3a440): versioned canonical full-head draft-KV layout with a geometry gate whose bound is the rejected alternative's own cost rather than a constant in MB, plus the L1-L11 lift criterion for the #631a refusal. #631b composition verdict (66da8b3): --draft-kv-layout and disaggregation_mode were never considered together. Under the default 'replicated' layout the draft pool is sized to the full global context while the decode arm addresses every registered buffer with compact owner-rule rows -- two coordinate systems, silent wrong-slot drift. That makes 'dcp' the only layout on which the existing shared-index transfer is correct at all, not merely the cheaper one. sgl-project#642 (66ca5ee, a79fc3c): guards for that composition, with a hazard test that stays green when the guard is removed so the arms distinguish a broken guard from a broken hazard, and an identity-hash invariant pinned by behaviour after the source-text pin was shown to miss the likely refactor. The #631a refusal stays in force: the wiring slice needs cards and a half-wired draft path that boots and silently decodes without speculation is the failure this line exists to remove.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
… J's root cause Supersedes the defect-G and acceptance-readiness sections of the original sgl-project#656 handoff. Records: G fixed and proven on metal; the first policy-driven flip in the feature's history committing at 01:37:12Z; the self-contradictory quiescence predicate that was the real unblocker; and defect J in three parts -- J.1 slot scope (proven, fixed), J.2 row extent (measured, deliberately not cut, with the live-spec-reserve measurement owed before it is), J.3 the cutover not carrying the resident decode set (root cause, and the reason a flip under load is currently impossible). Names the next build precisely -- resident-request carry across the stack/topology swap in build_production_flip_cutover and the scheduler topology snapshot, covering request objects, scheduler bookkeeping and mamba/GDN state AND locks, not just KV cells -- and flags the standing architecture context that design must compose with (sgl-project#635/sgl-project#636 PP dcp_size=1 to TP dcp_size=3 handover and its four silent preconditions; sgl-project#212 store routes truncating GDN state). Also records the two hypotheses of mine that died on the way, so they are not re-derived from a symptom they both fit perfectly, and the SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0 demotion that unmasked J.3. Documentation only; no behaviour change. Suite unchanged at 379 passed.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 12, 2026
… precondition P2 -- the uneven-TP replicated-KV layout -- sat in validate_pd_dcp_token_ shard_contract's APPLICABILITY test rather than among its violations, so an arm that failed it was waved through instead of refused. That is the open leg of sgl-project#636: a PD decode arm with stock head-sharded DCP passed every boot gate, loaded its weights, reported 200, accepted traffic and raised "Stock head-sharded DCP receive is not supported" (decode.py:1153-1159) on the FIRST handover -- verbatim the lateness this gate exists to remove. The predicate was also the wrong one. The runtime decides on uneven_dcp_owner_bounds() (distributed/utils.py:421), non-None iff uneven_dcp_kv_replicated(dcp_size) == `dcp_size > 1 and get_tp_partition_ratios() is not None` (utils.py:346-354) -- there is no equality between dcp_size and tp_size in it. The gate's `dcp == tp` test was therefore both too narrow (a dcp != tp arm WITH a ratio plan does build compact rows, and its page_size/hisparse/transport preconditions are live there, but the gate skipped them) and blind to P2 itself. Mirror the runtime predicate: dcp_size <= 1 is the only genuinely inert shape, and a missing --rank-tp-ratio is now a named refusal. Ordering is unchanged and still correct: the sgl-project#636 gate runs before #631b's, so this refusal preempts the speculation admission that shares the old `dcp == tp` spelling. Also fixes a literal percent in --phase-flip-spill-depth's help text. `% o` parses as the octal conversion with a space flag, so `--help` raised "TypeError: %o format: an integer is required, not dict" for EVERY option. Tests, red first: - test_head_sharded_dcp_decode_arm_is_refused_at_boot and test_the_owner_rule_does_not_require_dcp_equal_tp both fail "ValueError not raised" against the pre-fix gate, pass after. - test_inert_off_the_token_sharded_layout split: dcp_size=1 stays inert, the dcp=2/tp=3 case moved to the refusal it actually is. - test/registered/unit/disaggregation/ + test/registered/unit/server_args/: 821 passed, 0 failed (2 failed before the help-text fix). - sgl-project#631 flip family: 1076 passed / 0 failed, inherited baseline unmoved. Ship config is untouched: it carries no --disaggregation-mode, so the gate returns at its first line.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 12, 2026
…ws name a path this container cannot route to sgl-project#636's open leg (P2) closed red-first and sgl-project#659 cut 1 landed as the first live sgl-project#407 consumer. The shift's main finding is a refusal: the brief's preferred second tier, Rig-2 RAM over the 40G link, does not exist from this process -- the fast NICs are on unroutable subnets, both reachable paths measure 75 MB/s, and the far box has 8.6 GiB of swap-backed RAM against a 12.9 GB KV region. Booked as C24 with the measurements, because rig1.json claims 2.83 GB/s 'measured' for that tier and nothing may size against it unre-measured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc @Ying1123 @merrymercy @hnyls2002
tested with
internlm2-chat-7btransformers
sglang