Repository navigation
remove redundant pad_input_ids function - #500
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
…sgl-project#500) * add tgl kernel init impl * rename team * remove unuseful file * add prefill calls * refine * fix * fix build
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…inst FEATURE_CATALOG.md Adds docs/dev/AUDIT_500_mechanism_reach.md (239 catalog condition claims checked against the code predicate that gates them: 39 WIDER, 63 NARROWER, 126 EXACT, 9 NOT-FOUND, 20 consolidated bug candidates in §5). Corrects FEATURE_CATALOG.md §1 in place with file:line-cited predicates, resolving real conflicts against the two intervening merges that also touched §1 (sgl-project#492 attention/KV replication axis, sgl-project#496 merge-duty backlog): - Uneven-DCP / Draft-KV-DCP bullet: replaced with the audit's precise predicates (dcp_size > 1 and get_tp_partition_ratios() is not None, distributed/utils.py:346; three entry paths; REPLICATED-KV geometry at utils.py:1048; MLA no-combine at dcp/comm.py:81) and the audit's S1-34 NOT-FOUND finding that demotes "replicated is DEGRADED above TP>kv_heads" from a two-sided code rule to CLI-help prose only, surfacing the sgl-project#500-B3/ sgl-project#108 open defect (--rank-kv-ratio route refused by the draft-KV-DCP gate). The sgl-project#492 "TWO DISTRIBUTION AXES" paragraph is kept unchanged immediately after it (additive, not contradicted by the audit). - Matrix Doctrine (§2) joint-prefill paragraph: replaced the sgl-project#492 correction's enumerator claim with the audit's more precise one (uneven_perf.py:4133-4148, placement.py:813, scheduler.py:5860, :6408, :5218) and folded the sgl-project#492 measured bracket/loose-ctx findings back in as "still the right empirical read on this rig, refined by sgl-project#500-B1" rather than dropping them. Reconciles NOTE_485_joint_phase_vectors.md's sgl-project#492 in-place REFUTED markers with the audit's own correction of the same passages (header blockquote, "attention grid" table row, conclusion item 2): both corrections are kept, in chronological order, with the audit framed as refining sgl-project#492 rather than replacing it — no duplicate REFUTED markers, full correction history intact. Verified: branch's own diff (3b7569f..1d64fe3) touches only the three doc files, no *.py; all 17 FEATURE_CATALOG.md §§ present; the five additive paragraphs from intervening merges (two-axis §1, matrix doctrine §2, sgl-project#491 DSpark §4, merge-duty §12, NVENC §13) all present; codespell clean on the three touched files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…ually binds sgl-project#449 shipped the per-query-row chunking budget at 2048 MiB, above the peak it was meant to bound on this fork's serving geometry (--chunked-prefill-size 256: 903 rows fit the old cap against 256 asked for, so the cap discarded the whole query axis). Lowers the default to 256 MiB (the largest power of two that binds at both SEQ_CHUNK settings), makes the transient nameable at boot (dsv4_indexer_prefill_scratch_mib) and itemized in the pinned-reserve shortfall note. GPU arm packaged under scripts/dev/493_indexer_transient/, not run (falsifier: peak_bytes_max must fall ~326 MiB/rank between arms). FEATURE_CATALOG.md §15/§16 additions merged clean (no overlap with the sgl-project#500 audit merge's §1/§2 edits — adjacent but disjoint hunks). Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test/registered/unit/layers/attention/: 152 passed, 9 skipped, 480 subtests passed (matches pre-merge expectation). - test/registered/unit/server_args/ + migration/cli-metadata/ratchet: 526 passed. test_server_args_mutation_ratchet fails (count 10 > baseline 0), but the offending lines are all in topology.py, dual_group_lane.py, cross_algo_utils.py and uneven_perf.py -- none of which this branch touches, and the same count (10) reproduces on 9fab358 before this merge. Pre-existing, not a regression from sgl-project#493. - ruff check --select=F401,F821,UP037 and codespell clean on all touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…not strand the overhang reclaim try_spill committed the speculative-overhang reclaim (allocator.free + kv_allocated_len + kv_overallocated_freed = True) before three later declines (empty spill plan, sgl-project#236 budget regler, tail wider than one host region). Each decline returns False and leaves the victim RUNNING, so the request's eventual pop_overallocated_kv_cache() asserted and killed the scheduler instead of falling back to stock retraction. Fixed by planning the reclaim where the state used to be written and committing it past the last decline (reclaim_overhang / allocated_after_reclaim in managers/kv_session_offload.py) -- an irreversible mutation now sits behind the LAST decline of its function, not the first. _OWNER_SITES in test_decode_bookkeeping_ownership.py updated in the same merge per the §12 MERGE-DUTY rule: the audited comment now describes the post-sgl-project#501 code path, but the site count is unchanged (still exactly one kv_allocated_len write in try_spill, only later in the function) -- no ratchet impact. FEATURE_CATALOG.md §12 new family entry (reclaim-then-decline, sgl-project#501) merged clean, immediately above the preserved MERGE-DUTY paragraph from the sgl-project#500 audit merge -- adjacent, non-overlapping hunks. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - Ownership test run STANDALONE (AST scan of the whole srt/ tree; avoid batching it with other suites): 2 passed. - kv-session-offload / bookkeeping sweep (test_kv_spill_budget_unit.py, test_kv_spill_class_unit.py, test_kv_spill_destination_unit.py, test_kv_session_offload_unit.py, test_kvso_reclaim_decline_501.py, managers/test_retract_decode_fcfs.py, managers/test_kv_ladder_auto_421.py, spec/test_born_spilled_draft_extend_skip.py): 246 passed (batched) + 2 (ownership, standalone) = 248 total, matching the pre-merge expectation. Same 7 pre-existing files score 242 on 9fab358 (pre-merge base) instead of 246 -- the +4 delta is exactly the new file, diff-identical otherwise. - ruff check --select=F401,F821,UP037 and codespell clean on all touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
Adds docs/dev/AUDIT_506_unswept_axes.md (1231 lines): a targeted sweep of four axes this fork's existing audits (sgl-project#500, sgl-project#492) never covered -- index/dtype overflow (native byte-stride, not Python index tensors), state-changing HTTP endpoints (auth/CORS), persistent-cache key completeness (the sgl-project#241 class generalised), and test honesty (the sgl-project#380 "cannot fail" class). Top PRIO findings: /hibernate takes an unvalidated hibernate_dir override from the request body and accepts GET (A2-F1); the api-key middleware is skipped entirely unless a key is configured AND a route carries ADMIN_FORCE, so the fork's newest state-changing routes are open by default, plus wildcard CORS with credentials defeats the loopback-only bind (A2-F2/F3); the only numeric gate on the native GGUF MXFP4 kernels is loose enough that an all-zeros or sign-flipped output passes (A4-F1); a GGUF MoE MMQ byte-stride computed in 32-bit crosses at 256 experts on DSV4-Flash Q4_K (A1-1). Pure docs, no code diff (git diff --stat d653405..HEAD -- '*.py' empty). Clean merge, single new file, no conflict. codespell clean. Scrubbed for personal email/PAT/private-IP leakage (grep for gmail/googlemail, github_pat/ghp_, 192.168., 10.10.10., the user's private address) -- no hits; the only IPs present are loopback (127.0.0.1), 0.0.0.0 bind examples, and the well-known 169.254.169.254 cloud-metadata address used as an attack-payload example, none of which are a leak.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…tall predicate; refute sgl-project#500-B1 sgl-project#500-B1 claimed the phase-prefill enumerator's kv-head grid was an artefact and that the runtime always ran on a replicated-KV / continuous-token-vector geometry. sgl-project#503 executed the check and REFUTES that claim: two independent predicates govern this family, and sgl-project#500-B1 conflated them. `attn_kv_replicated(tp_size, total_num_kv_heads)` is strictly `kv < tp` (`distributed/utils.py:1081`) and is what `linear.py:1423` / `model_config.py:1332` read to build the q/k/v shards -- so at `kv_heads >= tp` (4 kv heads over 3 ranks on the reference rig) the k/v PROJECTIONS head-shard and the enumerator's kv-head grid is correct. `uneven_dcp_kv_replicated` (`dcp_size > 1 and get_tp_partition_ratios() is not None`, `distributed/utils.py:346`, now consolidated in plan_uneven_dcp_kv_replicated and called from placement.py) replicates the KV POOL, not the projections. Pricing attention weights on the pool predicate models a layout the sgl-project#105 ragged-kernel guard refuses at the first forward. Fix: the planner now reports the token axis only where its own install predicate (`uneven_dcp_kv_replicated`) actually holds -- a bare `--rank-kv-ratio coupled` boot without a non-uniform base plan never reaches `dcp_size > 1`, head-shards the KV cache, and installs no token vector, so the axis is refused by name there instead of being priced as a lever the boot cannot actuate. `planner/placement.py:813` is flagged as an open (narrower) defect: it still uses one flag for both mechanisms and reports k/v projection heads as replicated under uneven DCP where the runtime shards them. Corrects FEATURE_CATALOG.md §1/§2, NOTE_485_joint_phase_vectors.md (three correction sites) and TICKET_485_int8_joint_arm.md to carry this refutation alongside the sgl-project#492/sgl-project#500 correction history already there (append-only, strikethrough + REFUTED markers, nothing deleted). Clean auto-merge -- no conflict: the touched lines in FEATURE_CATALOG.md §1/§2 were untouched by every merge since d653405 (the sgl-project#500-audit merge these lines came from), so the 3-way merge applied sgl-project#503's diff directly. All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2205 passed, 1 skipped, 334 subtests -- above the >=2202 target. - test_attn_replication_axis_492.py alone: 21 passed (up from 18 at the sgl-project#492 merge -- sgl-project#503 added 3 new methods to TestWhatTheRuntimeCanActuallyDoToday). - test/registered/unit/distributed/ (distributed/utils.py is touched): 24 failed, 2131 passed, 8 skipped, 703 subtests. Verified NOT a regression: every one of the 24 failing tests lives in a file sgl-project#503's diff never touches (test_bar1_host_cleanup.py, test_dcp_token_vector_collective.py, test_s12_log_analyse.py, test_uneven_tp_nccl_env.py, test_vmm_utils.py, test_dcp_context_ceiling.py), and the failing code paths themselves (model_runner_kv_cache_mixin.py's uneven_kv_derived_mode() call, the _PoolRowStub fixture, the MPS-warning assertion, LOCAL_RANK env lookups) already existed byte-identical in the pre-merge base d653405 -- confirmed by diff --stat, not just a count match. Root causes are a mix of missing test-fixture attributes, missing LOCAL_RANK env var under hermetic CPU-only invocation, and historical-log numeric drift, none related to the attention/KV geometry this merge touches. - ruff check --select=F401,F821,UP037: 2 findings (placement.py:166 `"PlacementFlags"`, uneven_perf.py:5028 `"Dict[str, List[float]]"`), both confirmed pre-existing at d653405 and outside sgl-project#503's diff hunks. - codespell clean on all 8 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…each with a falsifier B3 (draft-KV-DCP now keys on the installer's own predicate uneven_weighted_dcp_enabled(), admitting the --rank-kv-ratio route the sibling speculation x DCP gate already accepted -- the -67% draft-KV win is no longer unreachable for flag-route boots); B8 (validate_breakable_boot's None arm split into a NO_SERVER_ARGS sentinel that skips vs. an unresolvable backend that refuses by name); B10 (verdict: the kvso x speculation refusal is DELIBERATE, not stale -- reason and the one unobserved round now in the message, KVSO_ALLOW_SPEC surfaced in --enable-kv-session-offload's CLI help for the first time); B2 (--rank-kv-ratio without a placement refuses by name instead of being accepted-and-inert); B18 (the marlin uneven-TP coarsening fold reads a per-backend marlin_packable_linear declaration instead of a class-name list; of the three configs the audit named, only MarlinConfig was a genuine latent gap, W8A8Fp8Config/QuarkConfig reach no marlin repack entry point); planner/flags.py's two inverted registry edges (I-2/I-3) plus a new contract test that drives every declared uneven-TP edge against the live ServerArgs validation instead of trusting the registry by inspection. Adds a status header to AUDIT_500_mechanism_reach.md §5 recording the six fixes and two corrections-from-execution. Manually extended that header with the sgl-project#503 B1-refutation (the sgl-project#503 branch predates B1's discovery entirely, so it couldn't record it): combined board is now B1 REFUTED (sgl-project#503), B2/B3/B8/B10/B18 + the flags.py registry FIXED (sgl-project#504), B4-B7, B9, B11-B17, B19, B20 still OPEN -- one place to read the whole status rather than two disjoint headers. Clean auto-merge, no conflict: FEATURE_CATALOG.md's touched regions (near the preamble, and in §1/§3/§12-adjacent prose) were untouched by every merge since the ac84f5d base; server_args.py was untouched by sgl-project#503/sgl-project#505/sgl-project#506/ sgl-project#509 (its only other change on this line, sgl-project#493, is already inside the base). All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present; the sgl-project#505 §16 corrections (gpu-arb convention-not-enforcement, forward_peak off-by-default, CollectiveClock prefill-only) verified present and unaffected. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_flag_registry_contract_500.py alone: 15 passed (exact target). - test/registered/unit/server_args/: 532 passed, 86 subtests (up from 526 pre-merge -- the new/extended test files account for the delta). - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2223 passed, 1 skipped, 352 subtests -- crossing-check against sgl-project#503 (which also worked the planner edge): both merges coexist without collision, count is above the sgl-project#503-alone baseline of 2205 as expected from sgl-project#504's added tests. - test/registered/unit/layers/moe/: 153 passed, 395 subtests (includes the new test_breakable_boot_gate_500.py). - test/registered/unit/distributed/ (touched via test_marlin_unit_coarsening.py): 24 failed, 2134 passed, 8 skipped, 721 subtests. Diffed the failing-test list against the sgl-project#503-merge state line by line: BYTE-IDENTICAL set of 24 failures, only the passed/subtests counts grew (from marlin coarsening's new cases) -- confirmed non-regression, not just a count match. - ruff check --select=F401,F821,UP037: 2 findings (planner/flags.py:1717 `"Profile"`, test_rank_kv_ratio_args.py's unused `math` import), both confirmed pre-existing at the ac84f5d base and outside sgl-project#504's diff hunks. No ruff format run (CLAUDE.md: do not reformat planner/* and other listed subtrees -- unrelated churn buries the semantic diff). - codespell clean on all 18 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…-wrongness sweep Eight commits, each closing or correcting a sgl-project#505 finding: - B-01: offload_movement splits the wave-in error path -- a destination release that already landed is not a failed retrieval; the item is only marked PARKED and its memtier booking only held when the bytes genuinely are still at the park target. - C-01: step 0 of the sgl-project#505-C-01 liveness-timeout task REFUTES its own ranking -- TTFT is outside the 90s budget (the streaming response is only built after generator.__anext__() returns, so the watchdog does not exist during queueing/prefill). The default is deliberately left unchanged; what sgl-project#514 pins instead is the ORDERING that makes that true (TimeToFirstTokenIsOutsideTheBudgetTest, proven can-fail by removing the pre-pull). Same lesson as sgl-project#500-B1: an audit finding is a hypothesis until its falsifier is red. - C-05: CONVENTION_bounding_defaults.md + a value-pinning test convention, with the first three defaults pinned against removal/halving. - A1-01: widens the draft-load completeness check and makes its non-reach audible instead of silent. - A2-04: PD topology returns UNKNOWN instead of fabricating a CUDA->NVML identity mapping when it cannot resolve one. - D3: scrubs the sgl-project#152 GitHub result-share route (was rendering the start command's argv verbatim) and binds the posted body to what the user actually previewed. - A2-03/A2-01: the kv-session-offload spill verdict is now rank-uniform (a group decision), and a carve-out that used to be assumed is now verified against the live state. - A2-05: the barlink transport fallback is a GROUP decision, not a per-rank one. Also PARTIALLY REFUTES the sgl-project#505-A2-05 damage story: the byte proof already reduces its verdict group-wide, so the three named sites desync into a hang, not a silent split; the genuine split window is narrower (a rank-local exception after a transport's last bring-up collective) and is what this fix actually closes. The broader desync class is untouched, recorded as a separate GPU-window ticket. Corrects AUDIT_505_silent_wrongness.md and FEATURE_CATALOG.md in place with the same append-only CORRECTION/REFUTED convention used throughout this line (blockquotes, nothing deleted). Clean auto-merge, no conflicts in either file -- all 17 catalog sections and every previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#510/sgl-project#513/sgl-project#501 §12 families, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. test_kvso_reclaim_decline_501.py (sgl-project#501's own test) picks up a 1-line change from sgl-project#514's kv_session_offload.py work -- sgl-project#514's base already contains sgl-project#501, so this is sgl-project#514 building on top of it, not a collision. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - Targeted run across disaggregation/, the new/changed test files, managers, distributed (barlink/gpu_battery), and liveness: 530 passed, 21 subtests, 1 failure (test_decode_queue_cleanup.py, untouched by sgl-project#514's diff; reproduced byte-identically on the pre-merge base 96c7dc5 -- same generic retry()-wrapped environmental failure seen throughout this line). - test/registered/unit/distributed/ (full): 24 failed, 2147 passed, 8 skipped, 721 subtests -- matches the established 24-failure baseline. - test/registered/unit/model_loader/ (full, since loader.py/weight_utils.py are touched): 32 failed, 229 passed, 9 skipped, 41 subtests on the merged tree vs 32 failed, 226 passed on the pre-merge base -- diffed the two failing-test-ID lists directly: BYTE-IDENTICAL. The three extra passes are sgl-project#514's own new/extended test coverage elsewhere in the directory. - ruff check --select=F401,F821,UP037: 2 findings (offload_movement.py:206 "DeviceOps", loader.py's unused _export_static_state import), both confirmed pre-existing at 96c7dc5 and outside sgl-project#514's diff hunks. No ruff format run (CLAUDE.md sgl-project#514-lesson rule -- ironically the same task number). - codespell: one hit, "unter" in test_gpu_battery_checks.py:1965 (a pre-existing German test-fixture string, `"reason": "unter der Marke"`, already present at 96c7dc5 and untouched by sgl-project#514's diff -- a pre-existing English-only-code violation, not introduced here).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…-cost probe, the F2 instrument sgl-project#462 (the breakable MoE-offload CUDA-graph route) was already merged onto this line long ago; this pulls in two new commits added on top since, building the sgl-project#494 F2 measurement instrument that ticket named as missing. srt/utils/break_cost_clock.py prices ONE CUDA-graph crossing of a breakable capture (segment_end -> eager slot -> segment_start) with a CUDA event pair around every segment and every break slot in BreakableCUDAGraph.replay(). Per crossing: gap_in_ms / slot_ms / gap_out_ms (device wait, device compute, device wait) plus host_ms split into the four F2 terms (rendezvous, planning, publish, fetch), bracketed via break_cost_phase() in MoEExpertOffloadCache.prepare_breakable; per round: compute_ms / wait_ms / span_ms / residual_ms, per rank, as JSONL. Read via SGLANG_BREAK_COST_DEFER_ROUNDS (default 2) rounds late through query() only, so the read never synchronises the round it measures. SGLANG_BREAK_COST_PROBE=1, OFF by default and byte-neutral off (the disabled replay() loop is the original one statement-for-statement -- no event created, none recorded, nothing allocated per round, pinned by a call-count spy). Route-agnostic: any eager_on_graph break point is priced, not only the MoE one. Reader: scripts/dev/494_break_cost/summarise.py. 21 hermetic tests, three executed can-fail arms. No GPU number exists yet -- the probe has never run on a card; it is the instrument F2 was missing, not a result. FEATURE_CATALOG.md §16 conflict as expected: this branch's own base predates every merge from today (sgl-project#493/sgl-project#505/sgl-project#459/etc.), so its diff carried the OLD short §16 base prose alongside its new "Break-cost probe" paragraph. Kept HEAD's full precise §16 (all of today's corrections) and appended the Break-cost-probe paragraph after it, discarding only the superseded old base text the branch's own diff had inherited. §3's small in-place edit ("its instrument now exists (sgl-project#494, §16 break-cost probe)") landed with no conflict at all -- untouched by anything else on the line. All 17 sections and every other additive paragraph verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree; test tree is tests/moe_offload/, not test/registered/): - test_break_cost_probe_494.py + test_breakable_route_462.py: 58 passed (21 + 37, matching the catalog's own count for each file). - tests/moe_offload/ (full): 2 failed, 455 passed, 34 skipped -- both failures in test_gguf_moe_offload.py, a file this branch's diff never touches. Bisected across today's entire merge history AND the prior session's tips back to 9fab358 (before sgl-project#500's audit branch base): the same 2 failures reproduce identically at every single point, including before this session's work began. Confirmed NOT a new regression from sgl-project#494/sgl-project#462 or anything merged today -- a long-standing pre-existing issue, out of scope for this merge. - ruff check --select=F401,F821,UP037: 18 findings in expert_offload.py and break_cost_clock.py (including two F821 undefined-name hits), ALL confirmed pre-existing at cc5a92e and outside this branch's three diff hunks (import addition + two break_cost_phase() wraps). - codespell: clean on all 6 touched/added files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pad_input_ids in LlavaQwenForCausalLM can be inherited from LlavaLlamaForCausalLM