Repository navigation
Revert "Make the server random by default" - #492
Merged
Merged
Conversation
This reverts commit 80a33ce.
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
* new workflow-oss monitor * update name --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…-vector residual The axis was not merely unpriced before sgl-project#485 -- NOTE_475 SS4 'Residual honesty' already attributed 14 of the 41 ms/1k the sgl-project#424 INT8 pair moved to --rank-kv-ratio redistributing DCP token ownership 'and therefore prefill attention work', naming it a skew source the family model does not represent. Recorded as the axis's outstanding measured anchor, explicitly NOT claimed as a result: the CORE-PACED endpoint redistributes mass rather than growing it, so it cannot reproduce a residual of that size, and asserting the match would be the fitted constant the bracket exists to avoid.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…he joint prefill solve
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…n annotations (UP037) _prefill_sharded_time and prefill_time_model's core/base_core parameters quoted AttnCorePlan, which is defined earlier in the module (line 3611) and needs no forward-reference string -- ruff --select=UP037 flagged all three as new. The pre-existing UP037 hit at per_family_prefill_compute_times' return type (predates this merge) is left as documented pre-existing rot.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…inst FEATURE_CATALOG.md Adds docs/dev/AUDIT_500_mechanism_reach.md (239 catalog condition claims checked against the code predicate that gates them: 39 WIDER, 63 NARROWER, 126 EXACT, 9 NOT-FOUND, 20 consolidated bug candidates in §5). Corrects FEATURE_CATALOG.md §1 in place with file:line-cited predicates, resolving real conflicts against the two intervening merges that also touched §1 (sgl-project#492 attention/KV replication axis, sgl-project#496 merge-duty backlog): - Uneven-DCP / Draft-KV-DCP bullet: replaced with the audit's precise predicates (dcp_size > 1 and get_tp_partition_ratios() is not None, distributed/utils.py:346; three entry paths; REPLICATED-KV geometry at utils.py:1048; MLA no-combine at dcp/comm.py:81) and the audit's S1-34 NOT-FOUND finding that demotes "replicated is DEGRADED above TP>kv_heads" from a two-sided code rule to CLI-help prose only, surfacing the sgl-project#500-B3/ sgl-project#108 open defect (--rank-kv-ratio route refused by the draft-KV-DCP gate). The sgl-project#492 "TWO DISTRIBUTION AXES" paragraph is kept unchanged immediately after it (additive, not contradicted by the audit). - Matrix Doctrine (§2) joint-prefill paragraph: replaced the sgl-project#492 correction's enumerator claim with the audit's more precise one (uneven_perf.py:4133-4148, placement.py:813, scheduler.py:5860, :6408, :5218) and folded the sgl-project#492 measured bracket/loose-ctx findings back in as "still the right empirical read on this rig, refined by sgl-project#500-B1" rather than dropping them. Reconciles NOTE_485_joint_phase_vectors.md's sgl-project#492 in-place REFUTED markers with the audit's own correction of the same passages (header blockquote, "attention grid" table row, conclusion item 2): both corrections are kept, in chronological order, with the audit framed as refining sgl-project#492 rather than replacing it — no duplicate REFUTED markers, full correction history intact. Verified: branch's own diff (3b7569f..1d64fe3) touches only the three doc files, no *.py; all 17 FEATURE_CATALOG.md §§ present; the five additive paragraphs from intervening merges (two-axis §1, matrix doctrine §2, sgl-project#491 DSpark §4, merge-duty §12, NVENC §13) all present; codespell clean on the three touched files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
Adds docs/dev/AUDIT_506_unswept_axes.md (1231 lines): a targeted sweep of four axes this fork's existing audits (sgl-project#500, sgl-project#492) never covered -- index/dtype overflow (native byte-stride, not Python index tensors), state-changing HTTP endpoints (auth/CORS), persistent-cache key completeness (the sgl-project#241 class generalised), and test honesty (the sgl-project#380 "cannot fail" class). Top PRIO findings: /hibernate takes an unvalidated hibernate_dir override from the request body and accepts GET (A2-F1); the api-key middleware is skipped entirely unless a key is configured AND a route carries ADMIN_FORCE, so the fork's newest state-changing routes are open by default, plus wildcard CORS with credentials defeats the loopback-only bind (A2-F2/F3); the only numeric gate on the native GGUF MXFP4 kernels is loose enough that an all-zeros or sign-flipped output passes (A4-F1); a GGUF MoE MMQ byte-stride computed in 32-bit crosses at 256 experts on DSV4-Flash Q4_K (A1-1). Pure docs, no code diff (git diff --stat d653405..HEAD -- '*.py' empty). Clean merge, single new file, no conflict. codespell clean. Scrubbed for personal email/PAT/private-IP leakage (grep for gmail/googlemail, github_pat/ghp_, 192.168., 10.10.10., the user's private address) -- no hits; the only IPs present are loopback (127.0.0.1), 0.0.0.0 bind examples, and the well-known 169.254.169.254 cloud-metadata address used as an attack-payload example, none of which are a leak.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…tall predicate; refute sgl-project#500-B1 sgl-project#500-B1 claimed the phase-prefill enumerator's kv-head grid was an artefact and that the runtime always ran on a replicated-KV / continuous-token-vector geometry. sgl-project#503 executed the check and REFUTES that claim: two independent predicates govern this family, and sgl-project#500-B1 conflated them. `attn_kv_replicated(tp_size, total_num_kv_heads)` is strictly `kv < tp` (`distributed/utils.py:1081`) and is what `linear.py:1423` / `model_config.py:1332` read to build the q/k/v shards -- so at `kv_heads >= tp` (4 kv heads over 3 ranks on the reference rig) the k/v PROJECTIONS head-shard and the enumerator's kv-head grid is correct. `uneven_dcp_kv_replicated` (`dcp_size > 1 and get_tp_partition_ratios() is not None`, `distributed/utils.py:346`, now consolidated in plan_uneven_dcp_kv_replicated and called from placement.py) replicates the KV POOL, not the projections. Pricing attention weights on the pool predicate models a layout the sgl-project#105 ragged-kernel guard refuses at the first forward. Fix: the planner now reports the token axis only where its own install predicate (`uneven_dcp_kv_replicated`) actually holds -- a bare `--rank-kv-ratio coupled` boot without a non-uniform base plan never reaches `dcp_size > 1`, head-shards the KV cache, and installs no token vector, so the axis is refused by name there instead of being priced as a lever the boot cannot actuate. `planner/placement.py:813` is flagged as an open (narrower) defect: it still uses one flag for both mechanisms and reports k/v projection heads as replicated under uneven DCP where the runtime shards them. Corrects FEATURE_CATALOG.md §1/§2, NOTE_485_joint_phase_vectors.md (three correction sites) and TICKET_485_int8_joint_arm.md to carry this refutation alongside the sgl-project#492/sgl-project#500 correction history already there (append-only, strikethrough + REFUTED markers, nothing deleted). Clean auto-merge -- no conflict: the touched lines in FEATURE_CATALOG.md §1/§2 were untouched by every merge since d653405 (the sgl-project#500-audit merge these lines came from), so the 3-way merge applied sgl-project#503's diff directly. All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2205 passed, 1 skipped, 334 subtests -- above the >=2202 target. - test_attn_replication_axis_492.py alone: 21 passed (up from 18 at the sgl-project#492 merge -- sgl-project#503 added 3 new methods to TestWhatTheRuntimeCanActuallyDoToday). - test/registered/unit/distributed/ (distributed/utils.py is touched): 24 failed, 2131 passed, 8 skipped, 703 subtests. Verified NOT a regression: every one of the 24 failing tests lives in a file sgl-project#503's diff never touches (test_bar1_host_cleanup.py, test_dcp_token_vector_collective.py, test_s12_log_analyse.py, test_uneven_tp_nccl_env.py, test_vmm_utils.py, test_dcp_context_ceiling.py), and the failing code paths themselves (model_runner_kv_cache_mixin.py's uneven_kv_derived_mode() call, the _PoolRowStub fixture, the MPS-warning assertion, LOCAL_RANK env lookups) already existed byte-identical in the pre-merge base d653405 -- confirmed by diff --stat, not just a count match. Root causes are a mix of missing test-fixture attributes, missing LOCAL_RANK env var under hermetic CPU-only invocation, and historical-log numeric drift, none related to the attention/KV geometry this merge touches. - ruff check --select=F401,F821,UP037: 2 findings (placement.py:166 `"PlacementFlags"`, uneven_perf.py:5028 `"Dict[str, List[float]]"`), both confirmed pre-existing at d653405 and outside sgl-project#503's diff hunks. - codespell clean on all 8 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…tor co-solved The sgl-project#602 capacity solve withdrew the cut for the POOL objective. The question actually asked -- more prefill on the 5090 -- is a time objective and was never solved for. This adds it, and then corrects its own first framing. Timing model, calibrated from the measured 49.2/154.8/116.4 ms at [28,20,16] (1.757/7.740/7.275 ms per layer). Two objectives are reported side by side because they disagree: SERIAL sum(layers_r * ms_r) and PIPELINED max over stages. Backtest: the model reproduces the calibration point exactly, and the hand-arithmetic anchor holds -- [42,12,10] gives 239.43 ms against 320.4, i.e. 1.338x, matching the 1.34x expected by hand. Result table under the rank0 cap (x_serial / x_pipelined vs incumbent): [28,20,16] 1.000 / 1.000 incumbent [42,12,10] 1.338 / 1.667 the anchor [42,11,11] 1.341 / 1.818 DOMINATES the anchor on BOTH axes [42, 1,21] 1.367 / 1.013 serial-optimal and useless once pipelined The serial optimum is a degenerate cut that collapses pipelined throughput, so reporting only one objective would have recommended it. [42,11,11] is the candidate to arm. Rank0 cap recomputed rather than quoted: at 724.3 MiB/layer against a 31,800 MiB budget, 42 layers leave 1,379 MiB (corridor OK), 43 leave 655 MiB (CORRIDOR VIOLATED), 44 overflow by 69 MiB. So DESIGN_691's "about 42" is confirmed, and the binding constraint at 42 is the CORRIDOR, not raw weight overflow -- 43 fits the card and still must be refused. Co-solve correction (user, restating sgl-project#320/sgl-project#492): revision 1 priced a "pool cost" per candidate while holding the KV token vector PINNED. That is single-family optimization and sgl-project#485 forbids it. Layers moved to rank0 free exactly their weight bytes on the rank they left, and the uneven-DCP / rank-kv-ratio machinery relocates the displaced KV share onto those bytes. Rank0's cap therefore bounds rank0's SHARE of the token split, not the world pool. The conservation is exact, not approximate, and the tests prove it rather than assert it: total VRAM is fixed and the same 64 layers of weights exist wherever they sit, so total free bytes are invariant; under DCP token-sharding one token costs kv_per_token_per_layer * total_layers wherever it lands. World pool is therefore identical across every feasible cut, and the only residual is second-order -- seam/staging in both directions plus TP-phase redistribution -- which is ITEMIZED and whose sum equals the delta exactly (no fudge term). Scope note owed to the earlier verdict: "29,19,16 is 6.3 percent worse" held only under a pinned vector. Co-solved, that comparison does not stand as a capacity argument. Calibration limit stated in the model docstring and pinned by a test: one measured cut gives one point per rank and cannot separate per-layer slope from fixed per-stage cost. fixed_ms defaults to zero, the OPTIMISTIC end, so every speedup here is an upper bound. A second measured cut per rank would pin it. Tests: 15, red first. Planner regression 41 passed across the 702 and 602 suites. ruff clean. No deploy, no arm.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…tor co-solved The sgl-project#602 capacity solve withdrew the cut for the POOL objective. The question actually asked -- more prefill on the 5090 -- is a time objective and was never solved for. This adds it, and then corrects its own first framing. Timing model, calibrated from the measured 49.2/154.8/116.4 ms at [28,20,16] (1.757/7.740/7.275 ms per layer). Two objectives are reported side by side because they disagree: SERIAL sum(layers_r * ms_r) and PIPELINED max over stages. Backtest: the model reproduces the calibration point exactly, and the hand-arithmetic anchor holds -- [42,12,10] gives 239.43 ms against 320.4, i.e. 1.338x, matching the 1.34x expected by hand. Result table under the rank0 cap (x_serial / x_pipelined vs incumbent): [28,20,16] 1.000 / 1.000 incumbent [42,12,10] 1.338 / 1.667 the anchor [42,11,11] 1.341 / 1.818 DOMINATES the anchor on BOTH axes [42, 1,21] 1.367 / 1.013 serial-optimal and useless once pipelined The serial optimum is a degenerate cut that collapses pipelined throughput, so reporting only one objective would have recommended it. [42,11,11] is the candidate to arm. Rank0 cap recomputed rather than quoted: at 724.3 MiB/layer against a 31,800 MiB budget, 42 layers leave 1,379 MiB (corridor OK), 43 leave 655 MiB (CORRIDOR VIOLATED), 44 overflow by 69 MiB. So DESIGN_691's "about 42" is confirmed, and the binding constraint at 42 is the CORRIDOR, not raw weight overflow -- 43 fits the card and still must be refused. Co-solve correction (user, restating sgl-project#320/sgl-project#492): revision 1 priced a "pool cost" per candidate while holding the KV token vector PINNED. That is single-family optimization and sgl-project#485 forbids it. Layers moved to rank0 free exactly their weight bytes on the rank they left, and the uneven-DCP / rank-kv-ratio machinery relocates the displaced KV share onto those bytes. Rank0's cap therefore bounds rank0's SHARE of the token split, not the world pool. The conservation is exact, not approximate, and the tests prove it rather than assert it: total VRAM is fixed and the same 64 layers of weights exist wherever they sit, so total free bytes are invariant; under DCP token-sharding one token costs kv_per_token_per_layer * total_layers wherever it lands. World pool is therefore identical across every feasible cut, and the only residual is second-order -- seam/staging in both directions plus TP-phase redistribution -- which is ITEMIZED and whose sum equals the delta exactly (no fudge term). Scope note owed to the earlier verdict: "29,19,16 is 6.3 percent worse" held only under a pinned vector. Co-solved, that comparison does not stand as a capacity argument. Calibration limit stated in the model docstring and pinned by a test: one measured cut gives one point per rank and cannot separate per-layer slope from fixed per-stage cost. fixed_ms defaults to zero, the OPTIMISTIC end, so every speedup here is an upper bound. A second measured cut per rank would pin it. Tests: 15, red first. Planner regression 41 passed across the 702 and 602 suites. ruff clean. No deploy, no arm.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…le, and what the phase matrix owes Docs only. No code changed, nothing built or wired. ## sgl-project#702 decision table (docs/DECISION_702_cut_table.md) Re-derived on the sgl-project#707 closed form with the calibrated fixture. Incumbent pool reads 436,275 here against the 436,766 in older notes; all percentages use 436,275 so they are internally consistent. cut attn pool speed noise(14.1%) gate [28,20,16] (7,5,4) 436,275 1.0000 -- is the gate [30,18,16] (7,5,4) 436,275 (+0.0%) 1.1111 BELOW IT PASSES [33,15,16] (8,4,4) 415,859 (-4.7%) 1.3299 above FAILS Switch cost 1575 ms from the incumbent either way (distance-independent, per #704a); payback 15.75 s and 6.35 s respectively. THE TRADE: the two candidates fail in OPPOSITE places, which is the decision. [30,18,16] passes the stage-1 "pool >= incumbent" gate and cannot be validated by measurement -- its 11.1% gain sits BELOW this rig's measured 14.1% A-vs-A noise floor, so no boot here can tell it from the incumbent. [33,15,16] clears the noise floor comfortably but needs a 4.7% capacity waiver. No ranking is offered; the pick is the user's. [30,18,16]'s pool is identical STRUCTURALLY, not coincidentally: it keeps the (7,5,4) attention split and leaves PP2 (layers 48-63) untouched, and PP2 is the binder. Per-rank capacity is [594,615/462,920/436,275] at the incumbent and [517,464/570,931/436,275] here -- PP2's column is unchanged, so moving two layers between PP0 and PP1 cannot move a pool PP2 sets. CORRECTION: I could not reproduce a "+20% pool" figure for [30,18,16]; its coupled pool is +0.0%. The nearest +17.7% on the list is the DECOUPLED pool, which is cut-INDEPENDENT (the #704b R6 sum-rule result: 513,875 for every cut including the incumbent) and therefore not attributable to any cut. SOLVER DEFECT FOUND, flagged not fixed: prefill_frontier.py:144-156 enumerates, per lead depth, ONLY the tail split that minimises pipelined time. At n0=30 that is [30,16,18] (130.95 ms), so [30,18,16] (139.32 ms) is never generated -- yet [30,16,18] holds 343,951 tokens (-21.2%) against [30,18,16]'s 436,275 (+0.0%). The frontier is Pareto-optimal in SPEED but not complete in POOL: a roomier, slightly slower cut at the same lead depth cannot appear on it. [30,18,16] strictly dominates the incumbent (equal pool, more speed) and was invisible for exactly that reason, which is why it never reached the sgl-project#702 block. The fix would be to keep the Pareto set over (pool, speed) per lead depth instead of the speed-argmin; other pool-preserving cuts may also be hidden. Validation-boot evidence is specified in falsification order, and includes the one that matters most: for [30,18,16] the prefill A/B is expected to be INCONCLUSIVE by construction -- a boot reporting a clean 1.11x has measured its own noise, and that result must be rejected rather than celebrated. Appended to the morning plan file as a dated block per its amendment convention; head-298 sha256 verified byte-identical before and after (81560e94b7628e6c8...). That file is not version-controlled, hence this copy. ## sgl-project#485 completion verdict (docs/VERDICT_485_phase_matrix.md) Directive quoted verbatim from /spinning/htsglang/CLAUDE.md:87-96 (user law 2026-08-03), which is NOT tracked in this worktree. The two facets under sgl-project#485 are the same ticket, not a number collision: the matrix doctrine plus its prefill-column solver, and the seam/memory certification gating it. VERDICT: PARTIALLY DELIVERED, do not close. DELIVERED prefill column for the attention-vs-GDN/linear boundary (c08f613, b675630, b3c9802, e645aa7), with a second axis on the attention row (sgl-project#492 d82a778, corrected by sgl-project#503 447249a). Its own feasibility facet (407b382) is RED: the threshold is not reachable at the certification pool. SUPERSEDED d937d5f refuses the decode-side family split on sound MEASURED reasoning -- against the honest proportional (uneven-TP) baseline the split is worth +0.090 ms/round, ~0.3% of a 30 ms bs=1 round. But this closes ONE CANDIDATE CELL (concentration), not the column. Named tension: the directive says single-family arms are DIAGNOSTIC only and must never be phrased as a phase-level verdict, and that commit is worded as one. The narrow reading is adopted. OPEN O1 no (family x phase) structure exists anywhere -- pp_cut.py solves prefill only, family_split.py returns by_family with NO phase argument or dimension; the "matrix" is prose over two single-phase solvers. O2 the decode solver is DEAD CODE: solve_family_placement has exactly two references, its definition (family_split.py:141) and its test, zero call sites. O3 vocab, experts, nonlinear kernels and per-quant-lane linear layers were never cut at all. O4 the law covers diffusion, SR/video, TTS/ASR and training tenants -- entirely untouched, and the largest open surface. O5 the prefill column is not arm-able while Facet B is red. NOT sgl-project#485 sgl-project#704/#704a/#704b are a PP-cut layout LADDER and its arming machinery. They consume the prefill-column solve as an input and must not be counted toward the matrix; counting them would inflate its delivery with work on a different axis. Status in one line: one of two columns, one of many families, one of several workloads -- and the table is still prose. Every load-bearing claim spot-verified against the tree rather than taken from the survey: directive text read at source, solve_family_placement call sites grepped, d937d5f's baselines read from the commit body, [30,18,16] priced directly rather than inferred from frontier membership. Sanity: 35 passed (test_prefill_frontier_702.py, test_fill_ladder_704a.py). Desk only, no GPU, deploy tree untouched.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 13, 2026
…not exclusive split (W1)
BOOT7 (Zwerg Qwen3.5-2B, num_key_value_heads=2, D=TP3): every rank's
canonical draft-KV window install raised CanonicalPageError "draft head
window mismatch (W1)" identically. _draft_head_window (cache_controller.py)
already gives every rank the full (0, total) window when
attn_kv_replicated(tp_size, total) is True (kv < tp) -- mirroring
ModelConfig._uneven_tp_num_kv_heads, which already sizes the draft device/
host pool the same way ("REPLICATED-KV geometry -- every rank holds ALL kv
heads. Rank-independent by construction."). local_head_window
(disaggregation/draft_kv_canonical.py), used as the independent reference
both P and D compute the window from, instead did a largest-remainder-first
EXCLUSIVE split with an empty window for ranks beyond num_kv_heads -- the
wrong convention below the threshold, disagreeing with attn_kv_replicated
everywhere else in the tree (distributed/utils.py, the sgl-project#492 falsifier,
test_kv_eq_tp_stays_in_normal_mode_by_measurement).
Fix: local_head_window returns (0, num_kv_heads) for every rank when
num_kv_heads < tp_size, before falling back to the existing largest-
remainder split (unchanged, still correct, for num_kv_heads >= tp_size).
Kept as a plain arithmetic check rather than importing attn_kv_replicated/
tp_plan_active: the module's whole point is that P (tp_size=1, no installed
ratio plan) and D compute the window independently from
(num_kv_heads, tp_size, tp_rank) alone, with no shared plan state.
Host-pool allocation needed no change -- already replicated via
_uneven_tp_num_kv_heads -- only a pinning test, closing the "head_num=0 or
replicated" question the ticket posed: replicated, and already so.
Tests, hermetic (CUDA_VISIBLE_DEVICES=""), reproduced red before the fix
(build_draft_window raising W1 at kv=2/tp=3 exactly as on BOOT7) and green
after:
- test_draft_kv_canonical_631b.py: rewrote the test that pinned the wrong
convention (test_more_ranks_than_heads_yields_empty_windows ->
test_kv_lt_tp_yields_full_replicated_windows_not_empty, with the reasoning
in the docstring), scoped the exclusive-split exactness test to its real
domain (num_heads >= tp_size).
- test_weg2_draftkv_canonical_1233.py: new TestDraftWindowKvLtTp -- full-
window pin, kv==tp boundary stays exclusive-split, and a mutant test for
the danger direction (a reintroduced exclusive split at kv<tp must still
be refused by W1, not silently accepted as a smaller-but-plausible
window).
- test_uneven_tp_memory.py: TestNumKvHeadsAutoRank.
test_kv_lt_tp_replicates_full_head_count_on_every_rank pins the host/
device pool allocation side.
Green: test_draft_kv_canonical_631b.py, test_weg2_draftkv_canonical_1233.py,
test_weg2_draftkv_keys_1233.py, test_canonical_page_store_706.py,
test_canonical_kv_page_kv_major_1233.py, test_canonical_kv_page_704b.py,
test_canonical_kv_page_args_706.py, test/registered/unit/disaggregation/
(250 passed), test_uneven_tp_memory.py, test_uneven_tp_args.py,
test_attn_replication_axis_492.py, test_weg2_draftkv_l3_write_1233.py,
test_embedding_cache_controller.py,
test_cache_controller_reset_joins_io_aux_1068.py. Import smoke clean.
ruff clean on all four changed files; codespell clean.
Two pre-existing failures unrelated to this change, verified not touched by
this diff: test_cache_controller_reset_terminates_inflight_1068.py
(AttributeError 'has_draft' in a background thread) and
test_hybrid_decode_pool_attrs_631.py (mamba pool attribute contract) --
both outside this ticket's scope.
Design doc: /spinning/gpu-arb/weg2/DESIGN_draftkv_kvlt_tp_0913.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #488.
We need the same seed for tensor parallelism