Repository navigation
Revert "Revert "fix: update flashinfer to 0.1.2 to fix sampling for cu118"" - #806
Merged
Merged
Conversation
…u118" (#…" This reverts commit 7d352b4.
zhyncs
approved these changes
Jul 29, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
litmei
pushed a commit
to litmei/sglang_litmei
that referenced
this pull request
Jun 16, 2026
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…ly-refusal insert Third drift of this one cite in a day, same mechanism each time. The sgl-project#806 launch refusal adds ~61 lines to server_args.py above the PP guard, so the cite that sgl-project#810's merge had just moved to :19042/:19057 now names unrelated code again. Read off the block, not searched: the guard "if self.pp_size > 1:" is at :19103 and the speculation assert at :19118. test_rejected_evidence_pins.py 1 failed, 5 passed -> 6 passed Worth a decision by whoever owns the register (NOT taken here): the suite's own docstring claims "the pin is deliberately on the assert's TEXT rather than on its line number, so ordinary edits above it do not turn this red". That is true of the ASSERTIONS but not of the DATA they read -- the evidence string itself carries FILE:LINE, and the test resolves those numbers against the current file. So every merge that inserts a line above the guard turns this red, which is the opposite of what the docstring promises. Three mechanical re-pins in one day is the cost. A text-anchored evidence form would end it; changing the register's evidence format is a design call.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818 Merge-checklist duty for the nine stages on this branch. Entries follow each section's own house style, and the mechanism text is taken from the commits' measured evidence rather than restated from the ticket titles. §3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap mechanism that bullet already describes: the census reading the withheld block as a leak (340262 of 465190 ids), the lift being reachable only from a cutover (one boot at 26.8% of its id space for the life of the process), and exposure exceeding the backing (417850 rows over 105413 committed, the device-side assert in masked_set_kv_buffer_kernel). §3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two companion refusals -- the unbounded-file-tier refusal and the boot preflight ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across three PP ranks previously reached the preflight as nothing. §7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the existing narrative: the gate could wait forever on a peer that no longer exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor defer_stall_for_building_peer (900 s off a build marker) caught it. §12 Robustness canon -- three new families: contradictory-flag (sgl-project#806), read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820). §18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a reusable module adds its entry in the SAME merge, and this module had none. Records the removal of fits_pinned_host_budget so it is not reintroduced. §18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N (sgl-project#777). No existing entry was contradicted. Checked before writing: none of sgl-project#814, sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone. Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""): battery test/registered/unit/{managers,planner,server_args,mem_cache} baseline integ @ 78d27da 44 failed c56d238 (through sgl-project#818) 30 failed, 8452 passed 0 new ids, 14 fixed 1c4eadb (through sgl-project#816) 30 failed, 8461 passed 0 new ids, same set test_barlink_abort_gate_liveness_818.py (outside the battery dirs) 10 passed ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same file set on 78d27da (16 ruff / 6 codespell exist identically on base) docs-only change; codespell clean on the catalog itself
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…t inventory USER DECISION 2026-08-24, binding, verbatim: "das kv soll niemals vom layer flip her stammen, einfach aus dem hicache laden fertig." Recorded BEFORE the build so the build cannot quietly become something else, and so the ledger that follows reads as validation rather than as a vote. THE TREE ALREADY AGREED, which is the part worth landing on its own. hicache_flip_writeback.py:21-23 states the premise in its own words: "a prefix's only way across the flip is the geometry-free STORE (sgl-project#706): the disk tier, whose keys carry content alone and whose pages are cut at read time for whichever geometry asks." And it names exactly why a mover exists today: "device rows survive the flip, because the live row set (radix tree values UNION parked requests' rows) is relocated between the two phase pools BY ROW ID." That relocation IS the wave mover. THE BLOCKER THAT SHAPES THE BUILD, found before writing any code. phase_flip_resident_carry.py:64-76 -- a carried Req keeps its req_pool_idx across the swap "by construction", and the reason it stays valid is that "the bytes behind those ids are what the KV and GDN movers relocate". So PHASE-FLIP-CARRY is NOT a KV mover and NOT a retirement candidate, but its correctness today DEPENDS on the mover. Retire the mover alone and a resident request's req_pool_idx points at unwritten memory. The fence and the retirement must therefore land together, and the cutover must leave the new phase's device tier in a state where a lookup MISSES rather than returning stale rows. That is the correctness core, and it is where the red-first tests must bite hardest. WHY THE PRIZE IS FUNDING, NOT LATENCY -- stated plainly because the opposite is the natural assumption. The whole KV+GDN movement is 901 ms of an 11.6 s seam (W25 epoch 11, 116502 live slots). Removing it leaves ~10.5 s. What it DOES remove is wave_peak = incoming + max(outgoing, local) + one_layer_window + backing_slack (phase_flip_runtime.py:7314) -- every term a KV quantity -- which is the 2339.11 MiB tp_to_pp staging reserve behind W25's 33 refused arms, 25 of them on the staging rate limit, and 17 FLIP ABANDONED. RETIREMENT INVENTORY, scoped to the flip path, REPLACE/RETIRE/KEEP-WITH-REASON with file:line, in the note. Rules held to: "hardened against corruption" does not count as reconciled; no bulk deletion; shared machinery with a named other consumer is KEPT (kv_reshard's sgl-project#297 domain, gdn_flip_preconditions -- whose "no other consumer found" is recorded WITH the narrow search set that failed, not as a licence to delete); anything merely deletable-later is a separate section and marked un-re-verified. EXISTING MACHINERY TO EXTEND, NOT REBUILD: hicache_flip_writeback.py already IS the fence (stage + bounded ack drain, deadline 2.0 s); hicache_demotion.py already covers evict-before-persist WITH counters and is merely off by default; sgl-project#719 rebind + sgl-project#718 disarm already own the read-path switch; mamba_ckpt_utils' anchor grid is already what anchor-resume needs. NO FALLBACKS, per standing doctrine: every gap is fixed inside the HiCache route. A flip enabled without hierarchical cache becomes a validate-early launch refusal on the sgl-project#806 precedent (c0a6347, ServerArgs.__post_init__ after materialize_declarations) -- NOT a silent mover revival. That refusal is deliberately NOT landed yet: it only becomes true once the flip carries no KV, and shipping it early would reject launches that work correctly today. Forced build order, recorded rather than discovered later. VALIDATION METRIC CHANGED, and one half of it does not exist yet: cutover- blocking time (fence + weights refill) can reuse the existing seam_census.mark("flip_writeback") and the DONE stats dict, but there is NO instrument for post-cutover warm-up cost as served-request latency -- searched and named as a build item rather than assumed present. OPEN GAPS ARE LISTED AS GAPS, with the search sets that failed: sgl-project#735's "sgl-project#706 rows on the full plan" could not be located (both greps resolve to a different topic -- non-contiguous PP placement), fp8 kv_cache_dtype against the canonical page format is unverified, and the worst-case un-hashed resident tail at the quiescent-flip instant is reasoned from the quiescence predicate rather than measured. No code changes. Documentation only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…ure what warm-up costs TWO PIECES THE DESIGN OWES, both consequences of carrying no KV. 1. VALIDATE-EARLY REFUSAL (sgl-project#806 pattern, ServerArgs.__post_init__). The flip retracts every resident request and DROPS the prefix tree, so the next phase starts with an empty device tier and restores prefixes by read-through. With --enable-hierarchical-cache off there is nothing to read through: the sgl-project#703 flip-time writeback has nowhere to persist, the retracted prefixes are gone, and every conversation re-prefills from scratch on every flip. That is a correctness-shaped cost, not a tuning one, and it is INVISIBLE at runtime -- the flip completes, the requests complete, and only the token bill says anything happened. A silent, expensive, correct-looking failure is what a launch gate is for. Placed beside sgl-project#806's own check and after `materialize_declarations`, for the reason sgl-project#806 documents: the hierarchical cache is switched by handlers above, so an earlier check would pass a launch this one refuses. sgl-project#806 refuses the flip that cannot ENUMERATE what it must move; this refuses the flip that cannot RESTORE what it deliberately drops. NO FALLBACK IS OFFERED AND THE MESSAGE SAYS SO -- a test asserts the phrase. Reviving the mover here would reintroduce the seam this ticket retired, the staging reserve behind W25's 33 refused arms, and the resident carry that crashed three ranks in sgl-project#825. Both exits are named instead, because which one is right depends on what the operator wanted. 2. THE WARM-UP LEDGER (managers/warmup_latency.py), the metric the user named: not "rows carried" but served-request latency by rounds since cutover. PRIOR ART CHECKED AND IT IS A NEAR MISS, not a hit: `regime_classifier.PhaseDwellGate.rounds_since_flip` has the rounds-since-flip concept but is a GATE deciding whether a flip may happen, and carries no latency. Also searched: phase_flip_runtime for warm/latency/post-cutover, the seam census (times the seam, not what follows it), the sgl-project#605 flight recorder (the seam's own peaks). Nothing measures this. Geometric bands (<=1, <=4, <=16, <=64, steady) because the CLAIM is that the cost concentrates in the first rounds and decays; a mean cannot be right or wrong in a way that shows that. Every band reports as a ratio against THIS instance's own steady state, so the figure needs no remembered number from another boot to be readable. The can-fail directions are the file's point. `None` survives wherever nothing was compared -- a ratio against an absent control reads as "no warm-up cost" while meaning "nothing was compared", the sgl-project#606 shape in the one number this ticket is judged on. "Has not flipped" never folds into "has flipped and settled". An empty ledger still says it has nothing to say. HALF-WIRED, AND SAID PLAINLY. `note_cutover` is wired at the cutover. The request feed is NOT: request latency is assembled in `tokenizer_manager`, a DIFFERENT PROCESS, so feeding it is a cross-process change rather than a line. Recorded as an open integration point instead of guessed at -- a wrongly-wired instrument reports a number, which is worse than reporting none, and this is the number the proof window judges the design on. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_phase_flip_needs_hicache_856.py in server_args/ -- 776 passed overall test_warmup_latency_856.py 13 passed, 5 subtests managers -k "phase_flip or seam or warmup or uncached" 564 passed, 28 subtests, 0 failed ruff format clean; ruff check adds ZERO errors -- verified per file from the repo root against HEAD: server_args 358 -> 358, phase_flip_runtime 0, warmup_latency 0, corridor_admission 0. RUFF MEASUREMENT CORRECTION, since a previous commit reasoned from it: ruff must be run FROM THE REPO ROOT or it does not find pyproject.toml and reports a different rule set entirely (phase_flip_runtime.py read as "110 errors" from a stray cwd and "All checks passed" from the root). The earlier 144-vs-144 comparison was taken with the same cwd on both sides, so its conclusion -- zero added -- stands; the absolute number was just scheduler.py's 102 plus schedule_batch.py's 42.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #805
0.1.2 ready