Skip to content

Revert "Revert "fix: update flashinfer to 0.1.2 to fix sampling for cu118"" - #806

Merged
Ying1123 merged 1 commit into
mainfrom
revert-805-revert-803-sample
Jul 29, 2024
Merged

Ying1123 merged 1 commit into
mainfrom
revert-805-revert-803-sample

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

Reverts #805

0.1.2 ready

@Ying1123
Ying1123 requested a review from zhyncs July 29, 2024 19:15
@Ying1123
Ying1123 merged commit eba458b into main Jul 29, 2024
@Ying1123
Ying1123 deleted the revert-805-revert-803-sample branch July 29, 2024 19:20
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
litmei pushed a commit to litmei/sglang_litmei that referenced this pull request Jun 16, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…ly-refusal insert

Third drift of this one cite in a day, same mechanism each time. The sgl-project#806
launch refusal adds ~61 lines to server_args.py above the PP guard, so the
cite that sgl-project#810's merge had just moved to :19042/:19057 now names unrelated
code again. Read off the block, not searched: the guard "if self.pp_size > 1:"
is at :19103 and the speculation assert at :19118.

  test_rejected_evidence_pins.py   1 failed, 5 passed -> 6 passed

Worth a decision by whoever owns the register (NOT taken here): the suite's
own docstring claims "the pin is deliberately on the assert's TEXT rather
than on its line number, so ordinary edits above it do not turn this red".
That is true of the ASSERTIONS but not of the DATA they read -- the evidence
string itself carries FILE:LINE, and the test resolves those numbers against
the current file. So every merge that inserts a line above the guard turns
this red, which is the opposite of what the docstring promises. Three
mechanical re-pins in one day is the cost. A text-anchored evidence form
would end it; changing the register's evidence format is a design call.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818

Merge-checklist duty for the nine stages on this branch. Entries follow each
section's own house style, and the mechanism text is taken from the commits'
measured evidence rather than restated from the ticket titles.

§3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given
a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap
mechanism that bullet already describes: the census reading the withheld block
as a leak (340262 of 465190 ids), the lift being reachable only from a cutover
(one boot at 26.8% of its id space for the life of the process), and exposure
exceeding the backing (417850 rows over 105413 committed, the device-side
assert in masked_set_kv_buffer_kernel).

§3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two
companion refusals -- the unbounded-file-tier refusal and the boot preflight
ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across
three PP ranks previously reached the preflight as nothing.

§7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the
existing narrative: the gate could wait forever on a peer that no longer
exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor
defer_stall_for_building_peer (900 s off a build marker) caught it.

§12 Robustness canon -- three new families: contradictory-flag (sgl-project#806),
read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820).

§18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a
reusable module adds its entry in the SAME merge, and this module had none.
Records the removal of fits_pinned_host_budget so it is not reintroduced.

§18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N
(sgl-project#777).

No existing entry was contradicted. Checked before writing: none of sgl-project#814,
sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and
the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different
mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone.

Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""):

  battery test/registered/unit/{managers,planner,server_args,mem_cache}
    baseline integ @ 78d27da       44 failed
    c56d238 (through sgl-project#818)         30 failed, 8452 passed   0 new ids, 14 fixed
    1c4eadb (through sgl-project#816)         30 failed, 8461 passed   0 new ids, same set
  test_barlink_abort_gate_liveness_818.py (outside the battery dirs)  10 passed
  ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same
    file set on 78d27da (16 ruff / 6 codespell exist identically on base)
  docs-only change; codespell clean on the catalog itself
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…t inventory

USER DECISION 2026-08-24, binding, verbatim: "das kv soll niemals vom layer
flip her stammen, einfach aus dem hicache laden fertig." Recorded BEFORE the
build so the build cannot quietly become something else, and so the ledger
that follows reads as validation rather than as a vote.

THE TREE ALREADY AGREED, which is the part worth landing on its own.
hicache_flip_writeback.py:21-23 states the premise in its own words: "a
prefix's only way across the flip is the geometry-free STORE (sgl-project#706): the disk
tier, whose keys carry content alone and whose pages are cut at read time for
whichever geometry asks." And it names exactly why a mover exists today:
"device rows survive the flip, because the live row set (radix tree values
UNION parked requests' rows) is relocated between the two phase pools BY ROW
ID." That relocation IS the wave mover.

THE BLOCKER THAT SHAPES THE BUILD, found before writing any code.
phase_flip_resident_carry.py:64-76 -- a carried Req keeps its req_pool_idx
across the swap "by construction", and the reason it stays valid is that "the
bytes behind those ids are what the KV and GDN movers relocate". So
PHASE-FLIP-CARRY is NOT a KV mover and NOT a retirement candidate, but its
correctness today DEPENDS on the mover. Retire the mover alone and a resident
request's req_pool_idx points at unwritten memory. The fence and the
retirement must therefore land together, and the cutover must leave the new
phase's device tier in a state where a lookup MISSES rather than returning
stale rows. That is the correctness core, and it is where the red-first tests
must bite hardest.

WHY THE PRIZE IS FUNDING, NOT LATENCY -- stated plainly because the opposite
is the natural assumption. The whole KV+GDN movement is 901 ms of an 11.6 s
seam (W25 epoch 11, 116502 live slots). Removing it leaves ~10.5 s. What it
DOES remove is wave_peak = incoming + max(outgoing, local) + one_layer_window
+ backing_slack (phase_flip_runtime.py:7314) -- every term a KV quantity --
which is the 2339.11 MiB tp_to_pp staging reserve behind W25's 33 refused
arms, 25 of them on the staging rate limit, and 17 FLIP ABANDONED.

RETIREMENT INVENTORY, scoped to the flip path, REPLACE/RETIRE/KEEP-WITH-REASON
with file:line, in the note. Rules held to: "hardened against corruption" does
not count as reconciled; no bulk deletion; shared machinery with a named other
consumer is KEPT (kv_reshard's sgl-project#297 domain, gdn_flip_preconditions -- whose
"no other consumer found" is recorded WITH the narrow search set that failed,
not as a licence to delete); anything merely deletable-later is a separate
section and marked un-re-verified.

EXISTING MACHINERY TO EXTEND, NOT REBUILD: hicache_flip_writeback.py already
IS the fence (stage + bounded ack drain, deadline 2.0 s); hicache_demotion.py
already covers evict-before-persist WITH counters and is merely off by
default; sgl-project#719 rebind + sgl-project#718 disarm already own the read-path switch;
mamba_ckpt_utils' anchor grid is already what anchor-resume needs.

NO FALLBACKS, per standing doctrine: every gap is fixed inside the HiCache
route. A flip enabled without hierarchical cache becomes a validate-early
launch refusal on the sgl-project#806 precedent (c0a6347, ServerArgs.__post_init__
after materialize_declarations) -- NOT a silent mover revival. That refusal is
deliberately NOT landed yet: it only becomes true once the flip carries no KV,
and shipping it early would reject launches that work correctly today. Forced
build order, recorded rather than discovered later.

VALIDATION METRIC CHANGED, and one half of it does not exist yet: cutover-
blocking time (fence + weights refill) can reuse the existing
seam_census.mark("flip_writeback") and the DONE stats dict, but there is NO
instrument for post-cutover warm-up cost as served-request latency -- searched
and named as a build item rather than assumed present.

OPEN GAPS ARE LISTED AS GAPS, with the search sets that failed: sgl-project#735's "sgl-project#706
rows on the full plan" could not be located (both greps resolve to a different
topic -- non-contiguous PP placement), fp8 kv_cache_dtype against the canonical
page format is unverified, and the worst-case un-hashed resident tail at the
quiescent-flip instant is reasoned from the quiescence predicate rather than
measured.

No code changes. Documentation only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…ure what warm-up costs

TWO PIECES THE DESIGN OWES, both consequences of carrying no KV.

1. VALIDATE-EARLY REFUSAL (sgl-project#806 pattern, ServerArgs.__post_init__).
The flip retracts every resident request and DROPS the prefix tree, so the
next phase starts with an empty device tier and restores prefixes by
read-through. With --enable-hierarchical-cache off there is nothing to read
through: the sgl-project#703 flip-time writeback has nowhere to persist, the retracted
prefixes are gone, and every conversation re-prefills from scratch on every
flip.

That is a correctness-shaped cost, not a tuning one, and it is INVISIBLE at
runtime -- the flip completes, the requests complete, and only the token bill
says anything happened. A silent, expensive, correct-looking failure is what a
launch gate is for.

Placed beside sgl-project#806's own check and after `materialize_declarations`, for the
reason sgl-project#806 documents: the hierarchical cache is switched by handlers above,
so an earlier check would pass a launch this one refuses. sgl-project#806 refuses the
flip that cannot ENUMERATE what it must move; this refuses the flip that
cannot RESTORE what it deliberately drops.

NO FALLBACK IS OFFERED AND THE MESSAGE SAYS SO -- a test asserts the phrase.
Reviving the mover here would reintroduce the seam this ticket retired, the
staging reserve behind W25's 33 refused arms, and the resident carry that
crashed three ranks in sgl-project#825. Both exits are named instead, because which one
is right depends on what the operator wanted.

2. THE WARM-UP LEDGER (managers/warmup_latency.py), the metric the user
named: not "rows carried" but served-request latency by rounds since cutover.

PRIOR ART CHECKED AND IT IS A NEAR MISS, not a hit:
`regime_classifier.PhaseDwellGate.rounds_since_flip` has the rounds-since-flip
concept but is a GATE deciding whether a flip may happen, and carries no
latency. Also searched: phase_flip_runtime for warm/latency/post-cutover, the
seam census (times the seam, not what follows it), the sgl-project#605 flight recorder
(the seam's own peaks). Nothing measures this.

Geometric bands (<=1, <=4, <=16, <=64, steady) because the CLAIM is that the
cost concentrates in the first rounds and decays; a mean cannot be right or
wrong in a way that shows that. Every band reports as a ratio against THIS
instance's own steady state, so the figure needs no remembered number from
another boot to be readable.

The can-fail directions are the file's point. `None` survives wherever
nothing was compared -- a ratio against an absent control reads as "no
warm-up cost" while meaning "nothing was compared", the sgl-project#606 shape in the one
number this ticket is judged on. "Has not flipped" never folds into "has
flipped and settled". An empty ledger still says it has nothing to say.

HALF-WIRED, AND SAID PLAINLY. `note_cutover` is wired at the cutover. The
request feed is NOT: request latency is assembled in `tokenizer_manager`, a
DIFFERENT PROCESS, so feeding it is a cross-process change rather than a line.
Recorded as an open integration point instead of guessed at -- a
wrongly-wired instrument reports a number, which is worse than reporting none,
and this is the number the proof window judges the design on.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_phase_flip_needs_hicache_856.py   in server_args/ -- 776 passed overall
  test_warmup_latency_856.py             13 passed, 5 subtests
  managers -k "phase_flip or seam or warmup or uncached"
                                         564 passed, 28 subtests, 0 failed
  ruff format clean; ruff check adds ZERO errors -- verified per file from the
  repo root against HEAD: server_args 358 -> 358, phase_flip_runtime 0,
  warmup_latency 0, corridor_admission 0.

RUFF MEASUREMENT CORRECTION, since a previous commit reasoned from it: ruff
must be run FROM THE REPO ROOT or it does not find pyproject.toml and reports
a different rule set entirely (phase_flip_runtime.py read as "110 errors" from
a stray cwd and "All checks passed" from the root). The earlier 144-vs-144
comparison was taken with the same cwd on both sides, so its conclusion --
zero added -- stands; the absolute number was just scheduler.py's 102 plus
schedule_batch.py's 42.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants