Repository navigation
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Garrybest
pushed a commit
to Garrybest/sglang
that referenced
this pull request
Jan 9, 2026
…taset is too low (sgl-project#551) * fix the problem gemma accuracy task takes too long on one machine * let the threshold of dataset aime to be -1 * fix the problem score for math problems too low
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…m a source that can lie TWO INDEPENDENT DEFECTS, both rooted in reading a number or a set from the wrong place. (A) THE GDN BLOB THAT GOT THROWN AWAY. build_gdn_slot_executor's idle_holders closure enumerated live offload sessions through kv_session_offload.spills. _commit_park POPS a sgl-project#224-parked session out of spills while the session stays alive and keeps its req-pool slot, its radix tree lock and its Mamba/GDN state slot. GdnSlotRuntime.on_round treats a parked id it cannot find in the inventory as DEAD and calls forget(sid) on it. So: a session spills, the ladder exports its GDN state to a blob, the session parks its KV to the next tier, and the next ladder round discards the blob. It later unparks, restores its KV, resumes -- and continues with recurrent state that no longer exists. No crash, no log, wrong tokens. The comment ("a parked session that died while parked leaves a blob nobody will ever ask for") was not wrong; its PREMISE was. Absence from the inventory meant "dead" only while every live session was in spills. Fixed with KVSessionOffloadManager.live_offload_reqs() -- spilled AND parked -- as the single enumeration, plus kv_session_spill_destination.parked_reqs() as the one place that knows where parked requests live (inflight_batches now goes through it too, so the abort scan and the ladder share one answer). Note the abort scan had ALREADY been taught about parked sessions; the knowledge existed in the module and the second consumer simply never received it, which is what a missing accessor costs. (B) THE HOST-RAM NUMBER. memtier/profile._host_memory_bytes read /proc/meminfo only, and the kvso host-pool guard sized a PINNED pool against psutil. Inside this container both are lxcfs-synthesised and do not describe what the process may have: MemAvailable can EXCEED MemTotal (an arithmetic impossibility, so the guard was comparing against a number that denotes nothing), and with memory.max unlimited they report the HOST's figures on a box other containers are also spending. Over-committing pinned memory is the OOM killer picking a victim, not a swap. honest_host_memory_bytes() is now the pure decision: a finite memory.max is the ceiling (capped by MemTotal), available is that ceiling minus anon and kernel -- page cache is reclaimable and deliberately NOT subtracted, since subtracting it would refuse boots that would have succeeded; with no finite limit MemTotal is the ceiling and MemAvailable the estimate, CLAMPED to it and clamped again by the cgroup's own resident accounting. Anything that cannot be established stays None rather than being guessed, and the kvso guard then SKIPS with a warning -- refusing to guess beats refusing a good boot on a fabricated figure. memtier owns this per DESIGN_407 §2.9 (eight ownerless psutil.virtual_memory() call sites; this module is meant to become the owner, not the ninth), so the guard reads host_memory_bytes_for_pinning() there rather than adding a ninth. This is also the honest number sgl-project#547's joint pinned-RAM budget will be built on. Tests (CUDA_VISIBLE_DEVICES=99, hermetic, no GPU, no server): PYTHONPATH=python pytest \ test/registered/unit/mem_cache/test_gdn_slot_runtime.py \ test/registered/unit/memtier test/registered/unit/server_args \ test/registered/unit/test_kv_spill_destination_unit.py \ test/registered/unit/test_kv_session_offload_unit.py \ test/registered/unit/test_kv_spill_budget_unit.py \ test/registered/unit/test_kvso_reclaim_decline_501.py -q -> 942 passed, 19 warnings, 103 subtests passed in 14.35s CAN-FAIL PROVEN, both halves: - pointing idle_holders back at manager.spills turns test_the_builder_sees_a_parked_session red with "Lists differ: ['s'] != ['p', 's']" -- the parked request missing from the inventory is exactly the state in which the ladder throws its blob away; - dropping the MemAvailable clamp turns test_the_impossible_meminfo_reading_is_clamped red with "134217728000 != 128849018880". The live-path test executes the real reader on this machine, so neither half is desk-written-never-executed. Every input to honest_host_memory_bytes is passed in, so no test asserts anything about the machine it runs on (directive sgl-project#434). FEATURE_CATALOG gains the absent-means-dead family entry. Handoff: /spinning/spill-night-20260804/handoffs/ TICKET_551_gdn_park_and_honest_ram.md -- half A needs a hybrid-model boot with --kv-session-offload-max-spills 1 (to force a park) and a binding --gdn-resident-state-slots (to force a vacate), and explicitly states that a run in which no park or no vacate happened proves nothing. ruff check: no new findings on any changed file. ruff format: clean on all added lines. codespell: clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…and a five-cut list Desk analysis only, no build, as scoped. docs/dev/ANALYSE_553_elastic_coresidence.md. What the four named building blocks actually are, read from the code rather than from their titles: - sgl-project#330 VRAM dial: a REAL mechanism, and its decisive property for sgl-project#553 is that store_bound_rows is a LIFETIME bound baked into captured graphs. So the dial has a FREE BAND (shrink to any floor, re-grow to the boot bound) and a HARD WALL above it. The free band is exactly where CUDA-graph compatibility is preserved. - sgl-project#364 GDN slot ladder: a real, executing hot/cold mover, and graph-neutral by construction (it moves state between slots, not geometry). It is the existence proof that a hot/cold event need not touch capture, and its inventory seam -- the owner enumerates its live set, consumers never read a container directly -- is the pattern the other consumers need (sgl-project#551). - sgl-project#287 KV pressure ladder: PLANNER ONLY. Seventeen NotImplementedError sites; the docstring says so plainly. But the ordering invariants are already enforced in code and the relief rungs reference existing features BY NAME, so the relief half can be made to execute by delegation without new mechanism. - sgl-project#286 register: the right vocabulary, but exactly one ledger exists and it is wired to the translator. There is no generic tenant to raise an event against. The pattern across the event table: both directions are largely built, and what is missing is ARBITRATION. Two registries exist (dial participants, sgl-project#286 asset classes) and neither can see the other, so "free 4 GiB for the video tenant" has no addressee. Cut list, ordered so each cut is provable alone and no cut before the last touches capture: (1) one registry or an explicit bridge; (2) a generic tenant mover factored out of AudioAssetLedger; (3) make sgl-project#287's RELIEF rungs execute by delegation -- the largest block of stubs, and it never approaches the graph wall because relief rungs are declared "no KV layout change"; (4) the arbitration policy on top; (5) LAST and only with the rig, cross the graph wall via the per-rung pre-captured graphs sgl-project#287 already specifies, with cold rungs parked as register class graph_rungs. Graph compatibility is treated as the target state throughout, per directive: every cut states its effect on graph state, and cut 5 carries the explicit route back to full coverage. Until cut 5 lands, sgl-project#553 must state its band -- "elastic within the boot capture bound" -- rather than implying unlimited elasticity. Instrumentation named honestly: SGLANG_KVSO_TICK_TRACE=1 (throttled per-session regulator trace: effective interval, measured tick_cost, binding headroom ratio, host-tail size; pure logging, no control effect). The analysis states what it is NOT -- a regulator trace, not a per-phase breakdown of device time -- and proposes no measurement it has not checked exists. One decision deliberately left open and named as such: whether sgl-project#553 owns the arbitration policy or whether it belongs to sgl-project#363. That fork needs the sgl-project#363 stage feed bound first, since an arbitration policy fed by a table holding one stage would have nothing to choose between -- the same failure sgl-project#363/S8 documents one level up. codespell: clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…mposability fixes) Desk strand (fix/spill-composability, included via 2a156c8): six bugs with proven can-fail falsifiers (sgl-project#552 parked-abort leak, sgl-project#552 spec-costs-spill, sgl-project#547 dead interop honesty, sgl-project#551 GDN blob loss + lxcfs-honest host RAM, sgl-project#568 ledger non-persistent buffer corruption), S8 docstring fix, analyses P6/P7/sgl-project#553. Full-sweep regression proof: failing-test sets byte-identical vs base e08946d (911=911), +41 newly passing. GPU strand: 8 matrix cells passed on hardware (kvso under uneven DCP, partial spill, restore gate, kvso x NEXTN, multi-session, ladder actuation, hibernate round trip 1.29x), matrix landed in-tree as docs/dev/ANALYSE_spill_matrix_20260804.md.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
The night merge 2a156c8 was cut at b9a9fd6 while the desk strand kept committing; this brings in the missing tail: sgl-project#551 GDN-ladder parked-session fix + honest host RAM, sgl-project#568 generic ledger non-persistent-buffer fix, S8 stage-feed docstring fix + AUDIT_421 B.9, analyses sgl-project#553/P6/P7 and the rejected-register entry. Restores the Absent-means-dead and Round-trip-asymmetry catalog families the earlier merge missed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
Design only, no code. Records the mandate plus all three addenda as requirements with falsifiers, and fixes the authority model: the planner decides, the ledger is the denominator, every existing mechanism becomes a sensor or an actuator. A slice that reimplements a carrier instead of driving it has failed the mandate. Addendum 3 becomes R6 and R7, both stated as error classes the design must be structurally unable to commit rather than behaviours it should prefer: R6 -- a state change re-SOLVES the cut (layer/TP split, draft placement, KV/token vectors, phase layouts) under the new usage profile. Pool-only squeezing is one candidate, not the search space. The planner may still choose pool-only, but that must be its cost calculation under R5, never a structural inability. F6 asserts on the CANDIDATE SET as well as the choice, because a planner that picked correctly from a set of one is not implementing this; F6b is the mirror, where pool-only wins on horizon and the planner must record which re-cut it rejected and why. R7 -- evacuation heat is PHASE-DEPENDENT, not static per item class. During prefill everything decode-owned is cold and goes first; during decode the ranking inverts. F7 is deliberately a PAIR (prefill-burst ranks decode graphs below KV, decode-burst inverts) because a static ranking passes either case alone. F7b adds imminence: at a prefill burst whose successor is decode, decode graphs must NOT be evacuated -- separating 'phase-dependent' from 'reacts to the instantaneous phase', which is a different and worse thing. Carriers verified present in code before being built on: sgl-project#297 kv_ladder_auto, sgl-project#485 uneven_perf, sgl-project#363 regime_act/regime_stages, sgl-project#261 scheduler:5063. Notably regime_stages ALREADY classifies REGIME_PREFILL_HEAVY/REGIME_DECODE_HEAVY, so R7 consumes that signal rather than adding a second phase detector -- a second detector would be a new source of disagreement about what phase it is, the same defect class as a second VRAM constant. Slices 2-4 (enumeration, objective+movement cost, phase-dependent ranking) are desk-provable in full; only slice 5 needs a card. Open and named, not glossed: sgl-project#551 elastic co-residency has NO marker in python/sglang/srt (only ANALYSE_553) and must be resolved before slice 5 plans to drive it; R4's objective is marked CONFIRM because throughput and latency can disagree and the design must not pick for the user; re-cut cost is not measurable today and slice 3 must measure it or make it a fingerprinted calibrated term, never a literal; the pressure ladder's autonomy overlaps the planner's authority and slice 5 must resolve it or the two will fight over the same pool under load.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
… objective Four coordinator answers folded in; two of them close holes in the first draft. R9 (NEW, user 2026-08-05) -- every prefill/draft/decode(verify) cut is itself a load change, and an ENFORCER must actually cause a different layout per load. Flip granularity becomes a PRICED AXIS rather than a blanket rule: per knob, quantify the flip cost and classify by two checkable criteria. A knob is tick-flippable exactly when flipping it costs neither a CUDA-graph recapture nor a byte moved; everything else flips at regime boundaries with hysteresis. Grounded, not asserted: sgl-project#439 cold-expert compute assignment is verified VRAM-neutral in tree (expert_heat_migration.py:31 names "the same size before and after" as the sgl-project#439 sizing latch's invariant), which is precisely the zero-data-movement property that puts it on the tick side. Within a tick the layout is fixed by definition, so the right tools are overlap/behaviour adaptation (sgl-project#128/sgl-project#199, sgl-project#125, sgl-project#274, sgl-project#156) -- not lesser substitutes for flipping, but the correct instrument at that timescale. F9 is the falsifier for the silent error: a wrongly-regime-classified zero-move knob never demonstrates the gain it could have made, so nothing ever surfaces it. F9b guards the opposite mistake (a 3-6 s recapture at tick granularity would spend the whole tick budget on recapture). F9c asserts on ACTUATED state, not on a logged decision, because sgl-project#578 is exactly the case where the decision existed and the actuation did not. SLICE 0 (sgl-project#578) -- verified in tree, not taken on report: regime_runtime.py:911 calls planner_candidates(server_args) without solve_fn, so regime_stages.py:383 takes the None branch, returns [], and the stage table permanently holds the booted stage alone. The docstring at regime_stages.py:357 says it outright -- "it is not broken, it is unfed". The enforcer R9(i) requires therefore EXISTS AS CODE AND ACTUATES NEVER, which made every actuated requirement in the first draft unreachable. Promoted from unknown risk to a named prerequisite slice. The work is wiring plus a SolverAnswer->Stage mapping (key_solver.solve already exists and is objective-aware), but it needs plan_inputs, base plan, per-rank budgets and RigRates -- i.e. a card probe. So slice 0 shares TICKET_582 gate (a) as a prerequisite and should be scheduled into the same window; recorded. R4 -- answered, [CONFIRM] removed. The objective is JOINT (maximum compute AND maximum VRAM throughput/latency at all times), expressed as two axes rather than collapsed into one scalar on the user's behalf. Dominance on disagreement is decided by the running task's phase: decode is memory-bound so bandwidth IS throughput there, prefill is compute-bound. The weights come from the same regime signal R7 consumes. F4 is a pair for the same reason as F7: a fixed scalar objective passes exactly one of the two regimes. sgl-project#551 -- resolved as in-tree sgl-project#553, and my find is confirmed: DESIGN/ANALYSE only, no mechanism. Slice 5 therefore BUILDS elastic co-residency as the planner's tenant-event actuator. Stated plainly in the slice plan and §5 so the plan cannot be misread as driving something that exists. Also recorded: R9's knob inventory is incomplete by construction (only the knobs the addendum named are classified); a full sweep with a cost per knob is part of slice 4b, since an unclassified knob is an unpriced assumption and R9 forbids those. And "3-6 s" for recapture is an observed range, not a model -- R5's movement-cost work must cover it too.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…parked carrier NOT WIRED YET, AND SAID SO UP FRONT. This is the pure decision core with its pins; the scheduler wiring (park at the finished-prefill boundary, re-admit at TP entry, feed the slot bound into get_num_allocatable_reqs) is a separate change and this module currently has no caller. A module nothing calls measures nothing -- that is why it is labelled rather than claimed. THE WEDGE WAS A COUNTING DEFECT. At 2026-08-16 06:04 the instance held twelve GDN slots with EIGHT FREE, four running against a cap of four, and 403779 tokens of prefill it could not admit. Freeing a GDN slot would have relieved nothing: admission is `min(pp_max_micro_batch_size, limiter.current) - running_bs` then `min(..., req_to_token_pool.available_size())`, and HybridReqToTokenPool does not override available_size, so that second term is the REQUEST-slot count. Neither term sees the GDN pool at all; the mamba allocator is consulted only later, inside alloc_req_slots. What blocked admission was four requests PP is FORBIDDEN to decode being counted against the concurrency cap for the whole residency. NOTHING MOVES IN PHASE 1. The carrier keeps its GDN slot, its KV -- exactly the KV that would have been resident anyway -- and its req_to_token row. That is deliberate: no state movement means no new correctness surface from the sgl-project#450/sgl-project#444 verify-write family, the sgl-project#461 DEVICE_BOUND law, or sgl-project#551 GDN-Vacate x kvso. The blob park is phase 2, gated behind the sgl-project#551 read. BOTH BOUNDS ARE SOLVED FROM BOOT DIMENSIONING and phase 1 raises nothing: parked + running <= the GDN slot pool (12), refused EARLY and by name because alloc_req_slots would refuse it late anyway; and running_bs <= max_running (4) at all times including TP, with re-admission in capture-set-sized batches so every pool stays inside what it was built for. THE ONE BOOKKEEPING EDGE is that a parked request is still RESIDENT -- `resident_ids` exists for the pressure ladder and retract paths, so a parked carrier is neither double-counted (it is out of running_batch) nor invisible (it is in the resident set). A PARK FAILURE DEGRADES TO THE SAFETY NET, pinned: disabled or slot-pool-full, the arithmetic is byte-identical to the pre-change gate, the request stays a carrier and keeps counting, and the sgl-project#677 progress exit still breaks the stall. The failure mode is the behaviour it replaces, never a wedge. A HAZARD THE WIRING MUST ANSWER, found while building this and recorded here rather than discovered on metal: a parked request is outside running_batch, so sgl-project#682's `harvest_resident_batches` and the seam's KV reshard would not see it. Phase 1 keeps its KV, so the seam must still carry it -- `resident_ids` is the hook for that, and the wiring change has to use it or a parked request's KV is left behind in the PP layout at the cutover. 16 hermetic cases, red-first: the 06:04 scenario with four carriers parked and the fifth prefill admitting; a control stating the composition exactly (this is an ADDITIONAL slot-pool bound, not a replacement for the caller's gate); parked+running never exceeding 12; TP re-admission never exceeding 4 and FIFO so no carrier starves; both receipts naming id, set size and binding bound; the disabled path reproducing the old gate; and evacuate() handing every parked request back so a crash strands none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes #549 and upgrades the Outlines version to 0.0.44.
\x00issue in byte level fsm resulting inKeyErrorinBetterFSM::FSMInfodottxt-ai/outlines#930