Skip to content

Fix the Jump-Forward with Chinese - #551

Merged
hnyls2002 merged 1 commit into
mainfrom
fix-jf
Jun 16, 2024
Merged

hnyls2002 merged 1 commit into
mainfrom
fix-jf

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

This PR fixes #549 and upgrades the Outlines version to 0.0.44.

@hnyls2002
hnyls2002 merged commit ad5f04d into main Jun 16, 2024
@hnyls2002
hnyls2002 deleted the fix-jf branch June 16, 2024 13:45
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026
…taset is too low (sgl-project#551)

* fix the problem gemma accuracy task takes too long on one machine

* let the threshold of dataset aime to be -1

* fix the problem score for math problems too low
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…m a source that can lie

TWO INDEPENDENT DEFECTS, both rooted in reading a number or a set from the
wrong place.

(A) THE GDN BLOB THAT GOT THROWN AWAY. build_gdn_slot_executor's idle_holders
closure enumerated live offload sessions through kv_session_offload.spills.
_commit_park POPS a sgl-project#224-parked session out of spills while the session stays
alive and keeps its req-pool slot, its radix tree lock and its Mamba/GDN state
slot. GdnSlotRuntime.on_round treats a parked id it cannot find in the
inventory as DEAD and calls forget(sid) on it. So: a session spills, the ladder
exports its GDN state to a blob, the session parks its KV to the next tier, and
the next ladder round discards the blob. It later unparks, restores its KV,
resumes -- and continues with recurrent state that no longer exists. No crash,
no log, wrong tokens.

The comment ("a parked session that died while parked leaves a blob nobody will
ever ask for") was not wrong; its PREMISE was. Absence from the inventory meant
"dead" only while every live session was in spills.

Fixed with KVSessionOffloadManager.live_offload_reqs() -- spilled AND parked --
as the single enumeration, plus kv_session_spill_destination.parked_reqs() as
the one place that knows where parked requests live (inflight_batches now goes
through it too, so the abort scan and the ladder share one answer). Note the
abort scan had ALREADY been taught about parked sessions; the knowledge existed
in the module and the second consumer simply never received it, which is what a
missing accessor costs.

(B) THE HOST-RAM NUMBER. memtier/profile._host_memory_bytes read /proc/meminfo
only, and the kvso host-pool guard sized a PINNED pool against psutil. Inside
this container both are lxcfs-synthesised and do not describe what the process
may have: MemAvailable can EXCEED MemTotal (an arithmetic impossibility, so the
guard was comparing against a number that denotes nothing), and with memory.max
unlimited they report the HOST's figures on a box other containers are also
spending. Over-committing pinned memory is the OOM killer picking a victim, not
a swap.

honest_host_memory_bytes() is now the pure decision: a finite memory.max is the
ceiling (capped by MemTotal), available is that ceiling minus anon and kernel --
page cache is reclaimable and deliberately NOT subtracted, since subtracting it
would refuse boots that would have succeeded; with no finite limit MemTotal is
the ceiling and MemAvailable the estimate, CLAMPED to it and clamped again by
the cgroup's own resident accounting. Anything that cannot be established stays
None rather than being guessed, and the kvso guard then SKIPS with a warning --
refusing to guess beats refusing a good boot on a fabricated figure.

memtier owns this per DESIGN_407 §2.9 (eight ownerless psutil.virtual_memory()
call sites; this module is meant to become the owner, not the ninth), so the
guard reads host_memory_bytes_for_pinning() there rather than adding a ninth.
This is also the honest number sgl-project#547's joint pinned-RAM budget will be built on.

Tests (CUDA_VISIBLE_DEVICES=99, hermetic, no GPU, no server):
  PYTHONPATH=python pytest \
    test/registered/unit/mem_cache/test_gdn_slot_runtime.py \
    test/registered/unit/memtier test/registered/unit/server_args \
    test/registered/unit/test_kv_spill_destination_unit.py \
    test/registered/unit/test_kv_session_offload_unit.py \
    test/registered/unit/test_kv_spill_budget_unit.py \
    test/registered/unit/test_kvso_reclaim_decline_501.py -q
  -> 942 passed, 19 warnings, 103 subtests passed in 14.35s

CAN-FAIL PROVEN, both halves:
- pointing idle_holders back at manager.spills turns
  test_the_builder_sees_a_parked_session red with "Lists differ: ['s'] !=
  ['p', 's']" -- the parked request missing from the inventory is exactly the
  state in which the ladder throws its blob away;
- dropping the MemAvailable clamp turns
  test_the_impossible_meminfo_reading_is_clamped red with
  "134217728000 != 128849018880".
The live-path test executes the real reader on this machine, so neither half is
desk-written-never-executed.

Every input to honest_host_memory_bytes is passed in, so no test asserts
anything about the machine it runs on (directive sgl-project#434).

FEATURE_CATALOG gains the absent-means-dead family entry.
Handoff: /spinning/spill-night-20260804/handoffs/
TICKET_551_gdn_park_and_honest_ram.md -- half A needs a hybrid-model boot with
--kv-session-offload-max-spills 1 (to force a park) and a binding
--gdn-resident-state-slots (to force a vacate), and explicitly states that a run
in which no park or no vacate happened proves nothing.

ruff check: no new findings on any changed file. ruff format: clean on all
added lines. codespell: clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…and a five-cut list

Desk analysis only, no build, as scoped. docs/dev/ANALYSE_553_elastic_coresidence.md.

What the four named building blocks actually are, read from the code rather
than from their titles:

- sgl-project#330 VRAM dial: a REAL mechanism, and its decisive property for sgl-project#553 is that
  store_bound_rows is a LIFETIME bound baked into captured graphs. So the dial
  has a FREE BAND (shrink to any floor, re-grow to the boot bound) and a HARD
  WALL above it. The free band is exactly where CUDA-graph compatibility is
  preserved.
- sgl-project#364 GDN slot ladder: a real, executing hot/cold mover, and graph-neutral by
  construction (it moves state between slots, not geometry). It is the
  existence proof that a hot/cold event need not touch capture, and its
  inventory seam -- the owner enumerates its live set, consumers never read a
  container directly -- is the pattern the other consumers need (sgl-project#551).
- sgl-project#287 KV pressure ladder: PLANNER ONLY. Seventeen NotImplementedError sites;
  the docstring says so plainly. But the ordering invariants are already
  enforced in code and the relief rungs reference existing features BY NAME, so
  the relief half can be made to execute by delegation without new mechanism.
- sgl-project#286 register: the right vocabulary, but exactly one ledger exists and it is
  wired to the translator. There is no generic tenant to raise an event
  against.

The pattern across the event table: both directions are largely built, and
what is missing is ARBITRATION. Two registries exist (dial participants, sgl-project#286
asset classes) and neither can see the other, so "free 4 GiB for the video
tenant" has no addressee.

Cut list, ordered so each cut is provable alone and no cut before the last
touches capture: (1) one registry or an explicit bridge; (2) a generic tenant
mover factored out of AudioAssetLedger; (3) make sgl-project#287's RELIEF rungs execute by
delegation -- the largest block of stubs, and it never approaches the graph
wall because relief rungs are declared "no KV layout change"; (4) the
arbitration policy on top; (5) LAST and only with the rig, cross the graph wall
via the per-rung pre-captured graphs sgl-project#287 already specifies, with cold rungs
parked as register class graph_rungs.

Graph compatibility is treated as the target state throughout, per directive:
every cut states its effect on graph state, and cut 5 carries the explicit
route back to full coverage. Until cut 5 lands, sgl-project#553 must state its band --
"elastic within the boot capture bound" -- rather than implying unlimited
elasticity.

Instrumentation named honestly: SGLANG_KVSO_TICK_TRACE=1 (throttled per-session
regulator trace: effective interval, measured tick_cost, binding headroom
ratio, host-tail size; pure logging, no control effect). The analysis states
what it is NOT -- a regulator trace, not a per-phase breakdown of device time --
and proposes no measurement it has not checked exists.

One decision deliberately left open and named as such: whether sgl-project#553 owns the
arbitration policy or whether it belongs to sgl-project#363. That fork needs the sgl-project#363
stage feed bound first, since an arbitration policy fed by a table holding one
stage would have nothing to choose between -- the same failure sgl-project#363/S8
documents one level up.

codespell: clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…mposability fixes)

Desk strand (fix/spill-composability, included via 2a156c8): six bugs with
proven can-fail falsifiers (sgl-project#552 parked-abort leak, sgl-project#552 spec-costs-spill,
sgl-project#547 dead interop honesty, sgl-project#551 GDN blob loss + lxcfs-honest host RAM,
sgl-project#568 ledger non-persistent buffer corruption), S8 docstring fix, analyses
P6/P7/sgl-project#553. Full-sweep regression proof: failing-test sets byte-identical
vs base e08946d (911=911), +41 newly passing.

GPU strand: 8 matrix cells passed on hardware (kvso under uneven DCP,
partial spill, restore gate, kvso x NEXTN, multi-session, ladder actuation,
hibernate round trip 1.29x), matrix landed in-tree as
docs/dev/ANALYSE_spill_matrix_20260804.md.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
The night merge 2a156c8 was cut at b9a9fd6 while the desk strand kept
committing; this brings in the missing tail: sgl-project#551 GDN-ladder parked-session
fix + honest host RAM, sgl-project#568 generic ledger non-persistent-buffer fix, S8
stage-feed docstring fix + AUDIT_421 B.9, analyses sgl-project#553/P6/P7 and the
rejected-register entry. Restores the Absent-means-dead and
Round-trip-asymmetry catalog families the earlier merge missed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
Design only, no code. Records the mandate plus all three addenda as
requirements with falsifiers, and fixes the authority model: the planner
decides, the ledger is the denominator, every existing mechanism becomes a
sensor or an actuator. A slice that reimplements a carrier instead of driving
it has failed the mandate.

Addendum 3 becomes R6 and R7, both stated as error classes the design must be
structurally unable to commit rather than behaviours it should prefer:

R6 -- a state change re-SOLVES the cut (layer/TP split, draft placement,
KV/token vectors, phase layouts) under the new usage profile. Pool-only
squeezing is one candidate, not the search space. The planner may still choose
pool-only, but that must be its cost calculation under R5, never a structural
inability. F6 asserts on the CANDIDATE SET as well as the choice, because a
planner that picked correctly from a set of one is not implementing this; F6b
is the mirror, where pool-only wins on horizon and the planner must record
which re-cut it rejected and why.

R7 -- evacuation heat is PHASE-DEPENDENT, not static per item class. During
prefill everything decode-owned is cold and goes first; during decode the
ranking inverts. F7 is deliberately a PAIR (prefill-burst ranks decode graphs
below KV, decode-burst inverts) because a static ranking passes either case
alone. F7b adds imminence: at a prefill burst whose successor is decode, decode
graphs must NOT be evacuated -- separating 'phase-dependent' from 'reacts to
the instantaneous phase', which is a different and worse thing.

Carriers verified present in code before being built on: sgl-project#297 kv_ladder_auto,
sgl-project#485 uneven_perf, sgl-project#363 regime_act/regime_stages, sgl-project#261 scheduler:5063. Notably
regime_stages ALREADY classifies REGIME_PREFILL_HEAVY/REGIME_DECODE_HEAVY, so
R7 consumes that signal rather than adding a second phase detector -- a second
detector would be a new source of disagreement about what phase it is, the same
defect class as a second VRAM constant.

Slices 2-4 (enumeration, objective+movement cost, phase-dependent ranking) are
desk-provable in full; only slice 5 needs a card.

Open and named, not glossed: sgl-project#551 elastic co-residency has NO marker in
python/sglang/srt (only ANALYSE_553) and must be resolved before slice 5 plans
to drive it; R4's objective is marked CONFIRM because throughput and latency
can disagree and the design must not pick for the user; re-cut cost is not
measurable today and slice 3 must measure it or make it a fingerprinted
calibrated term, never a literal; the pressure ladder's autonomy overlaps the
planner's authority and slice 5 must resolve it or the two will fight over the
same pool under load.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
… objective

Four coordinator answers folded in; two of them close holes in the first draft.

R9 (NEW, user 2026-08-05) -- every prefill/draft/decode(verify) cut is itself a
load change, and an ENFORCER must actually cause a different layout per load.
Flip granularity becomes a PRICED AXIS rather than a blanket rule: per knob,
quantify the flip cost and classify by two checkable criteria. A knob is
tick-flippable exactly when flipping it costs neither a CUDA-graph recapture
nor a byte moved; everything else flips at regime boundaries with hysteresis.
Grounded, not asserted: sgl-project#439 cold-expert compute assignment is verified
VRAM-neutral in tree (expert_heat_migration.py:31 names "the same size before
and after" as the sgl-project#439 sizing latch's invariant), which is precisely the
zero-data-movement property that puts it on the tick side. Within a tick the
layout is fixed by definition, so the right tools are overlap/behaviour
adaptation (sgl-project#128/sgl-project#199, sgl-project#125, sgl-project#274, sgl-project#156) -- not lesser substitutes for
flipping, but the correct instrument at that timescale.

F9 is the falsifier for the silent error: a wrongly-regime-classified zero-move
knob never demonstrates the gain it could have made, so nothing ever surfaces
it. F9b guards the opposite mistake (a 3-6 s recapture at tick granularity
would spend the whole tick budget on recapture). F9c asserts on ACTUATED state,
not on a logged decision, because sgl-project#578 is exactly the case where the decision
existed and the actuation did not.

SLICE 0 (sgl-project#578) -- verified in tree, not taken on report: regime_runtime.py:911
calls planner_candidates(server_args) without solve_fn, so regime_stages.py:383
takes the None branch, returns [], and the stage table permanently holds the
booted stage alone. The docstring at regime_stages.py:357 says it outright --
"it is not broken, it is unfed". The enforcer R9(i) requires therefore EXISTS
AS CODE AND ACTUATES NEVER, which made every actuated requirement in the first
draft unreachable. Promoted from unknown risk to a named prerequisite slice.
The work is wiring plus a SolverAnswer->Stage mapping (key_solver.solve already
exists and is objective-aware), but it needs plan_inputs, base plan, per-rank
budgets and RigRates -- i.e. a card probe. So slice 0 shares TICKET_582 gate
(a) as a prerequisite and should be scheduled into the same window; recorded.

R4 -- answered, [CONFIRM] removed. The objective is JOINT (maximum compute AND
maximum VRAM throughput/latency at all times), expressed as two axes rather
than collapsed into one scalar on the user's behalf. Dominance on disagreement
is decided by the running task's phase: decode is memory-bound so bandwidth IS
throughput there, prefill is compute-bound. The weights come from the same
regime signal R7 consumes. F4 is a pair for the same reason as F7: a fixed
scalar objective passes exactly one of the two regimes.

sgl-project#551 -- resolved as in-tree sgl-project#553, and my find is confirmed: DESIGN/ANALYSE
only, no mechanism. Slice 5 therefore BUILDS elastic co-residency as the
planner's tenant-event actuator. Stated plainly in the slice plan and §5 so the
plan cannot be misread as driving something that exists.

Also recorded: R9's knob inventory is incomplete by construction (only the
knobs the addendum named are classified); a full sweep with a cost per knob is
part of slice 4b, since an unclassified knob is an unpriced assumption and R9
forbids those. And "3-6 s" for recapture is an observed range, not a model --
R5's movement-cost work must cover it too.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…parked carrier

NOT WIRED YET, AND SAID SO UP FRONT. This is the pure decision core with its
pins; the scheduler wiring (park at the finished-prefill boundary, re-admit at
TP entry, feed the slot bound into get_num_allocatable_reqs) is a separate
change and this module currently has no caller. A module nothing calls measures
nothing -- that is why it is labelled rather than claimed.

THE WEDGE WAS A COUNTING DEFECT. At 2026-08-16 06:04 the instance held twelve
GDN slots with EIGHT FREE, four running against a cap of four, and 403779
tokens of prefill it could not admit. Freeing a GDN slot would have relieved
nothing: admission is `min(pp_max_micro_batch_size, limiter.current) -
running_bs` then `min(..., req_to_token_pool.available_size())`, and
HybridReqToTokenPool does not override available_size, so that second term is
the REQUEST-slot count. Neither term sees the GDN pool at all; the mamba
allocator is consulted only later, inside alloc_req_slots. What blocked
admission was four requests PP is FORBIDDEN to decode being counted against
the concurrency cap for the whole residency.

NOTHING MOVES IN PHASE 1. The carrier keeps its GDN slot, its KV -- exactly
the KV that would have been resident anyway -- and its req_to_token row. That
is deliberate: no state movement means no new correctness surface from the
sgl-project#450/sgl-project#444 verify-write family, the sgl-project#461 DEVICE_BOUND law, or sgl-project#551
GDN-Vacate x kvso. The blob park is phase 2, gated behind the sgl-project#551 read.

BOTH BOUNDS ARE SOLVED FROM BOOT DIMENSIONING and phase 1 raises nothing:
parked + running <= the GDN slot pool (12), refused EARLY and by name because
alloc_req_slots would refuse it late anyway; and running_bs <= max_running (4)
at all times including TP, with re-admission in capture-set-sized batches so
every pool stays inside what it was built for.

THE ONE BOOKKEEPING EDGE is that a parked request is still RESIDENT --
`resident_ids` exists for the pressure ladder and retract paths, so a parked
carrier is neither double-counted (it is out of running_batch) nor invisible
(it is in the resident set).

A PARK FAILURE DEGRADES TO THE SAFETY NET, pinned: disabled or slot-pool-full,
the arithmetic is byte-identical to the pre-change gate, the request stays a
carrier and keeps counting, and the sgl-project#677 progress exit still breaks the stall.
The failure mode is the behaviour it replaces, never a wedge.

A HAZARD THE WIRING MUST ANSWER, found while building this and recorded here
rather than discovered on metal: a parked request is outside running_batch, so
sgl-project#682's `harvest_resident_batches` and the seam's KV reshard would not see it.
Phase 1 keeps its KV, so the seam must still carry it -- `resident_ids` is the
hook for that, and the wiring change has to use it or a parked request's KV is
left behind in the PP layout at the cutover.

16 hermetic cases, red-first: the 06:04 scenario with four carriers parked and
the fifth prefill admitting; a control stating the composition exactly (this is
an ADDITIONAL slot-pool bound, not a replacement for the caller's gate);
parked+running never exceeding 12; TP re-admission never exceeding 4 and FIFO
so no carrier starves; both receipts naming id, set size and binding bound;
the disabled path reproducing the old gate; and evacuate() handing every parked
request back so a crash strands none.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chinese Regex BUG in req.jump_forward_map.jump_forward_byte

1 participant