Repository navigation
Add SSL Cert Functionality - #224
Merged
Merged
Conversation
Closed
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
EdwardXuy
pushed a commit
to shun8686/sglang
that referenced
this pull request
Apr 6, 2026
update memory_and_scheduling & optimization_debug_options & runtime_options testcases
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…ding holders Kills the TP=4 A3 remove_hbm_not_device_leaf reject storm (1384/cycle, 187 hot units re-failing ~15x). Root cause is persistent TOCTOU, not a mis-identification: a session-tail leaf is a device-leaf *at dump time* only in the brief gap between an active program's forward passes (its lock_ref oscillates per pass; the dump samples a released instant). By apply time the state is up to ~840ms stale and the holder has re-locked its tail, so sglang rejects. Fix: gate on the program-level signal, which is STABLE across the per-pass oscillation. _holder_actively_decoding(u, state) returns True iff any holder pid has T26 per_program_usage[pid].hbm.inflight > 0 (a request in the running batch). migrate_candidates skips remove-HBM for such units, after the existing sgl-project#210 structural leaf guards. Node-level lock_ref is the wrong signal (0 at the dump instant). Tool-parked holders (inflight empty) keep their idle tail evictable -- the core demote-during-the-tool-gap relief. Absent/cold-start inflight never suppresses. Layers cleanly with sgl-project#223 (reactive post-reject cooldown) and sgl-project#210 (structural leaf guards); sgl-project#224 is the proactive complement. Audit (sgl-project#225 follow-ups): holders come from add-only session_ids, so the guard slightly over-approximates (safe direction -- forgoes a valid demote, never proposes an invalid one); bounded small under temp-0 linear agentic decode. Re-run must confirm migrate_applied>0 + HBM occ drops, not merely that rejects fell to zero. Tests: verify/joint_decide A-inflight (active blocked, parked/zero/ no-hbm/cold preserved, shared + multi-subpool blocked). DESIGN.md §7/§9 apply-feasibility pre-filter subsection. All pure suites green.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
… — un-clog the hints flood Root cause of the residual migrate reject rate (post-sgl-project#224): the daemon pushes a hints PUT every event (T40), so hints are ~99% of outbound traffic (~11.7k vs ~130 migrate/cycle). The outbound channel is single-flight at sglang, so every time-sensitive migrate waited ~1.8s behind that idempotent flood in the FIFO queue, ageing until the radix tree diverged and the apply-time leaf check rejected it. Transport is not the bottleneck (localhost sub-ms) and daemon-side concurrency does not help (sglang serialises apply via one communicator) -- the fix is to stop the flood from blocking migrates. sgl-project#228 (latency, the relief lever): the worker drains the queued burst each wake and emits at most ONE dispatch per endpoint by temporal semantics -- hints merge to one PUT (highest stamp per hash; sglang is itself overwrite-by-stamp), migrates stay individual (rare; FIFO + per-POST escalation preserved), program_paused coalesced by pid. Order: program_paused -> migrate -> hints, so time-sensitive intents never queue behind the flood. current_oldest_pending_age_ms() now also counts the in-flight drained burst so /health backlog stays accurate. sgl-project#227 (correctness bound, generous): a migrate decided on a state snapshot is a prediction with a validity horizon; the worker drops one aged past AGINFER_MIGRATE_FRESHNESS_MS (default 30s -- a pathological- spike floor, NOT a tight knob; the coalescing removes normal-operation latency). Proactive complement to the reactive APPLY_FAILED re-issue. Latency proof (verify/outbound_coalesce B0/B1): with 200 hints queued ahead of a migrate, the migrate is dispatch call #0 (not sgl-project#201) and the 200 hints collapse to 1 PUT; wall-clock migrate latency mean 0.2ms over 3 trials vs the N*apply serial floor of 1000ms. Audit (no Critical/Major): hardened hints merge to max-stamp (order- independent); documented that the sgl-project#164 stall backstop relies on the always-on hints canary (a stale-dropped migrate emits no failure signal). task_done accounting, escalation, /health live-peek all preserved (t164/t36 green). Tests: new verify/outbound_coalesce (7 stages: coalesce correctness + latency proof). 17/17 touched suites green (t36 t164 t41 t187 t30 t22 t24 t23_t37 kv_scheduler_value_rule joint_decide admission_controller t34 t16 dump_leaf_flags program_tracker t21). DESIGN §6 updated. verify/t40 left as-is (pre-existing stub staleness, tracked sgl-project#229).
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…ler interface Pre-existing breakage at HEAD (not from sgl-project#224/sgl-project#227/sgl-project#228): the t40 stub policy/router lagged the kv_scheduler.handle() interface that evolved with the §9 joint_decide wiring. * _DeclinePolicy / _MigratePolicy: add costs=default_costs() + pi_u (handle() reads policy.costs/policy.pi_u for the §7 migrate candidate generator; real OursGreedyPolicy carries both). * _FakeRouter: add theta_hi/theta_lo (admission thresholds) + heartbeat_s (§8 forecast horizon). * B1b "hints alongside a migrate": post-sgl-project#194 the migrate comes from value-gated joint_decide, not the policy Action — so drive genuine HBM pressure (95% occ + h_max>0) and use MEMORY_PRESSURE, NOT TOOL_CALL_END (whose sgl-project#223 reuse-imminent carve-out suppresses the tail demote). _state_json gains optional pressure/h_max knobs (defaults preserve prior behavior). Production unaffected (real policy/router always had these). T40 17/17 green (1 e2e skipped).
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…oject#208) Real-stack do-no-harm verdict for the sgl-project#224+sgl-project#227+sgl-project#228 fix stack on TP=4 A3 (harbor/terminus-2 swebenchpro). Per-task agent_execution time (the LLM serving phase the daemon affects; docker env-setup + verifier excluded): a3 (ours) n=74 mean=734.2s median=681.2s std=296.5s a3_kvoff n=81 mean=695.2s median=628.2s std=275.6s Delta = +5.6% mean, NOT significant (std ~290s >> 39s gap) -> HOLDS No fatals/crashes, identical task outcomes (reward 0, 12 broken-env both arms). Benefit absent at saturation (expected per sgl-project#230 Tier-1). The cyc2 80-reject burst (migr=155) is the pre-migrate-coalescing single-flight wall that sgl-project#228-complete (f88c885) fixes; rejects were do-no-harm regardless.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 18, 2026
…hain (demote now dispatches) The whole-chain demote was proposed by migrate_candidates but joint_decide still declined it live (29x policy_declined) while the SAME code on the post-run state produced candidates. Cause: at the TOOL_CALL_START instant the just-finished turn's per-program inflight hasn't cleared in the /aginfer/state snapshot, so the sgl-project#224 _holder_actively_decoding gate suppressed the remove-HBM. But the event AUTHORITATIVELY says the holder is now tool-parked for the ETA — a parked unit won't re-lock (no evict-storm), so the lagging inflight is the wrong signal. Bypass the gate for promote_pending units. Result: tool_call_start now dispatches 29x (migrates=7..11 per decision = whole chains), occ=0.99 gate fires, migrate_candidates yields the 4.48MB prefix demote. REMAINING (next): migrate_applied still 0 — sglang rejects 736x remove_hbm_not_device_leaf. The deepest-first peel-sort orders correctly but a SINGLE apply_aginfer_migrations batch does not realize each remove's tree-mutation before the next action (deferred eviction), so parents never become device-leaves within the batch. The probe proved SEPARATE POSTs peel correctly. Fix = flush the unlink within the batch (apply-side) or dispatch the chain as sequential batches.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 18, 2026
…ppend) Audit per review: the demote<->promote coupling + whole-chain demote did not only ADD — they made three EXISTING passages incomplete/contradictory. Fixed in place: - _value() formula (the canonical V_u): it charged the candidate tier's load_back unconditionally; now carries the reuse_tier=HBM-if-promote_pending override inline (was only in an appended paragraph 380 lines away). - Structural leaf guard: stated as absolute per-unit; now documents the promote_pending whole-chain exception (propose all chain units, peel leaf-inward, dispatch as ONE coordinated group — the selector must not cherry-pick the bulk). - sgl-project#224 active-decode guard: claimed tool-parked => not gated, ignoring the TOOL_CALL_START inflight snapshot lag; now documents the promote_pending bypass (the event is authoritative over the lagging metric). Nothing was outright FALSE => no deletions, but all three needed editing, which the first additive pass skipped.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 2, 2026
Desk design only, no cards (CUDA_VISIBLE_DEVICES=99), no implementation. Verdict: sgl-project#407 is not a new abstraction, it is a NODE layer plus a reconciliation. The edge half — what does it cost to move a byte from here to there — already exists FOUR times with four readers and four absent-value rules: cost_model.PairMatrix (#348b, UUID-keyed), barlink_path_rates. PathProfile (sgl-project#279, affine per route), offload_movement.PeerPathCapability (sgl-project#286, CUDA-ordinal-keyed) and card_probe h2d/d2h (sgl-project#271). E2 and E3 both want the effective BAR1 aperture from p2p_readiness/capability_matrix.json and only E2 has a loader. What exists nowhere is the node layer, and in its place the tree carries four disjoint tier vocabularies (PARK_TARGETS sgl-project#286; LOCAL_DESTINATION/SUPPORTED_PARK_BACKENDS sgl-project#224; StorageMedium telemetry; HiCache choices=). Expert offload, the largest byte mover in the fork, is in none of them: its tier is device="cpu" + .pin_memory() written out five times. Two shipped contradictions the design bends around, both stated up front: C1 — two opposite total orders over one axis. sgl-project#224's destinations_error makes 'local' first a hard validation law ("a remote tier is a tier BELOW local, never above it"); sgl-project#286's ladder puts peer_vram ABOVE host_ram. Resolved by making the order a shortest path over a STAGING GRAPH: 'local first' is recovered as reachability (the only edge out of a device tier toward a blob tier runs through host), peer_vram is recovered because vram->vram is a direct edge. Neither consumer's shipped behaviour changes. C2 — the one existing tier ledger is positionally keyed. CapacityLedger keys peer-VRAM budgets by CUDA ordinal; AUDIT_331 calls that SCOPED and legitimate until it crosses a process boundary, which is exactly what sgl-project#305 §6 ("one ledger, no second accounting") asks for. On this rig the 5090 is CUDA ordinal 0 and NVML index 1, so the failure is silent card-swapping. Honesty correction on three numbers the brief assumed measured: the peer-VRAM 1-3 us posted-write class is an explicit ASSUMPTION (EVAL_p2p §"Tragende Annahmen"), NVMe latency does not exist in the tree in any unit, and host DRAM 38 GB/s is derived from an ASSUMED DDR4-3200 peak (the VM hides SPD). They ship as ESTIMATE/ABSENT under #348b's measured-or-named rule. Everything else in the inventory is sourced: 5090 32607 MiB / 3080 20480 MiB, membw 1558/723 GB/s, BAR1 256 MiB nominal but 96 MiB contiguous with 1.13-1.34x vs NCCL at 20 KiB-16 MiB, H2D 6.4/13/13 GB/s gen4 x4/x8/x8, NVMe 1.8 GB/s cold (ARC rows excluded), rig-2 2.07-2.83 GB/s with 1.47 us at 8 B and ~89/202 us serving collectives. Consumer survey, eight consumers read in their target-selection code: sgl-project#77/sgl-project#123 expert offload MECHANICAL for the UUID-vector half (it revives sgl-project#394's merged-but-starved link-proportional sharding, 145 -> 86 ms/token, 1.69x) and REBUILD for a third rung (two dicts, binary resident|spill, UVA aliasing); HiCache MECHANICAL for backends, ladder EXCLUDED (it is the type signature of every controller method, not configuration); sgl-project#224 MECHANICAL for the list, REBUILD for the ordering law, victim half untouched; sgl-project#89 MECHANICAL-PLUS (needs a persistence class or a tmpfs --hibernate-dir is a silent regression, and a driver-free arg-parse-time query); sgl-project#305 NO MIGRATION, it is the awaited mechanism; sgl-project#286 MECHANICAL and the donor — lift _select_target and CapacityLedger rather than write new ones; sgl-project#389 and sgl-project#306 greenfield. Cut plan, no big bang: 1 registry + read-only enumeration + GateRow dashboard row (S); 2 publish the rank->card-UUID vector without a collective so sgl-project#394 comes alive (S, the only cut whose yield is already measured); 3 sgl-project#224 consumes it (S-M); 4 CapacityLedger becomes a view of registry/ledger.py, keys widen to TierId (M); 5-9 expert offload, hibernate, sgl-project#305 transitions, HiCache backends, NVMe/sgl-project#306 declaration. Exclusions with hard reasons only: the HiCache ladder, GDN state (DEVICE_BOUND_ONLY admission, a refusal not an omission), cross-rig GPU-to-GPU (driver guard, sgl-project#280 closed in userspace), compute placement (sgl-project#302). Measurement plan names six gaps, five of them S, three of them new arms in the existing comm-suite which today has no BAR1, storage or host-DRAM arm. codespell clean; ruff not applicable (documentation only).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 2, 2026
… consumers yet Charter: every memory you have access to must be a level cache, a spill target or an offload target depending on volatility -- disk, RAM, VRAM, local as well as remote. Today four disjoint tier vocabularies coexist (sgl-project#286's PARK_TARGETS, sgl-project#224's destination chain, StorageMedium telemetry, HiCache's choices= list) and the largest byte mover in the fork, expert offload, is in none of them. This is the node layer they can share. The edge layer (what a byte costs between two places) is NOT re-implemented: planner.cost_model keeps it. python/sglang/srt/memtier/, six modules, no consumer touched, no write outside a test's temp directory: * tiers.py -- TierId grammar (UUID or a named absence of one; a bare CUDA ordinal is refused at the door, naming C2: on this rig the 5090 is CUDA 0 and NVML 1, so a positional key silently swaps cards across a process boundary), the record, and the volatility law. Admission is a REFUSAL, not a ranking: a registry that only ordered tiers by cost would happily offer the GDN pool a remote tier, and recurrent state that made a lossy round trip is a correctness failure. Two enums, tier-side and payload-side, mapped by one table -- which is what makes the falsifier expressible. * profile.py + profiles/rig1.json -- the measured numbers as DATA for exactly one machine, every rate carrying the probe it came off. Device tiers bind from card-MODEL templates when a live card is enumerated, so no UUID is hardcoded and an unprofiled model gets live capacity with ABSENT caps rather than a roofline (#348b D4 one layer down). Overlay merges by tier id. * registry.py -- enumerate (blocked tiers included; omission is how a spill target silently becomes a different spill target), filter by volatility, object class, bandwidth floor, capability property and capacity, and a NAMED refusal for every tier that did not make the list. The ordering key is public so cut 3 can swap in the staging-graph path cost visibly. * reservations.py -- one ledger, no second accounting (DESIGN_305 §6). A tier reservation is a named post in the sgl-project#260/sgl-project#400 Posten sense; VramLedgerHook forwards device tiers into registry/ledger.py so the corridor, lease and invariant stay where they are. * probe.py -- DESIGN_407 §4's M1-M8 as records, every arm declared and every arm refusing. apply_outcome is the only writer of a measured cap. The three honesty corrections ship as the design demanded: host DRAM bandwidth is an ESTIMATE (its 51.2 GB/s reference peak is an ASSUMED DDR4-3200 figure -- dmidecode is empty under this VM), the peer-VRAM "1-3 us posted write" class is ABSENT because it is an assumption and not a probe, and NVMe latency is ABSENT because it does not exist in this tree in any unit. M1 also records the harness it CANNOT use: barlink has no send/recv, so the BAR1 ladder needs p2p_readiness. Deviations from the design, all recorded in its new §8: aperture is a Rate not an Optional[int] (an int cannot say why it is missing); vram:unenumerated@<host> added to the grammar so rig-2's cards are declared-and-blocked rather than invented; no staging graph and no GET route (both cut 3 / a consumer). Tests: 82 hermetic cases in test/registered/unit/memtier/, base-a-test-cpu, no driver and no network -- 28 records/registry, 22 profile, 14 reservations, 18 probe. All green. collect_local_facts is executed for real against /proc/meminfo and statvfs, and VramLedgerHook is driven against a real ReservationStore in a temp directory, so neither path is desk-written. Can-fail proof, 12 mutations, each turning exactly its guard red: EXPENSIVE_OK admitting persistence; a CUDA ordinal accepted as a card key; the provenance rank dropped from the ordering key (an estimate 21x faster than a measurement then wins); an absent bandwidth read as zero; an unknown headroom read as unlimited; an unnamed ledger post; a profile rate with no source; a roofline substituted for an unprofiled card; a failed probe delivering its value; an estimate entering through a probe; a measurement silently overwritten; blocked tiers omitted from the enumeration. ruff / codespell / black / isort clean; mypy clean on the package. The test_module_state_ratchet failure on this branch is pre-existing (layers/moe/utils.py _barlink_overlap_announced) and untouched here. Desk work, no cards (CUDA_VISIBLE_DEVICES=99). Next: cut 2, publish the rank -> card-UUID vector so sgl-project#394's merged link-proportional sharding comes alive -- the only cut in the plan whose yield is already measured (145 -> 86 ms/token on the cold tier, 1.69x).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…m a source that can lie TWO INDEPENDENT DEFECTS, both rooted in reading a number or a set from the wrong place. (A) THE GDN BLOB THAT GOT THROWN AWAY. build_gdn_slot_executor's idle_holders closure enumerated live offload sessions through kv_session_offload.spills. _commit_park POPS a sgl-project#224-parked session out of spills while the session stays alive and keeps its req-pool slot, its radix tree lock and its Mamba/GDN state slot. GdnSlotRuntime.on_round treats a parked id it cannot find in the inventory as DEAD and calls forget(sid) on it. So: a session spills, the ladder exports its GDN state to a blob, the session parks its KV to the next tier, and the next ladder round discards the blob. It later unparks, restores its KV, resumes -- and continues with recurrent state that no longer exists. No crash, no log, wrong tokens. The comment ("a parked session that died while parked leaves a blob nobody will ever ask for") was not wrong; its PREMISE was. Absence from the inventory meant "dead" only while every live session was in spills. Fixed with KVSessionOffloadManager.live_offload_reqs() -- spilled AND parked -- as the single enumeration, plus kv_session_spill_destination.parked_reqs() as the one place that knows where parked requests live (inflight_batches now goes through it too, so the abort scan and the ladder share one answer). Note the abort scan had ALREADY been taught about parked sessions; the knowledge existed in the module and the second consumer simply never received it, which is what a missing accessor costs. (B) THE HOST-RAM NUMBER. memtier/profile._host_memory_bytes read /proc/meminfo only, and the kvso host-pool guard sized a PINNED pool against psutil. Inside this container both are lxcfs-synthesised and do not describe what the process may have: MemAvailable can EXCEED MemTotal (an arithmetic impossibility, so the guard was comparing against a number that denotes nothing), and with memory.max unlimited they report the HOST's figures on a box other containers are also spending. Over-committing pinned memory is the OOM killer picking a victim, not a swap. honest_host_memory_bytes() is now the pure decision: a finite memory.max is the ceiling (capped by MemTotal), available is that ceiling minus anon and kernel -- page cache is reclaimable and deliberately NOT subtracted, since subtracting it would refuse boots that would have succeeded; with no finite limit MemTotal is the ceiling and MemAvailable the estimate, CLAMPED to it and clamped again by the cgroup's own resident accounting. Anything that cannot be established stays None rather than being guessed, and the kvso guard then SKIPS with a warning -- refusing to guess beats refusing a good boot on a fabricated figure. memtier owns this per DESIGN_407 §2.9 (eight ownerless psutil.virtual_memory() call sites; this module is meant to become the owner, not the ninth), so the guard reads host_memory_bytes_for_pinning() there rather than adding a ninth. This is also the honest number sgl-project#547's joint pinned-RAM budget will be built on. Tests (CUDA_VISIBLE_DEVICES=99, hermetic, no GPU, no server): PYTHONPATH=python pytest \ test/registered/unit/mem_cache/test_gdn_slot_runtime.py \ test/registered/unit/memtier test/registered/unit/server_args \ test/registered/unit/test_kv_spill_destination_unit.py \ test/registered/unit/test_kv_session_offload_unit.py \ test/registered/unit/test_kv_spill_budget_unit.py \ test/registered/unit/test_kvso_reclaim_decline_501.py -q -> 942 passed, 19 warnings, 103 subtests passed in 14.35s CAN-FAIL PROVEN, both halves: - pointing idle_holders back at manager.spills turns test_the_builder_sees_a_parked_session red with "Lists differ: ['s'] != ['p', 's']" -- the parked request missing from the inventory is exactly the state in which the ladder throws its blob away; - dropping the MemAvailable clamp turns test_the_impossible_meminfo_reading_is_clamped red with "134217728000 != 128849018880". The live-path test executes the real reader on this machine, so neither half is desk-written-never-executed. Every input to honest_host_memory_bytes is passed in, so no test asserts anything about the machine it runs on (directive sgl-project#434). FEATURE_CATALOG gains the absent-means-dead family entry. Handoff: /spinning/spill-night-20260804/handoffs/ TICKET_551_gdn_park_and_honest_ram.md -- half A needs a hybrid-model boot with --kv-session-offload-max-spills 1 (to force a park) and a binding --gdn-resident-state-slots (to force a vacate), and explicitly states that a run in which no park or no vacate happened proves nothing. ruff check: no new findings on any changed file. ruff format: clean on all added lines. codespell: clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 12, 2026
…registry, not asserted Cut 1. The memtier package (sgl-project#407) has been complete and INERT since it landed: audit sgl-project#421 found zero production importers, and its own __init__ still says "No consumer reads any of this yet". Nothing in a serving path ever constructed a TierRegistry. This makes the KV spill rung the first one that does. The rung spills to exactly one place, and the ORDER of any further sgl-project#224 park tier is a hardcoded rule: destinations_error() refuses a list whose first entry is not 'local', quoting a measurement in prose. That rule is right and it is the wrong KIND of statement -- a name check standing in for a capability check, with its evidence in a comment where nothing can re-derive it. sgl-project#407 says so itself, at the field that exists for this exact edge: TierTransport.stages_through: "sgl-project#224's `local` must be first law is exactly this edge ... recording it now is what lets that law stop being a hardcoded total order later." This is that later. kv_spill_tier_selection builds a registry whose FIRST tier is local host RAM and derives the ladder from measured capacity and cost. Local-first stops being asserted and becomes DERIVED -- and falsifiable: local_first_disagreement() reports when the hardcoded law and the measured ladder stop agreeing, and deliberately does NOT reorder, because the law encodes a physical constraint (a device D2H copy has nowhere else to land), so a cheaper-looking park tier is evidence of a bad number, not a reason. TierCapacity.reserved is populated from a live source for the first time. sgl-project#407 documents it as "bytes declared by other holders, read from the cross-process ledger" and nothing ever wrote it, so every headroom the registry could compute was the headroom of an empty machine. pinned_host_budget.registered_posts() IS that ledger -- the join is now made, and a second pinned pool correctly reduces this tier's headroom. Exhaustion is a NAMED refusal (the ledger half of sgl-project#659 b). A region shortfall is the moment the local tier is full, and it was recorded only as a pressure timestamp: the operator saw a decline with no statement of what was full or how full. The ladder now reports one line per tier with a provenance, renders ABSENCES with their reason rather than as a dash ("unknown" and "zero" lead to opposite decisions), and degrades to the next tier when one exists. BYTE-IDENTICAL by construction: in this cut the registry is an OBSERVER under the rung. The sgl-project#224 chain still picks the destination; nothing in the spill path changes. That is deliberate -- the registry's verdict has to be comparable against the hardcoded law before anything is allowed to depend on it. Also fixes a latent bug found by smoking the wiring rather than reading it: kv_session_host_bytes returns None (not a pair) when offload is off, and unpacking it directly raised inside the caller's blanket except -- an ordinary "no pool" presenting as an invisible skip. Tests: test_kv_spill_tier_selection_659, 13 tests, green. Every load-bearing assertion has a sibling arm producing the opposite verdict from the same function, so the falsifier is proven can-fail rather than asserted: - local_first_disagreement returns None on the measured ladder AND non-None when a park tier is given absurd caps (test_disagreement_is_reported_ rather_than_obeyed); - a full local tier declines alone, and degrades to tier 2 when present; - roomy vs crowded proves the reserved-from-posts join actually binds. Live smoke (desk code that never ran is unvalidated): the wiring executed and printed kv-spill ladder: host:CT999 (ram-local) headroom=0.61 GB [measured] bandwidth=absent (no D2H bandwidth probe recorded for this pool) against a pool of 8 regions x 100000 tokens x 1024 B with 2 occupied -- arithmetic correct, absence rendered, rate-limit holding on the second call. sgl-project#631 flip family: 1076 passed / 0 failed, inherited baseline unmoved. ruff + codespell clean. Ship config is untouched: it carries no --enable-kv-session-offload, so live_kv_spill_ladder returns None at its first line.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 12, 2026
… and the ladder now picks Cut 1 put the sgl-project#407 registry under the KV spill rung as an observer with one tier on it. This is the second rung and the moment the registry stops observing. The brief's preferred second tier does not exist on this rig: s41 measured Rig-2 RAM as unreachable (unroutable subnets, 75 MB/s on the only routable path, 8.6 GiB of swap-backed RAM against a ~12.9 GB region) and peer VRAM is closed by the standing REBALANCE verdict. The cheapest REAL second tier is a local filesystem, and it is registered as what it is: a tier, not a remote tier. Every metric on the entry is measured at registration against the directory the blobs actually take -- never read from memtier/profiles, whose remote rows describe a link this process cannot route to (register C24). Two measurement traps are avoided by construction and documented with their numbers: probing with zeros measured the ZFS compressor at 6.2 GB/s rather than the disk, and reading back a just-written file measures ARC rather than the medium. The probe writes incompressible bytes, ends in fsync, ranks on the WRITE rate, and records the read-back as the upper bound it is. The probe size is 256 MiB and the entry says why: a 64 MiB probe overstates this volume by 2.8x. The capacity is the operator's ceiling floored by df, in that order -- min(budget, free - headroom) -- so a generous budget on a nearly-full volume yields a small tier instead of a disk-full outage. Selection: the ladder now ranks the configured park tiers by measured bandwidth and refuses the rest by name, replacing tier_index=0. It is byte-identical wherever budgets allow -- with one healthy tier the ranking is degenerate and returns 0, which is what the hardcoded law returned -- and cut 1's falsifier still passes. Every input to that decision is rank-uniform, deliberately: the ask is a boot constant, the capacities were measured at registration rather than re-read, and the ranking is computed once and cached. There is no per-park try/except around the choice, because a rank that fell into one and parked, beside a rank that succeeded and declined, would diverge into the completion min-reduce and wedge the instance rather than error (register law 14). The sgl-project#224 park counters are read for the first time. They were complete and correct for three shifts while every reference in the tree was a write, so a park that failed and a park that never happened produced the same silence. They now feed the selection policy (a tier is charged for the parks it eats and is refused on health once it has eaten enough), the ladder's ledger line, and the dashboard, which gains a measured denominator for park occupancy. Tests: 17 new, each load-bearing assertion with a sibling arm producing the opposite verdict from the same function. sgl-project#631 flip family 1076 passed / 0 failed before and after. Adjacent kv-spill, spill-tier, tier-occupancy and hicache-bounds modules 112 passed. ruff and codespell clean on the new files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 12, 2026
…ep path asserts, and it is what blocked sgl-project#659's end-to-end proof
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…s re-checked at code A parallel trace landed after cb0097c and contradicted one of my own statements. Correcting it rather than leaving it, and recording three further findings it surfaced. Each re-verified at code here, not taken on report. THE CORRECTION. §9.1(c) said "beyond that first slot, order is by measured bandwidth, not by operator preference". Wrong. Below the local slot the operator's configured order in --kv-session-offload-destinations IS respected; a measurement may promote a tier above it only when that tier is at least PROMOTION_RATIO = 4.0 times faster (kv_spill_park_tier.py:134, applied by _park_ranking, kv_session_spill_destination.py:1259-1300), and that ratio is a module constant behind no flag and no env var. This makes sgl-project#224 MORE delivered than I credited: "Zielliste + Richtung konfigurierbar" is satisfied below a physically fixed first slot. What is not configurable is the promotion threshold that can override the operator. FOUR REFINEMENTS (§9.5): 1. The remote row cannot reach a ladder even if read. I wrote that the spill path "deliberately does not read" rig1.json's remote rows -- true, but it understates the guard. host:rig-2 carries health.reachable = false (rig1.json:164) and TierRegistry refuses on exactly that before any other rule (registry.py:353), with a named reason. The tier is enumerable and can never enter a select() candidate list. Structural, not a convention. 2. sgl-project#286's `remote` is an always-refusing stub and says so in its refusal: "remote: stub tier (sgl-project#224 RDMA attachment point, not wired yet)" (offload_movement.py:750). Recorded because sgl-project#286 and sgl-project#224 both say "remote" for different things -- sgl-project#286's is a placeholder in the expert-offload ladder, not the kvso park tier. 3. The sgl-project#286 registry consumer is unreachable from production. price_park_target is the function that actually calls registry.select(TierQuery(...)) and is annotated as the fork's first memtier consumer, but its only callers are in test_short_term_offload_register.py. memtier/consumers.py's expert_offload_host_targets is the same shape. So §9.1(d)'s importer list needed the qualification now added: seven modules import memtier, but only kv_spill_tier_selection / kv_session_spill_destination perform tier SELECTION on a live path. The sgl-project#421-F3 shape again, inside sgl-project#407's own migration. 4. A provenance defect the profile records against itself: rig1.json:172 warns that the 3.43 GB/s and 1.47 us figures quoted together in destinations_error come from DIFFERENT lines (100G and 40G). Those paired numbers appear in the --kv-session-offload-destinations help text as the justification for the local-first law. The LAW is not in doubt -- it is a D2H-staging argument, not a bandwidth one -- but an operator reading that help is reading a mixed pair. Not fixed here: it is help-text prose and the correct replacement needs the sgl-project#266 run's numbers re-read rather than invented. Named so the next edit does not re-copy it. No code change; docs only. Tests unaffected (the 8 pointer-I/O cases and the memtier suite are unchanged by this commit).
1 of 5 tasks
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
verifyparameter to the RuntimeEndpoint and modifies utils.py.can be used like
This doesn't modify any of the openai python interactions since we can just do this instead