Skip to content

Add SSL Cert Functionality - #224

Merged
hnyls2002 merged 6 commits into
sgl-project:mainfrom
nivibilla:main
Mar 3, 2024
Merged

hnyls2002 merged 6 commits into
sgl-project:mainfrom
nivibilla:main

Conversation

@nivibilla

Copy link
Copy Markdown
Contributor

Adds a verify parameter to the RuntimeEndpoint and modifies utils.py.

can be used like

runtime = sgl.RuntimeEndpoint(url, verify='./ca_bundle.crt')

This doesn't modify any of the openai python interactions since we can just do this instead

import httpx
import openai
client = openai.Client(
    base_url=url + "/v1",
    http_client = httpx.Client(verify='./ca_bundle.crt'),
    api_key='none'
    )

# Chat completion
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)

@nivibilla nivibilla mentioned this pull request Feb 23, 2024
@hnyls2002
hnyls2002 merged commit 01b07ea into sgl-project:main Mar 3, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
EdwardXuy pushed a commit to shun8686/sglang that referenced this pull request Apr 6, 2026
update memory_and_scheduling & optimization_debug_options & runtime_options testcases
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 9, 2026
…ding holders

Kills the TP=4 A3 remove_hbm_not_device_leaf reject storm (1384/cycle,
187 hot units re-failing ~15x). Root cause is persistent TOCTOU, not a
mis-identification: a session-tail leaf is a device-leaf *at dump time*
only in the brief gap between an active program's forward passes (its
lock_ref oscillates per pass; the dump samples a released instant). By
apply time the state is up to ~840ms stale and the holder has re-locked
its tail, so sglang rejects.

Fix: gate on the program-level signal, which is STABLE across the
per-pass oscillation. _holder_actively_decoding(u, state) returns True
iff any holder pid has T26 per_program_usage[pid].hbm.inflight > 0 (a
request in the running batch). migrate_candidates skips remove-HBM for
such units, after the existing sgl-project#210 structural leaf guards. Node-level
lock_ref is the wrong signal (0 at the dump instant). Tool-parked
holders (inflight empty) keep their idle tail evictable -- the core
demote-during-the-tool-gap relief. Absent/cold-start inflight never
suppresses.

Layers cleanly with sgl-project#223 (reactive post-reject cooldown) and sgl-project#210
(structural leaf guards); sgl-project#224 is the proactive complement.

Audit (sgl-project#225 follow-ups): holders come from add-only session_ids, so the
guard slightly over-approximates (safe direction -- forgoes a valid
demote, never proposes an invalid one); bounded small under temp-0
linear agentic decode. Re-run must confirm migrate_applied>0 + HBM occ
drops, not merely that rejects fell to zero.

Tests: verify/joint_decide A-inflight (active blocked, parked/zero/
no-hbm/cold preserved, shared + multi-subpool blocked). DESIGN.md
§7/§9 apply-feasibility pre-filter subsection. All pure suites green.
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 9, 2026
… — un-clog the hints flood

Root cause of the residual migrate reject rate (post-sgl-project#224): the daemon
pushes a hints PUT every event (T40), so hints are ~99% of outbound
traffic (~11.7k vs ~130 migrate/cycle). The outbound channel is
single-flight at sglang, so every time-sensitive migrate waited ~1.8s
behind that idempotent flood in the FIFO queue, ageing until the radix
tree diverged and the apply-time leaf check rejected it. Transport is
not the bottleneck (localhost sub-ms) and daemon-side concurrency does
not help (sglang serialises apply via one communicator) -- the fix is
to stop the flood from blocking migrates.

sgl-project#228 (latency, the relief lever): the worker drains the queued burst
each wake and emits at most ONE dispatch per endpoint by temporal
semantics -- hints merge to one PUT (highest stamp per hash; sglang is
itself overwrite-by-stamp), migrates stay individual (rare; FIFO +
per-POST escalation preserved), program_paused coalesced by pid. Order:
program_paused -> migrate -> hints, so time-sensitive intents never
queue behind the flood. current_oldest_pending_age_ms() now also counts
the in-flight drained burst so /health backlog stays accurate.

sgl-project#227 (correctness bound, generous): a migrate decided on a state
snapshot is a prediction with a validity horizon; the worker drops one
aged past AGINFER_MIGRATE_FRESHNESS_MS (default 30s -- a pathological-
spike floor, NOT a tight knob; the coalescing removes normal-operation
latency). Proactive complement to the reactive APPLY_FAILED re-issue.

Latency proof (verify/outbound_coalesce B0/B1): with 200 hints queued
ahead of a migrate, the migrate is dispatch call #0 (not sgl-project#201) and the
200 hints collapse to 1 PUT; wall-clock migrate latency mean 0.2ms over
3 trials vs the N*apply serial floor of 1000ms.

Audit (no Critical/Major): hardened hints merge to max-stamp (order-
independent); documented that the sgl-project#164 stall backstop relies on the
always-on hints canary (a stale-dropped migrate emits no failure
signal). task_done accounting, escalation, /health live-peek all
preserved (t164/t36 green).

Tests: new verify/outbound_coalesce (7 stages: coalesce correctness +
latency proof). 17/17 touched suites green (t36 t164 t41 t187 t30 t22
t24 t23_t37 kv_scheduler_value_rule joint_decide admission_controller
t34 t16 dump_leaf_flags program_tracker t21). DESIGN §6 updated.
verify/t40 left as-is (pre-existing stub staleness, tracked sgl-project#229).
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 9, 2026
…ler interface

Pre-existing breakage at HEAD (not from sgl-project#224/sgl-project#227/sgl-project#228): the t40 stub
policy/router lagged the kv_scheduler.handle() interface that evolved
with the §9 joint_decide wiring.

  * _DeclinePolicy / _MigratePolicy: add costs=default_costs() + pi_u
    (handle() reads policy.costs/policy.pi_u for the §7 migrate
    candidate generator; real OursGreedyPolicy carries both).
  * _FakeRouter: add theta_hi/theta_lo (admission thresholds) +
    heartbeat_s (§8 forecast horizon).
  * B1b "hints alongside a migrate": post-sgl-project#194 the migrate comes from
    value-gated joint_decide, not the policy Action — so drive genuine
    HBM pressure (95% occ + h_max>0) and use MEMORY_PRESSURE, NOT
    TOOL_CALL_END (whose sgl-project#223 reuse-imminent carve-out suppresses the
    tail demote).  _state_json gains optional pressure/h_max knobs
    (defaults preserve prior behavior).

Production unaffected (real policy/router always had these). T40 17/17
green (1 e2e skipped).
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 9, 2026
…oject#208)

Real-stack do-no-harm verdict for the sgl-project#224+sgl-project#227+sgl-project#228 fix stack on TP=4 A3
(harbor/terminus-2 swebenchpro). Per-task agent_execution time (the LLM
serving phase the daemon affects; docker env-setup + verifier excluded):

  a3 (ours)     n=74  mean=734.2s  median=681.2s  std=296.5s
  a3_kvoff      n=81  mean=695.2s  median=628.2s  std=275.6s
  Delta = +5.6% mean, NOT significant (std ~290s >> 39s gap) -> HOLDS

No fatals/crashes, identical task outcomes (reward 0, 12 broken-env both
arms). Benefit absent at saturation (expected per sgl-project#230 Tier-1). The cyc2
80-reject burst (migr=155) is the pre-migrate-coalescing single-flight
wall that sgl-project#228-complete (f88c885) fixes; rejects were do-no-harm regardless.
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 18, 2026
…hain (demote now dispatches)

The whole-chain demote was proposed by migrate_candidates but joint_decide still
declined it live (29x policy_declined) while the SAME code on the post-run state
produced candidates. Cause: at the TOOL_CALL_START instant the just-finished
turn's per-program inflight hasn't cleared in the /aginfer/state snapshot, so the
sgl-project#224 _holder_actively_decoding gate suppressed the remove-HBM. But the event
AUTHORITATIVELY says the holder is now tool-parked for the ETA — a parked unit
won't re-lock (no evict-storm), so the lagging inflight is the wrong signal.
Bypass the gate for promote_pending units.

Result: tool_call_start now dispatches 29x (migrates=7..11 per decision = whole
chains), occ=0.99 gate fires, migrate_candidates yields the 4.48MB prefix demote.

REMAINING (next): migrate_applied still 0 — sglang rejects 736x
remove_hbm_not_device_leaf. The deepest-first peel-sort orders correctly but a
SINGLE apply_aginfer_migrations batch does not realize each remove's tree-mutation
before the next action (deferred eviction), so parents never become device-leaves
within the batch. The probe proved SEPARATE POSTs peel correctly. Fix = flush the
unlink within the batch (apply-side) or dispatch the chain as sequential batches.
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 18, 2026
…ppend)

Audit per review: the demote<->promote coupling + whole-chain demote did not only
ADD — they made three EXISTING passages incomplete/contradictory. Fixed in place:
- _value() formula (the canonical V_u): it charged the candidate tier's load_back
  unconditionally; now carries the reuse_tier=HBM-if-promote_pending override
  inline (was only in an appended paragraph 380 lines away).
- Structural leaf guard: stated as absolute per-unit; now documents the
  promote_pending whole-chain exception (propose all chain units, peel leaf-inward,
  dispatch as ONE coordinated group — the selector must not cherry-pick the bulk).
- sgl-project#224 active-decode guard: claimed tool-parked => not gated, ignoring the
  TOOL_CALL_START inflight snapshot lag; now documents the promote_pending bypass
  (the event is authoritative over the lagging metric).

Nothing was outright FALSE => no deletions, but all three needed editing, which the
first additive pass skipped.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 2, 2026
Desk design only, no cards (CUDA_VISIBLE_DEVICES=99), no implementation.

Verdict: sgl-project#407 is not a new abstraction, it is a NODE layer plus a
reconciliation. The edge half — what does it cost to move a byte from here
to there — already exists FOUR times with four readers and four absent-value
rules: cost_model.PairMatrix (#348b, UUID-keyed), barlink_path_rates.
PathProfile (sgl-project#279, affine per route), offload_movement.PeerPathCapability
(sgl-project#286, CUDA-ordinal-keyed) and card_probe h2d/d2h (sgl-project#271). E2 and E3 both want
the effective BAR1 aperture from p2p_readiness/capability_matrix.json and only
E2 has a loader. What exists nowhere is the node layer, and in its place the
tree carries four disjoint tier vocabularies (PARK_TARGETS sgl-project#286;
LOCAL_DESTINATION/SUPPORTED_PARK_BACKENDS sgl-project#224; StorageMedium telemetry;
HiCache choices=). Expert offload, the largest byte mover in the fork, is in
none of them: its tier is device="cpu" + .pin_memory() written out five times.

Two shipped contradictions the design bends around, both stated up front:

C1 — two opposite total orders over one axis. sgl-project#224's destinations_error makes
'local' first a hard validation law ("a remote tier is a tier BELOW local,
never above it"); sgl-project#286's ladder puts peer_vram ABOVE host_ram. Resolved by
making the order a shortest path over a STAGING GRAPH: 'local first' is
recovered as reachability (the only edge out of a device tier toward a blob
tier runs through host), peer_vram is recovered because vram->vram is a direct
edge. Neither consumer's shipped behaviour changes.

C2 — the one existing tier ledger is positionally keyed. CapacityLedger keys
peer-VRAM budgets by CUDA ordinal; AUDIT_331 calls that SCOPED and legitimate
until it crosses a process boundary, which is exactly what sgl-project#305 §6 ("one
ledger, no second accounting") asks for. On this rig the 5090 is CUDA ordinal
0 and NVML index 1, so the failure is silent card-swapping.

Honesty correction on three numbers the brief assumed measured: the peer-VRAM
1-3 us posted-write class is an explicit ASSUMPTION (EVAL_p2p §"Tragende
Annahmen"), NVMe latency does not exist in the tree in any unit, and host DRAM
38 GB/s is derived from an ASSUMED DDR4-3200 peak (the VM hides SPD). They
ship as ESTIMATE/ABSENT under #348b's measured-or-named rule. Everything else
in the inventory is sourced: 5090 32607 MiB / 3080 20480 MiB, membw 1558/723
GB/s, BAR1 256 MiB nominal but 96 MiB contiguous with 1.13-1.34x vs NCCL at
20 KiB-16 MiB, H2D 6.4/13/13 GB/s gen4 x4/x8/x8, NVMe 1.8 GB/s cold (ARC rows
excluded), rig-2 2.07-2.83 GB/s with 1.47 us at 8 B and ~89/202 us serving
collectives.

Consumer survey, eight consumers read in their target-selection code:
sgl-project#77/sgl-project#123 expert offload MECHANICAL for the UUID-vector half (it revives sgl-project#394's
merged-but-starved link-proportional sharding, 145 -> 86 ms/token, 1.69x) and
REBUILD for a third rung (two dicts, binary resident|spill, UVA aliasing);
HiCache MECHANICAL for backends, ladder EXCLUDED (it is the type signature of
every controller method, not configuration); sgl-project#224 MECHANICAL for the list,
REBUILD for the ordering law, victim half untouched; sgl-project#89 MECHANICAL-PLUS
(needs a persistence class or a tmpfs --hibernate-dir is a silent regression,
and a driver-free arg-parse-time query); sgl-project#305 NO MIGRATION, it is the awaited
mechanism; sgl-project#286 MECHANICAL and the donor — lift _select_target and
CapacityLedger rather than write new ones; sgl-project#389 and sgl-project#306 greenfield.

Cut plan, no big bang: 1 registry + read-only enumeration + GateRow dashboard
row (S); 2 publish the rank->card-UUID vector without a collective so sgl-project#394
comes alive (S, the only cut whose yield is already measured); 3 sgl-project#224 consumes
it (S-M); 4 CapacityLedger becomes a view of registry/ledger.py, keys widen to
TierId (M); 5-9 expert offload, hibernate, sgl-project#305 transitions, HiCache backends,
NVMe/sgl-project#306 declaration. Exclusions with hard reasons only: the HiCache ladder,
GDN state (DEVICE_BOUND_ONLY admission, a refusal not an omission), cross-rig
GPU-to-GPU (driver guard, sgl-project#280 closed in userspace), compute placement (sgl-project#302).

Measurement plan names six gaps, five of them S, three of them new arms in the
existing comm-suite which today has no BAR1, storage or host-DRAM arm.

codespell clean; ruff not applicable (documentation only).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 2, 2026
… consumers yet

Charter: every memory you have access to must be a level cache, a spill target
or an offload target depending on volatility -- disk, RAM, VRAM, local as well
as remote. Today four disjoint tier vocabularies coexist (sgl-project#286's PARK_TARGETS,
sgl-project#224's destination chain, StorageMedium telemetry, HiCache's choices= list) and
the largest byte mover in the fork, expert offload, is in none of them. This is
the node layer they can share. The edge layer (what a byte costs between two
places) is NOT re-implemented: planner.cost_model keeps it.

python/sglang/srt/memtier/, six modules, no consumer touched, no write outside
a test's temp directory:

* tiers.py -- TierId grammar (UUID or a named absence of one; a bare CUDA
  ordinal is refused at the door, naming C2: on this rig the 5090 is CUDA 0 and
  NVML 1, so a positional key silently swaps cards across a process boundary),
  the record, and the volatility law. Admission is a REFUSAL, not a ranking: a
  registry that only ordered tiers by cost would happily offer the GDN pool a
  remote tier, and recurrent state that made a lossy round trip is a
  correctness failure. Two enums, tier-side and payload-side, mapped by one
  table -- which is what makes the falsifier expressible.
* profile.py + profiles/rig1.json -- the measured numbers as DATA for exactly
  one machine, every rate carrying the probe it came off. Device tiers bind
  from card-MODEL templates when a live card is enumerated, so no UUID is
  hardcoded and an unprofiled model gets live capacity with ABSENT caps rather
  than a roofline (#348b D4 one layer down). Overlay merges by tier id.
* registry.py -- enumerate (blocked tiers included; omission is how a spill
  target silently becomes a different spill target), filter by volatility,
  object class, bandwidth floor, capability property and capacity, and a NAMED
  refusal for every tier that did not make the list. The ordering key is public
  so cut 3 can swap in the staging-graph path cost visibly.
* reservations.py -- one ledger, no second accounting (DESIGN_305 §6). A tier
  reservation is a named post in the sgl-project#260/sgl-project#400 Posten sense; VramLedgerHook
  forwards device tiers into registry/ledger.py so the corridor, lease and
  invariant stay where they are.
* probe.py -- DESIGN_407 §4's M1-M8 as records, every arm declared and every
  arm refusing. apply_outcome is the only writer of a measured cap.

The three honesty corrections ship as the design demanded: host DRAM bandwidth
is an ESTIMATE (its 51.2 GB/s reference peak is an ASSUMED DDR4-3200 figure --
dmidecode is empty under this VM), the peer-VRAM "1-3 us posted write" class is
ABSENT because it is an assumption and not a probe, and NVMe latency is ABSENT
because it does not exist in this tree in any unit. M1 also records the harness
it CANNOT use: barlink has no send/recv, so the BAR1 ladder needs p2p_readiness.

Deviations from the design, all recorded in its new §8: aperture is a Rate not
an Optional[int] (an int cannot say why it is missing); vram:unenumerated@<host>
added to the grammar so rig-2's cards are declared-and-blocked rather than
invented; no staging graph and no GET route (both cut 3 / a consumer).

Tests: 82 hermetic cases in test/registered/unit/memtier/, base-a-test-cpu, no
driver and no network -- 28 records/registry, 22 profile, 14 reservations, 18
probe. All green. collect_local_facts is executed for real against /proc/meminfo
and statvfs, and VramLedgerHook is driven against a real ReservationStore in a
temp directory, so neither path is desk-written.

Can-fail proof, 12 mutations, each turning exactly its guard red: EXPENSIVE_OK
admitting persistence; a CUDA ordinal accepted as a card key; the provenance
rank dropped from the ordering key (an estimate 21x faster than a measurement
then wins); an absent bandwidth read as zero; an unknown headroom read as
unlimited; an unnamed ledger post; a profile rate with no source; a roofline
substituted for an unprofiled card; a failed probe delivering its value; an
estimate entering through a probe; a measurement silently overwritten; blocked
tiers omitted from the enumeration.

ruff / codespell / black / isort clean; mypy clean on the package. The
test_module_state_ratchet failure on this branch is pre-existing
(layers/moe/utils.py _barlink_overlap_announced) and untouched here.

Desk work, no cards (CUDA_VISIBLE_DEVICES=99). Next: cut 2, publish the
rank -> card-UUID vector so sgl-project#394's merged link-proportional sharding comes
alive -- the only cut in the plan whose yield is already measured (145 -> 86
ms/token on the cold tier, 1.69x).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…m a source that can lie

TWO INDEPENDENT DEFECTS, both rooted in reading a number or a set from the
wrong place.

(A) THE GDN BLOB THAT GOT THROWN AWAY. build_gdn_slot_executor's idle_holders
closure enumerated live offload sessions through kv_session_offload.spills.
_commit_park POPS a sgl-project#224-parked session out of spills while the session stays
alive and keeps its req-pool slot, its radix tree lock and its Mamba/GDN state
slot. GdnSlotRuntime.on_round treats a parked id it cannot find in the
inventory as DEAD and calls forget(sid) on it. So: a session spills, the ladder
exports its GDN state to a blob, the session parks its KV to the next tier, and
the next ladder round discards the blob. It later unparks, restores its KV,
resumes -- and continues with recurrent state that no longer exists. No crash,
no log, wrong tokens.

The comment ("a parked session that died while parked leaves a blob nobody will
ever ask for") was not wrong; its PREMISE was. Absence from the inventory meant
"dead" only while every live session was in spills.

Fixed with KVSessionOffloadManager.live_offload_reqs() -- spilled AND parked --
as the single enumeration, plus kv_session_spill_destination.parked_reqs() as
the one place that knows where parked requests live (inflight_batches now goes
through it too, so the abort scan and the ladder share one answer). Note the
abort scan had ALREADY been taught about parked sessions; the knowledge existed
in the module and the second consumer simply never received it, which is what a
missing accessor costs.

(B) THE HOST-RAM NUMBER. memtier/profile._host_memory_bytes read /proc/meminfo
only, and the kvso host-pool guard sized a PINNED pool against psutil. Inside
this container both are lxcfs-synthesised and do not describe what the process
may have: MemAvailable can EXCEED MemTotal (an arithmetic impossibility, so the
guard was comparing against a number that denotes nothing), and with memory.max
unlimited they report the HOST's figures on a box other containers are also
spending. Over-committing pinned memory is the OOM killer picking a victim, not
a swap.

honest_host_memory_bytes() is now the pure decision: a finite memory.max is the
ceiling (capped by MemTotal), available is that ceiling minus anon and kernel --
page cache is reclaimable and deliberately NOT subtracted, since subtracting it
would refuse boots that would have succeeded; with no finite limit MemTotal is
the ceiling and MemAvailable the estimate, CLAMPED to it and clamped again by
the cgroup's own resident accounting. Anything that cannot be established stays
None rather than being guessed, and the kvso guard then SKIPS with a warning --
refusing to guess beats refusing a good boot on a fabricated figure.

memtier owns this per DESIGN_407 §2.9 (eight ownerless psutil.virtual_memory()
call sites; this module is meant to become the owner, not the ninth), so the
guard reads host_memory_bytes_for_pinning() there rather than adding a ninth.
This is also the honest number sgl-project#547's joint pinned-RAM budget will be built on.

Tests (CUDA_VISIBLE_DEVICES=99, hermetic, no GPU, no server):
  PYTHONPATH=python pytest \
    test/registered/unit/mem_cache/test_gdn_slot_runtime.py \
    test/registered/unit/memtier test/registered/unit/server_args \
    test/registered/unit/test_kv_spill_destination_unit.py \
    test/registered/unit/test_kv_session_offload_unit.py \
    test/registered/unit/test_kv_spill_budget_unit.py \
    test/registered/unit/test_kvso_reclaim_decline_501.py -q
  -> 942 passed, 19 warnings, 103 subtests passed in 14.35s

CAN-FAIL PROVEN, both halves:
- pointing idle_holders back at manager.spills turns
  test_the_builder_sees_a_parked_session red with "Lists differ: ['s'] !=
  ['p', 's']" -- the parked request missing from the inventory is exactly the
  state in which the ladder throws its blob away;
- dropping the MemAvailable clamp turns
  test_the_impossible_meminfo_reading_is_clamped red with
  "134217728000 != 128849018880".
The live-path test executes the real reader on this machine, so neither half is
desk-written-never-executed.

Every input to honest_host_memory_bytes is passed in, so no test asserts
anything about the machine it runs on (directive sgl-project#434).

FEATURE_CATALOG gains the absent-means-dead family entry.
Handoff: /spinning/spill-night-20260804/handoffs/
TICKET_551_gdn_park_and_honest_ram.md -- half A needs a hybrid-model boot with
--kv-session-offload-max-spills 1 (to force a park) and a binding
--gdn-resident-state-slots (to force a vacate), and explicitly states that a run
in which no park or no vacate happened proves nothing.

ruff check: no new findings on any changed file. ruff format: clean on all
added lines. codespell: clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…registry, not asserted

Cut 1. The memtier package (sgl-project#407) has been complete and INERT since it
landed: audit sgl-project#421 found zero production importers, and its own __init__
still says "No consumer reads any of this yet". Nothing in a serving path
ever constructed a TierRegistry. This makes the KV spill rung the first one
that does.

The rung spills to exactly one place, and the ORDER of any further sgl-project#224 park
tier is a hardcoded rule: destinations_error() refuses a list whose first
entry is not 'local', quoting a measurement in prose. That rule is right and
it is the wrong KIND of statement -- a name check standing in for a
capability check, with its evidence in a comment where nothing can re-derive
it. sgl-project#407 says so itself, at the field that exists for this exact edge:

  TierTransport.stages_through: "sgl-project#224's `local` must be first law is exactly
  this edge ... recording it now is what lets that law stop being a hardcoded
  total order later."

This is that later. kv_spill_tier_selection builds a registry whose FIRST
tier is local host RAM and derives the ladder from measured capacity and
cost. Local-first stops being asserted and becomes DERIVED -- and falsifiable:
local_first_disagreement() reports when the hardcoded law and the measured
ladder stop agreeing, and deliberately does NOT reorder, because the law
encodes a physical constraint (a device D2H copy has nowhere else to land),
so a cheaper-looking park tier is evidence of a bad number, not a reason.

TierCapacity.reserved is populated from a live source for the first time.
sgl-project#407 documents it as "bytes declared by other holders, read from the
cross-process ledger" and nothing ever wrote it, so every headroom the
registry could compute was the headroom of an empty machine.
pinned_host_budget.registered_posts() IS that ledger -- the join is now made,
and a second pinned pool correctly reduces this tier's headroom.

Exhaustion is a NAMED refusal (the ledger half of sgl-project#659 b). A region shortfall
is the moment the local tier is full, and it was recorded only as a pressure
timestamp: the operator saw a decline with no statement of what was full or
how full. The ladder now reports one line per tier with a provenance, renders
ABSENCES with their reason rather than as a dash ("unknown" and "zero" lead
to opposite decisions), and degrades to the next tier when one exists.

BYTE-IDENTICAL by construction: in this cut the registry is an OBSERVER under
the rung. The sgl-project#224 chain still picks the destination; nothing in the spill
path changes. That is deliberate -- the registry's verdict has to be
comparable against the hardcoded law before anything is allowed to depend on
it.

Also fixes a latent bug found by smoking the wiring rather than reading it:
kv_session_host_bytes returns None (not a pair) when offload is off, and
unpacking it directly raised inside the caller's blanket except -- an
ordinary "no pool" presenting as an invisible skip.

Tests: test_kv_spill_tier_selection_659, 13 tests, green. Every load-bearing
assertion has a sibling arm producing the opposite verdict from the same
function, so the falsifier is proven can-fail rather than asserted:
- local_first_disagreement returns None on the measured ladder AND non-None
  when a park tier is given absurd caps (test_disagreement_is_reported_
  rather_than_obeyed);
- a full local tier declines alone, and degrades to tier 2 when present;
- roomy vs crowded proves the reserved-from-posts join actually binds.

Live smoke (desk code that never ran is unvalidated): the wiring executed and
printed
  kv-spill ladder: host:CT999 (ram-local) headroom=0.61 GB [measured]
  bandwidth=absent (no D2H bandwidth probe recorded for this pool)
against a pool of 8 regions x 100000 tokens x 1024 B with 2 occupied --
arithmetic correct, absence rendered, rate-limit holding on the second call.

sgl-project#631 flip family: 1076 passed / 0 failed, inherited baseline unmoved.
ruff + codespell clean.

Ship config is untouched: it carries no --enable-kv-session-offload, so
live_kv_spill_ladder returns None at its first line.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
… and the ladder now picks

Cut 1 put the sgl-project#407 registry under the KV spill rung as an observer with one
tier on it. This is the second rung and the moment the registry stops
observing.

The brief's preferred second tier does not exist on this rig: s41 measured
Rig-2 RAM as unreachable (unroutable subnets, 75 MB/s on the only routable
path, 8.6 GiB of swap-backed RAM against a ~12.9 GB region) and peer VRAM is
closed by the standing REBALANCE verdict. The cheapest REAL second tier is a
local filesystem, and it is registered as what it is: a tier, not a remote
tier.

Every metric on the entry is measured at registration against the directory
the blobs actually take -- never read from memtier/profiles, whose remote rows
describe a link this process cannot route to (register C24). Two measurement
traps are avoided by construction and documented with their numbers: probing
with zeros measured the ZFS compressor at 6.2 GB/s rather than the disk, and
reading back a just-written file measures ARC rather than the medium. The
probe writes incompressible bytes, ends in fsync, ranks on the WRITE rate, and
records the read-back as the upper bound it is. The probe size is 256 MiB and
the entry says why: a 64 MiB probe overstates this volume by 2.8x.

The capacity is the operator's ceiling floored by df, in that order --
min(budget, free - headroom) -- so a generous budget on a nearly-full volume
yields a small tier instead of a disk-full outage.

Selection: the ladder now ranks the configured park tiers by measured
bandwidth and refuses the rest by name, replacing tier_index=0. It is
byte-identical wherever budgets allow -- with one healthy tier the ranking is
degenerate and returns 0, which is what the hardcoded law returned -- and cut
1's falsifier still passes.

Every input to that decision is rank-uniform, deliberately: the ask is a boot
constant, the capacities were measured at registration rather than re-read,
and the ranking is computed once and cached. There is no per-park try/except
around the choice, because a rank that fell into one and parked, beside a rank
that succeeded and declined, would diverge into the completion min-reduce and
wedge the instance rather than error (register law 14).

The sgl-project#224 park counters are read for the first time. They were complete and
correct for three shifts while every reference in the tree was a write, so a
park that failed and a park that never happened produced the same silence.
They now feed the selection policy (a tier is charged for the parks it eats
and is refused on health once it has eaten enough), the ladder's ledger line,
and the dashboard, which gains a measured denominator for park occupancy.

Tests: 17 new, each load-bearing assertion with a sibling arm producing the
opposite verdict from the same function. sgl-project#631 flip family 1076 passed / 0
failed before and after. Adjacent kv-spill, spill-tier, tier-occupancy and
hicache-bounds modules 112 passed. ruff and codespell clean on the new files.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…ep path asserts, and it is what blocked sgl-project#659's end-to-end proof
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…s re-checked at code

A parallel trace landed after cb0097c and contradicted one of my own
statements. Correcting it rather than leaving it, and recording three further
findings it surfaced. Each re-verified at code here, not taken on report.

THE CORRECTION. §9.1(c) said "beyond that first slot, order is by measured
bandwidth, not by operator preference". Wrong. Below the local slot the
operator's configured order in --kv-session-offload-destinations IS respected;
a measurement may promote a tier above it only when that tier is at least
PROMOTION_RATIO = 4.0 times faster (kv_spill_park_tier.py:134, applied by
_park_ranking, kv_session_spill_destination.py:1259-1300), and that ratio is a
module constant behind no flag and no env var.

This makes sgl-project#224 MORE delivered than I credited: "Zielliste + Richtung
konfigurierbar" is satisfied below a physically fixed first slot. What is not
configurable is the promotion threshold that can override the operator.

FOUR REFINEMENTS (§9.5):

1. The remote row cannot reach a ladder even if read. I wrote that the spill
   path "deliberately does not read" rig1.json's remote rows -- true, but it
   understates the guard. host:rig-2 carries health.reachable = false
   (rig1.json:164) and TierRegistry refuses on exactly that before any other
   rule (registry.py:353), with a named reason. The tier is enumerable and can
   never enter a select() candidate list. Structural, not a convention.

2. sgl-project#286's `remote` is an always-refusing stub and says so in its refusal:
   "remote: stub tier (sgl-project#224 RDMA attachment point, not wired yet)"
   (offload_movement.py:750). Recorded because sgl-project#286 and sgl-project#224 both say "remote"
   for different things -- sgl-project#286's is a placeholder in the expert-offload
   ladder, not the kvso park tier.

3. The sgl-project#286 registry consumer is unreachable from production. price_park_target
   is the function that actually calls registry.select(TierQuery(...)) and is
   annotated as the fork's first memtier consumer, but its only callers are in
   test_short_term_offload_register.py. memtier/consumers.py's
   expert_offload_host_targets is the same shape. So §9.1(d)'s importer list
   needed the qualification now added: seven modules import memtier, but only
   kv_spill_tier_selection / kv_session_spill_destination perform tier
   SELECTION on a live path. The sgl-project#421-F3 shape again, inside sgl-project#407's own
   migration.

4. A provenance defect the profile records against itself: rig1.json:172 warns
   that the 3.43 GB/s and 1.47 us figures quoted together in
   destinations_error come from DIFFERENT lines (100G and 40G). Those paired
   numbers appear in the --kv-session-offload-destinations help text as the
   justification for the local-first law. The LAW is not in doubt -- it is a
   D2H-staging argument, not a bandwidth one -- but an operator reading that
   help is reading a mixed pair. Not fixed here: it is help-text prose and the
   correct replacement needs the sgl-project#266 run's numbers re-read rather than
   invented. Named so the next edit does not re-copy it.

No code change; docs only. Tests unaffected (the 8 pointer-I/O cases and the
memtier suite are unchanged by this commit).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants