Skip to content

misc: update output token logic - #695

Merged
zhyncs merged 2 commits into
sgl-project:mainfrom
zhyncs:fix-eos
Jul 22, 2024
Merged

zhyncs merged 2 commits into
sgl-project:mainfrom
zhyncs:fix-eos

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 22, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs
zhyncs merged commit d198791 into sgl-project:main Jul 22, 2024
@zhyncs
zhyncs deleted the fix-eos branch July 22, 2024 09:34
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
* Update run_suite.py

* Update pr-test-npu-debug.yml

* debug testcase

* Update pr-test-npu-debug.yml

* debug testcase

* debug testcase

* Update pr-test-npu-debug.yml

* Update pr-test-npu-debug.yml

* debug testcase

* debug testcase
hhhh1252023 pushed a commit to hhhh1252023/sglang_public that referenced this pull request Jun 5, 2026
Add tokenizer,update weights from disk testcase
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…the suites

Follow-up to 58660f2. Both were found by running the manager suite, not by
reading the code, and the first falsified a claim I had written into the
census module's own docstring.

(a) THE CENSUS TOOK A BOOT DOWN. It was wired as
    log_host_shmem_census(rank=self.tp_rank), and Scheduler has no tp_rank
    -- the residency census next to it reads getattr(model_runner,
    "tp_rank", -1), off the model_runner and defensively, which I did not
    copy. The AttributeError propagated out of scheduler init, the launcher
    SIGKILLed the process group, and test_customized_info_streaming.py died
    at exit 137 with no summary at all.

    log_host_shmem_census already had a try/except and its docstring said
    it was "never able to break a boot". That was false twice: the argument
    is evaluated at the CALL SITE, outside the guard, and the guard covered
    only the collection and not the rendering after it. A try/except inside
    a function does not protect the expression that computes its arguments.

    Now: rank read with getattr off the model_runner, the call wrapped at
    the call site, and the whole body wrapped inside. Pinned by
    test_the_census_can_never_raise, which drives both the collector and
    the renderer to raise.

(b) torch.empty SILENTLY BECAME A ZEROING ALLOCATION. arena_image used
    torch.empty; routing it through _alloc_host_image made it zero-fill,
    faulting the whole image in up front -- resident pages, not virtual
    size, are what get a process killed. _alloc_host_image now takes
    zero=, and arena_image passes zero=False since every byte there is
    overwritten immediately. image_from_tensors keeps zero=True: its
    alignment-gap contract depends on it. Pinned by
    test_an_overwritten_image_is_not_faulted_in_up_front.

(b) was found first and did NOT fix the exit-137; (a) was the cause. Worth
recording: the first plausible explanation for a memory-shaped symptom was
memory-shaped, and wrong.

TESTS -- every directory now compared against a clean worktree of the
parent commit 281bbfb, because a failure count means nothing without the
number it is compared to:

  3 new files                                37 passed
  unit/{mem_ledger,memtier,model_executor,model_loader}
                    this branch  16 failed, 1470 passed
                    281bbfb   16 failed, 1433 passed   (+37 = new tests)
  unit/managers     this branch  13 failed, 1298 passed, 1 error
                    281bbfb   13 failed, 1298 passed, 1 error   (identical)
  unit/managers -k "flip or phase or arena or weights"  463 passed, 0 failed
  ruff --select=F401,F821,UP037 clean on every changed file.

The 16 and the 13 are identical on both sides and all GPU-dependent
("retry() exceed maximum number of retries", "Process group cannot be
None") -- pre-existing, untouched by this branch.

Handoff updated with both regressions (section 1.2) and the full
comparison table (section 7).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…runcated to 15 chars

discover_scheduler_pids tested comm.startswith("sglang::scheduler"). The
kernel stores comm in TASK_COMM_LEN = 16 bytes, so /proc/<pid>/comm returns
the 15-character prefix "sglang::schedul" and the test was False for every
process that ever existed. Against the live three-rank PP=3 boot the script
reported "no sglang::scheduler process found. Boot the server first".

Found on first contact with metal, which is the point: the branch's desk half
was hermetic and never called this function with a real comm, so the whole of
the sgl-project#695 validation recipe would have burned its first GPU window here.

Matches the truncated prefix instead, and adds an injectable read_comm/pids
seam so the matching is testable without a server. Red first: 5 of the 6 new
tests fail on the pre-fix script (the sixth pins the kernel-truncation premise
and passes on both sides).

Verified after the fix on the live instance: all three ranks discovered, and
the census reproduces the sgl-project#695 finding exactly -- nine anon-shared mappings,
every one a power of two, 74.40 GiB total against cgroup shmem of 75.06 GiB.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…-project#695 proven on metal

Two branches merged in the approved order, each tested against a
same-environment baseline and pushed to feat/route-a-631 and integration/r2
before the next was started. Flip family holds at 1116 throughout.

sgl-project#695 is proven on metal, all five gates. anon-shared 74.40 -> 60.24 GiB; no
large mapping is a power of two any more and the new sizes match the payload
figures already recorded at phase_flip_spill.py:851-854 to two decimals, which
is what makes the result a statement about the allocation rather than a
free-memory reading. The generation is byte-identical including output_ids.
Under 21 minutes of real load the cgroup shmem peak falls 76872 -> 62367 MiB
and MemAvailable min rises 30827 -> 44635 MiB, with no OOM kill.

Four defects found and fixed, all red-first:
 - the sgl-project#695 metal recipe could never find a rank (comm is truncated to 15
   chars, the test string was 17) -- it would have burned its first window;
 - SIGTERM had no launch-phase handler, so a terminated boot orphaned ranks;
 - preflight folded the NVML driver carve-out into "used" and refused an idle
   machine, while the rest of the tree already measures and subtracts it;
 - the five turnkey units hardcoded a checkout independently of [stack].repo.

And prod_boot.sh no longer keeps a private copy of the ship environment: the
capture governs it, with a refusal gate immediately before exec.

Register gains C23 (SGLANG_UNEVEN_TOKEN_VECTOR is the uneven-DCP KV-token
vector, not the PP stage ratio -- correcting a stated VAL-R4 hypothesis), C24,
and laws 37-39.

Corridor 0 breaches on both instruments, 546 flips, 273 each direction, 0
abandoned, 0 tracebacks. Serving verified with real generations.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
TICKET 1 -- flip-latency A/B: CLEAN, and the sign is the other way.
MERGE-R5 left flip p50 2036 -> 2153 ms (+5.7%) open against the sgl-project#695
exact-size pin, measured n=18 vs n=546 across different load profiles against
a baseline log its own window text contradicts. Same-harness replacement, one
md5-frozen tree, arms differing by SGLANG_PHASE_FLIP_EXACT_PIN alone:

  fix     n=540  min 1388.6  p50 2200.1  p90 2659.6  p95 2713.4  mean 2256.8
  revert  n=534  min 1703.2  p50 2257.5  p90 2722.4  p95 2903.2  mean 2309.5

The exact-size pin is FASTER on every statistic: -57.4 ms at p50, -314.6 ms
(-18.5%) at the minimum, which is the figure a change in the page-locked DMA
property would move and load noise would not. Both arms balanced across the
seam (270+270, 267+267), 0 breaches on both instruments, soak err=0 on both.
The arms are provably the two allocators: the revert arm reproduced merge-r4's
host profile (shmem peak 76919 vs 76872 MiB) and the fix arm reproduced
merge-r5's to the megabyte (62367 vs 62367). Ships on by default; the env gate
stays as the escape hatch. No part of the pricer or budget-post was reverted.

TICKET 2 -- sgl-project#644's residual ~16 GB: ALLOCATOR, not retention. On the GGUF MoE
checkpoint, in-process at end of load: live CPU tensor storage 20.1 MiB, both
named holders empty on every parameter, and malloc_trim(0) returns 14714.2 MiB
-- 91.6% of the residue. The outside sampler sees the same release at constant
process count (17.713 -> 3.339 GB). No holder to fix. Whether to trim
automatically is booked as a candidate, not built.

TICKET 3 -- the 40,12,12 planner arm is UNDISTURBED by the r4+r5 merges.
Deep prefill 64.974 s against N51's 65.257 s, ship control 81.867 s against
N51's 81.878 s -- 11 ms apart on an 82-second measurement, across two shifts
and two trees. Corridor 0 breaches, seam census 0 law breaks, 192+192 flips
both directions, 0 abandoned, soak err=0. A spot check: N51's boot-to-boot
spread caveat stands.

Errors first in the handoff: I used pkill -f against a standing prohibition
and it killed my own shell (law 40); the ship config has under 300 MiB of
slack on the 5090 so any co-tenant blocks the boot while the preflight gate
waits only above 2000 MiB (law 41); and a can-fail claim I wrote could not be
reproduced, so the test was corrected rather than the claim kept.

Register: C25, C26, laws 40-42.
Serving restored by capture replay and verified with two real generations,
argv identical 60/60, corridor 1571/3866/1959 MiB.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 13, 2026
…el collision

Both R7 source branches extend docs/dev/631/CONTRADICTIONS_REGISTER.md and the
union merged cleanly (+526 lines, 0 deletions), but the merged file carried
EIGHT ambiguous identifiers, which is a silent defect in the one document whose
purpose is that a claim can be looked up.

feat/vram-flightrecorder-605 appended eight table rows labelled C23-C30. Every
one of those labels was already in use at 481411a for an unrelated
contradiction:

  C23-C26  the four '### C<n>' sections further down the file (the
           SGLANG_UNEVEN_TOKEN_VECTOR reading, the sgl-project#695 recipe's rank
           discovery, the sgl-project#695 exact-size pin, sgl-project#644's allocator residual)
  C27-C30  narrative entries the body itself cites 4, 9, 3 and 6 times

The briefing for this shift anticipated a four-label collision (C23-C26). It is
eight: C27-C30 are load-bearing too, which is only visible by grepping the body
rather than the headings, since the incumbents are prose entries and the
arrivals are table rows -- the two forms do not textually conflict, so git
merged them without a murmur.

Resolved by renaming the ARRIVING side, which has no inbound references in this
file, to a namespace that cannot collide: C23->C605-1 ... C30->C605-8, in
arrival order. Verified programmatically that all eight row bodies are
byte-identical apart from the label; nothing was dropped, merged or reworded,
and no incumbent entry was touched. A note above the table records the mapping,
and both sgl-project#605 handoffs, which name their own labels in prose, carry a pointer to
the shipped ones.

Docs only: no code path, no test, no count changes.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
… explain

Measured 2026-08-17 02:18:09, all three ranks, inside the OOM this reporter
(sgl-project#695) was written to diagnose:

  RADIX SHAPE: walk failed after 3 nodes (RuntimeError('Boolean value of
  Tensor with more than one value is ambiguous')). Partial: tokens=1,
  locked_nodes=1.

Cause, and it is mine: `len(getattr(node, "value", ()) or ())`. node.value is a
TENSOR, so `tensor or ()` evaluates bool(tensor), which raises for any tensor
with more than one element. The walk therefore survived only empty or
single-element nodes -- an empty tree -- and fell over on the first real one.
It has presumably never worked on a populated tree since it was written.

A diagnostic that only works when there is nothing to diagnose is not a
diagnostic, and this one failed at precisely the moment its output was the
evidence needed: the tree shape at the crash would have shown whether the
147,456 tokens the counter certified were reachable from the unlocked leaf
frontier the actuator walks. That question is still open BECAUSE of this bug.

Length is now asked for directly and never via truthiness.

Tests, hermetic (CUDA_VISIBLE_DEVICES=""):
  test_multi_element_tensor_value_does_not_break_the_walk .. the specimen shape
  test_counts_tokens_across_a_deeper_tree ................. 100+30+7 counted
  test_none_and_empty_values_still_count_zero ............. CAN-FAIL boundary:
      the fix must not crash on a missing value nor count None as a token
  test_a_locked_multi_element_node_is_still_counted_as_locked
  -> 4 passed + 2 subtests; 16 passed across the 715 + 695 + 694 mem_cache
     suites. ruff clean.

CAN-FAIL PROVEN BY MUTATION: restoring the original expression fails 5 of 6;
reverted and re-verified 4 passed. The suite would have caught the bug on the
day it was written.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
The Flip+HiCache run-card boot refused on PP0 with "40.42 GB requested ... does
not fit in 33.97 GB available minus a 10.74 GB OS reserve = 23.23 GB usable".
35.18 GB of that demand was the three phase-flip weight images. They were
already allocated when the figure was read.

MEASURED BEFORE ANY DESIGN CHANGE WAS PRICED, because a cheaper answer outranks
a better one. Sampled against the live serving process, mid-flight and flipping
normally: MemAvailable 33.62 GB while the three schedulers held 86.51 GB
resident (PP0 39.85, PP1 22.27, PP2 24.40) on a 126.75 GB box. The images are in
RSS, therefore already absent from MemAvailable -- and the backstop added them
to the demand as well.

check_and_register_pinned_post summed every already-registered post plus the new
one and compared that total against LIVE, post-allocation availability. sgl-project#695
registers the weight images AFTER allocating them (weights_arena.py:428), so
from that point every later post in the process paid for them a second time.

joint_pinned_host_error is not at fault and is unchanged. It is exact where it
was designed to be used: once in the launcher, over CONFIGURED numbers, before
anything is pinned, where availability is untouched by any post. Reusing that
comparison as a runtime backstop is what introduced the error, because by then
the sum and the availability figure disagree about what has already happened.

Fixed by crediting the already-allocated posts back to availability. The true
marginal cost of that boot on PP0 is the 5.24 GB tier, which fits in 23.23 GB
with ~18 GB to spare. No dedup, no mmap, no freeing of host RAM is needed for
this ticket -- the blocker was arithmetic, not physics.

WHY CREDITING IS SOUND, and its one precondition: registration FOLLOWS
allocation for every producer of a post. check_and_register_pinned_post
registers after its own check, so the post being weighed is correctly NOT
credited to itself; the weight images register after _alloc_host_image returns.
A future producer that reserves ahead of allocating must not use this path, and
the comment says so. Other ranks' pins are still charged in full: they are
absent from availability and absent from this process's registry, so the
rank-divergence bound in the module docstring is unchanged.

TESTS  test/registered/unit/mem_cache/test_pinned_budget_double_count_706.py,
6 passed; mem_cache 940 failed / 999 passed, the 940 baseline unchanged and the
6 new passes are these. Mutation-proven: dropping the credit turns the specimen
test red. The suite pins the machine out of the measurement with the specimen's
own numbers -- read live, a roomy box would admit the tier without the fix and
the can-fail would prove nothing there.

Can-fails included: an impossible tier is still refused (crediting must not
disable a guard on non-swappable memory), the refusal still prices every post by
name and flag, the post under test is not credited to itself, and the first post
in a process is unaffected.

DOC  DESIGN_706_BOOT.md, three corrections found by executing section 5:
  * 5.2 said `--hicache-size 5G`, called itself verbatim, and is rejected by
    argparse (`invalid int value: '5G'`); the arg is an int in GB. Cost a boot.
  * new 1.2a: 1.2's measured figures are ONE rank's, and it is the smallest.
    PP2 reproduces them exactly (10.94/9.41/0.15); PP0 carries 18.06+17.12 =
    35.18 GB. So 1.3's 5.37 GB remainder does not hold on the binding stage,
    where it is -11.95 GB before any tier, and 5.5's "shrink --hicache-size"
    cannot rescue such a boot.
  * new 1.2b records the double-count and its fix, and precondition 1 now says
    which rank its 24.3 G floor is calibrated on.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…ct the sgl-project#550 fixtures

sgl-project#630: the parse-time blocker is back, narrowed to match 90e84ad EXACTLY
(pp_size > 1 AND enable_hierarchical_cache AND a storage backend), with the
same reason string and a pointer to the runtime twin. Refusing at parse costs a
second; the runtime twin fires after a full weight load, which cost 11 minutes
and a dead instance. The sgl-project#703 comment is rewritten to the new truth with its
provenance: 9da9dfd BOUNDED the wait and never rooted the desync, and
test_hicache_bounded_waits_630.py proves only that a bounded call raises on
schedule against mocked Work objects.

test_v1_blockers_named now names a storage backend, because the narrowed guard
deliberately no longer refuses the device+host-local tier.

DESIGN_706_BOOT.md gains PRECONDITION 0 -- this boot is BLOCKED until the
pp_sync rendezvous is rooted -- so the refused boot and the run-card that
describes it cannot drift apart, and it names what stays reachable meanwhile.

sgl-project#550 DETERMINATION, from the producers rather than from either model. Every
consumer except one goes through check_and_register_pinned_post. The exception
is weights_arena.py:428 (_register_image_post), and it registers BEFORE it
allocates -- lines 439-460 do the allocation -- so exec's stated invariant
'registration FOLLOWS allocation for every producer' is literally inaccurate.
But the gap closes inside the same call, so any LATER checker sees bytes that
are already allocated and the credit-back is sound where it is used. No
producer leaves a registered-but-unallocated post standing on a success path.

So the fixtures encoded an impossible machine, and they are corrected on
physics:  is read LIVE, so an allocated 30 GB post cannot coexist
with a static 40 GB available, and it cannot stay 40 GB after 20 GB is
allocated. Both tests keep their exact intent -- a post must not be admitted on
its own arithmetic, and the second post must see the first -- with numbers a
machine can actually be in.

Residual, named not buried: on the weights_arena allocation-FAILURE path the
post stays registered with nothing allocated, and there the credit-back would
under-charge. Narrow, real, and owned by sgl-project#706/sgl-project#695.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
…ving hot paths

An isolation boot wedged for 25+ minutes with 120 ADMISSION-WEDGE markers.
py-spy (two dumps 25 min apart, byte-identical, so hard-stuck) put PP0's
MainThread INSIDE logging.emit while PP1 and PP2 starved in
pp_chain_receiver.recv for a chain send PP0 never reached.

ROOT CAUSE. PP0 admitted the first radix-carrying request and reached the
sgl-project#767 instrument in HybridReqToTokenPool.alloc, which logged
req.mamba_pool_idx -- a 1-element CUDA tensor -- as a %s argument. The
format call runs Tensor.__repr__ -> _tensor_str -> a D2H copy -> a stream
synchronize, inside logging.emit, on the admission path. The device was
occupied by a spinning ncclDevKernel_SendRecv, so the sync never returned.
The instrument produced ZERO output on that boot: the record died mid-format,
which is why the log looks like the branch never ran.

THE TRAP, NAMED SO IT IS NOT RE-INTRODUCED. Every obvious way to print the
value synchronizes: %s/str()/repr(), .item(), .cpu(), .tolist(), float(),
int(), and f-string interpolation. A fix that swaps one for another only
relocates the sync. sync_free_tensor_repr returns host-resident metadata
instead -- shape, dtype, device, and id() to correlate one tensor handle
across lines -- and passes non-tensors through unchanged, so a value that is
sometimes a tensor and sometimes a plain int is safe either way.

SWEPT THE FAMILY (sgl-project#695: expensive work inside logging arguments on serving
paths), fixing six more sites of the identical shape:
  mamba_component.py               sgl-project#767-TRACE prefix-match
  model_runner.py                  sgl-project#767-TRACE cow_and_clear SKIP and body,
                                   per extend forward pass -- hotter than the
                                   original site
  mamba_radix_cache.py             cache_finished / cache_unfinished / match
  dflash_solo_pool.py:_reclaim     THE ONE WITH NO GATE

The dflash site matters most: every other hit sits behind an opt-in debug
flag, that one runs unconditionally on decode-time draft-slot allocation
under real load. It computed self._slot_epoch[victims].max().item() purely
for a diagnostic string. Victims are drawn from the ascending argsort's low
end, so their epoch is bounded above by self._epoch, which is already a host
int -- and the message text is REWORDED to match what is now reported
("at round X ... untouched since before this round") rather than quietly
printing a different number under the old wording.

LISTED, DELIBERATELY NOT FIXED, with reasons rather than silence:
phase_flip_output_trace.trace_round and phase_flip_resident_carry's cutover
falsifier (both bounded on purpose, the second says so in its own comment),
ngram_corpus.debug_result (reachable only from a __main__ demo), and
dspark_planner._log_verify_lens_decision -- mechanically the same shape and
genuinely hot when armed, but its whole purpose is the exact per-request
values, so an identity stand-in would gut the tool rather than trim it. It
wants rate-limiting, not this treatment, and is left as a named follow-up.

TEST. test_admission_log_no_device_sync_790.py, hermetic and CPU-only. A
tripwire monkeypatches Tensor.__repr__/__str__/item/tolist/cpu/__float__/
__int__ to raise, and the test drives the REAL HybridReqToTokenPool.alloc
through the sgl-project#767 branch rather than a re-implementation. Red-first was
demonstrated by reverting just the one call-site argument: the tripwire fires
through alloc -> logger.warning -> emit -> format -> getMessage -> msg % args,
reproducing the incident's exact path. Green after: 5 passed. A can-fail case
restores the identity formatting to prove the trap still fires from inside
the real branch.

Verified with the sgl-project#788 and sgl-project#787 suites alongside: 12 passed.

Pre-existing and NOT caused by this change: 4 failures in
test_mamba_anchor_seams_747.py (AttributeError: 'MambaComponent' object has
no attribute 'cache' in _raw_token_pos), confirmed identical with the change
reverted. Ruff findings in mamba_radix_cache.py and model_runner.py outside
these hunks are pre-existing lint debt.

This is the probable cure for the isolation boot's wedge: the log sync was
the linchpin edge of that cycle. It does not touch sgl-project#789 -- one relay, two
transports, no shared readiness contract -- which remains open debt.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…remise

POSTEN 1 -- MEASURED FIRST, AND THE PREMISE DOES NOT SURVIVE THE MEASUREMENT.

The instruction was: the guard is inert because there is no readable total, so
cap `memory.current` against a constant 111.3 GiB from sgl-project#721. Measured on this
box before touching anything:

    pinned_host_memory_bytes() -> total=118.05 GiB  available=112.95 GiB

so it does NOT land on the :245 "no honest number" branch. The guard is ACTIVE,
a total IS readable, and `PINNED_HOST_RESERVE_BYTES` = 10 GiB is therefore NOT
inert as a subtraction. The clamp binds too: MemAvailable 113.19 -> 112.95 GiB,
reduced by the cgroup's own resident accounting exactly as rule 2 of
`honest_host_memory_bytes` documents. `MemAvailable > MemTotal` is FALSE right
now, so the pathological lxcfs reading is not present either.

AND THE PROPOSED CAP WOULD NOT HAVE BOUND ANYTHING. Every level of the visible
hierarchy is unlimited -- root, system.slice and system.slice/claude.service
all read `memory.max = max`, and all three report `oom_kill 0` today (sgl-project#721's
17 kills predate this boot; `memory.peak` 101.98 GiB of 118.05). With no
enforcing cgroup anywhere, the physical machine IS the ceiling, which is what
MemTotal reports. A hand-carried 111.3 GiB constant would have been a third
floor with no enforcer behind it -- the rule this ticket already established,
applied to its own instruction. Not built.

THE REAL DEFECT IS ONE BUCKET OVER, AND IT IS THE sgl-project#695 CLASS AGAIN. Prior art
via the new index: commit c043235 measured that "CUDA pinned host memory is
accounted in the cgroup's `file` bucket, not `anon` ... the offload ledger
reported 20.78 + 14.44 + 14.44 = 49.66 GiB of pinned pool while `anon` sat
steady at 14.6 GiB." Confirmed live on this box: anon 2.12 GiB, file 19.06 GiB,
current 21.39 GiB -- the clamp charges ~2.4 GiB of a cgroup holding 21.4.
`honest_host_memory_bytes` charges anon + kernel + unreclaimable shmem and
NEVER charges `file`, correctly, because page cache is reclaimable. Pinned
bytes there are not. sgl-project#695 already fixed one member of this class (shmem hiding
in `file`); pinned host memory is the second and is still uncounted.

IT ALSO PUTS A DOCUMENTED PREMISE IN DOUBT, and the doubt is precise:
`pinned_host_budget.py:253` credits already-registered posts BACK to available
because "their bytes are therefore already missing from it". For a post whose
bytes land in `file`, they are NOT missing from it -- the clamp never
subtracted them -- so the credit-back would count them as free twice. The
2026-08-17 measurement quoted there is sound for the weight images it was taken
against; whether it holds for THIS pin depends on which bucket the pin's bytes
land in.

DELIBERATELY NOT FIXED HERE. That question is decidable only by watching `anon`
and `file` across an actual pin allocation, which is a boot. Changing admission
arithmetic on a shared path used by HiCache and kv-session-offload, on a guess
about which bucket, would be the "capping on a fabricated figure" the module
exists to prevent. So it is INSTRUMENTED instead -- the window script prints
the before/after pair and names the consequence -- and the arithmetic is left
alone until the boot answers it.

POSTEN 2 -- `chunk_blocks_quiescence` (phase_flip_runtime.py:144): SAME CLOCK
DIVERGENCE, and the determination is not a formality.

It is the same class as the retention gap: a consumer (the cutover) and a
writer (the chunk boundary that inserts into the tree) on different clocks.
Under STRICT the clocks are SYNCHRONISED -- the cutover waits for the prefill,
which is what "waiting for prefill to finish IS drain-and-flip" means in its
docstring -- and the residual is exactly the NON-STRICT case where they are
not.

THE DIFFERENCE THAT MATTERS, and why this is one root and not two: the chunk
case has a BOUNDED wait available (a prefill completes in finitely many
chunks), while a mid-decode resident does not. That is why STRICT can fix one
and nothing can fix the other by waiting. But the NON-STRICT chunk case cannot
be fixed by waiting either -- an unconditional block re-creates sgl-project#631 defect O,
the 32768-token prefill that ran in the slow layout and paid two cutovers for
nothing. So BOTH halves need the same missing capability: persisting PARTIAL
work at the seam. Neither is closable without it, and that capability is
`seam_copy_state` / `restore_seam_state` (schedule_batch.py:2054/:2089) --
sgl-project#875's active territory. Reported, not touched, per Posten 4. Unlike the
sgl-project#813/sgl-project#852/sgl-project#715 convergence I rejected last round, this one IS a shared root,
and I am saying so because the evidence supports it, not because it tidies.

POSTEN 3 -- `scripts/window_871a_verify.py`: one call, PASS/FAIL with numbers.

Decides all three open claims from a boot log plus the cgroup. Exit 0 all
passed, 1 a decided negative, 2 UNDECIDED -- and 2 is never a pass: "the
evidence was not there" is a different fact from "the claim is false", and it
sends the reader back to the boot. It boots nothing, claims no card, restores
no serving. It also prints the sgl-project#721 HOST-LEDGER pair (posts + memory.current /
peak / anon / file, before vs after) that Posten 1 needs.

IT SHIPPED A FALSE PASS AND I CAUGHT IT BEFORE THE WINDOW, which is the whole
argument for mock-smoking a window script. The first version summed `acked=`
over the WHOLE log and returned PASS on the W40 boot -- the very boot in which
all 21 fences reported `acked=0`. Seven lines there carry `acked=` from an
unrelated subsystem, three of them `acked=24`, so it credited 72
acknowledgements no fence ever made. A window script that reports a false PASS
is worse than none: it CLOSES an open claim. Now scoped to the fence's own
lines, with that exact log shape as a permanent self-test case.

The strongest desk check available is wired in: run against the real W40 log it
must reproduce that boot's known split -- tier armed PASS, store delivery FAIL,
exit 1 -- and it does.

TESTS (hermetic; no boot; no card touched).
CUDA_VISIBLE_DEVICES="" verified at the process and
`nvidia-smi --query-compute-apps` empty throughout; GPUs 0/0/0. Runner reported
its exclusions.
* the script's own --self-test decides SIX cases in the intended direction,
  both polarities, including the false-PASS regression.
* can-fail by mutation: un-scoping the acked regex reddens the real-boot
  reproduction and the self-test (2 red); restored green.
* A SECOND SELF-INFLICTED DEFECT FOUND AND FIXED: the test located the script
  by counting `dirname` calls, landed on `test/` instead of the repo root, and
  failed all six cases -- a green arm red for a reason unrelated to the thing
  under test. It now walks up to the root.
* partitioned tier-2 gate: 0 failing, 4421 passed (wide 3775 / narrow 292 /
  serial 354). Serial +6 over 348 is exactly the six tests added here.
  names=0 agrees with the summary on every lane.
* test/registered/unit/mem_cache: 2 failed / 1802 passed -- the known
  pre-existing test_acceptance_emitters_758 RefillTiming pair, unchanged.

POSTEN 4 -- boundary held. Nothing in schedule_batch.py, phase_flip_spill.py or
seam_kv_recover was read into or written. Both Posten 2's root and A's remedy
lead there; both are reported rather than resolved.

NOT ESTABLISHED. Which cgroup bucket the phase-flip pin's bytes land in -- the
question Posten 1 turns on -- needs the boot. Whether the admission ever
refuses on this box, likewise. And sgl-project#721's 17 oom_kills are from a previous
boot: today's counters are zero at every level, so nothing here re-measures
that incident, it only records that the enforcement it implies is absent.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
THE QUESTION WAS "DOES sgl-project#721'S HOST-POSTEN-LEDGER EXIST", AND THE ANSWER IS (a)
WITH TWO NAMED HOLES -- NEITHER OF WHICH IS WHAT sgl-project#721 ASKED FOR.

IT EXISTS AND IT IS WIRED. `mem_cache/pinned_host_budget.py` is a named,
individually-priced posts ledger, and it IS a hard boot precondition: three
paths reach `joint_pinned_host_error` from `ServerArgs.__post_init__` and raise
`ValueError` --
  server_args.py:6937 -> _handle_kv_session_offload      -> :8173
  server_args.py:6962 -> _handle_phase_flip              -> :8416 -> :8961
  server_args.py:7019 -> _handle_hicache                 -> :8916 -> :9029
Four posts are priced there over CONFIGURED numbers (HiCache host tier, HiCache
staging host tier x ranks, phase-flip rotation staging ring x ranks,
kv-session-offload spill pool), and eight more register at runtime in the
workers (memory_pool_host x4, read_buffer_pool, weights_arena, and the two
halves of the phase-flip staging pin this branch added in db1b07e). So it is
not (b) "nobody calls it" and not (c) "an intention".

BUT sgl-project#721'S TWO PRESCRIPTIONS WERE BOTH DECLINED, AND BOTH DECLINES ARE RIGHT.
  * "16G floor against `free -g`" -- not implemented, and must not be:
    `free -g` is lxcfs-synthesised in this container. That was settled in
    #871b, which moved the boot gate onto the sgl-project#407 owner instead.
  * "sonst Posten schrumpfen" -- deliberately not implemented. The module's own
    charter: "The guard SUMS, it never caps. Nothing in this module shrinks a
    request. Silently capping a pinned pool moves the failure from a boot-time
    message that names both posts to a later allocation whose victim the OOM
    killer chooses." That is a better answer than the measure it declines, and
    it is the reason the ledger prices every post separately -- so an operator
    can shrink the right flag himself.

THE ONE REAL DEFECT, AND IT IS A COUNTER DESCRIBED AS AN ACTUATOR (sgl-project#464/sgl-project#578/
sgl-project#852). `check_and_register_pinned_post` carried this, about the largest
claimant in the system:

    "`_register_image_post` declares the post BEFORE `_alloc_host_image`
     allocates ... deliberately, so the registry refuses an over-commitment at
     the DECLARATION rather than discovering it at the allocation."

THE REGISTRY REFUSES NOTHING THERE. `register_pinned_post` is a bare dict write
with no comparison in it; `_register_image_post`'s own first line says "Never
raises"; and `grep -c check_and_register_pinned_post weights_arena.py` -> 0. So
the ~27 GiB of phase-flip host weight images are COUNTED, never ADMITTED.

THE BEHAVIOUR IS CORRECT AND STAYS. sgl-project#695 chose it at the call site and wrote
why: "Registered, not CHECKED: a new refusal path here could break a boot that
works today, and the diagnosis this is for is served by the number being
present, not by a veto." Registering early still makes those bytes visible to
every LATER admission, which is the half that matters. Only the comment was
wrong -- and a comment promising a veto that does not exist is worse than none,
because the next person sizing a pinned pool against this module would leave
exactly the headroom that post already spent.

WHY A TEST AND NOT JUST A RE-COMMENT. Prose cannot be regression-tested; the
CONTRACT can. The split is now pinned in both directions -- counting never
refuses however absurd the post, admitting refuses and names every post, a
refused post is not left in the registry, and weights_arena uses the counting
entry point only. If someone later wires the admitting one in there, that is a
real decision with a boot-refusal consequence and it surfaces here instead of
in a boot.

A RULE I PROVED ON MYSELF, THIRD INSTANCE IN ONE SESSION. The first version of
the prose test asserted the false sentence was ABSENT from the module -- and
failed against the FIXED file, because a correction has to QUOTE the claim it
corrects. Same shape as #871a's floor-promise assertion and #871b's
/proc/meminfo probe assertion. The general rule, now written into the test:
ABSENCE OF PROSE IS NOT TESTABLE where a correction must restate what it
corrects. Prose gets one positive marker; the behaviour tests carry the
contract.

THE WEIGHTS TRANSIENT, DECIDED WITH REASONS AS ASKED. It is a SPIKE, not a
standing post, and the existing treatment is already the right compromise: it
is registered at declaration (so concurrent posts see it), and taken back both
when the allocation raises (weights_arena.py:1044) and when the image is freed
(:1082). Booking it as a permanent resident would refuse too early; leaving it
invisible would be the sgl-project#721 outcome. What it is NOT is admitted -- and turning
that on is a behaviour change with a boot-refusal consequence that needs a
loaded box to justify, not a desk.

LOAD REMAINS UNMEASURED, AND THAT IS IN THE CODE, NOT ONLY HERE. #871b's 0.24
GiB delta between the raw and honest readings is an IDLE measurement. A
precondition gate bites under load. No load figure is asserted anywhere as a
threshold.

TESTS (hermetic; no boot; no card touched). CUDA_VISIBLE_DEVICES="" verified at
the process, `nvidia-smi --query-compute-apps` empty throughout, cards 0/0/0.
* red-first: 1 of 6 red against the uncorrected module, extracted == summary.
* can-fail in the danger direction: making the COUNTING entry point refuse
  (a counter grown into an actuator) reddens
  `test_counting_never_refuses_however_absurd_the_post`. Green restored.
* turnkey + this suite together: 175 passed, 0 failed.
* partitioned tier-2 gate: wide 3775 / narrow 292 / serial 353 passed, 24
  exclusions reported by the runner, names=1 agrees with the summary on every
  lane. The single failure --
  test_pp_admission_wraparound_never_blocks::test_blocking_wraparound_wedges_the_ring
  -- is PRE-EXISTING: on the clean base 726f11e that module fails TWO tests,
  i.e. MORE than here, and it touches nothing in this commit.
* test/registered/unit/mem_cache: 2 failed / 1808 passed. The two are the known
  test_acceptance_emitters_758 RefillTiming pair; +6 over 1802 is exactly this
  suite.

NOT ESTABLISHED. Whether the ~27 GiB image post SHOULD be admitted -- that needs
a box tight enough for the answer to differ, i.e. load. Whether the four
launcher posts plus eight runtime posts are the COMPLETE set of pinned
claimants: I enumerated every `PinnedHostPost(` and every register call in the
tree, but a pinned allocation that never posts at all is invisible to that
method by construction. And ZFS cache, named in sgl-project#721's list, is kernel-side and
cannot be a process post in this ledger at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant