Repository navigation
Fix trt benchmark - #697
Merged
Merged
Conversation
zhyncs
reviewed
Jul 22, 2024
zhyncs
reviewed
Jul 22, 2024
zhyncs
left a comment
Contributor
There was a problem hiding this comment.
When we use min_length, does it have the same effect as we expected with ignore_eos, in terms of the number of output tokens from the end?
merrymercy
force-pushed
the
fix-trt-bench
branch
from
July 22, 2024 13:24
f4b8aeb to
a68b905
Compare
zhyncs
approved these changes
Jul 22, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
cen121212
pushed a commit
to cen121212/sglang
that referenced
this pull request
Nov 10, 2025
* Create sync-models-datasets.yml * Create pre-downloaded-datasets.ini * Create pre-downloaded-models.ini * Update sync-models-datasets.yml
cherryblo
added a commit
to cherryblo/sglang-project
that referenced
this pull request
Jul 2, 2026
Update test_npu_performance_utils.py
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…cords
THE SIGNATURE NAMED THE DELETER. The cache directory held no live seam records
and 56 `.bak-YYYYMMDD-HHMMSS` copies whose timestamps matched the
disappearances -- 15:06:10 for the mid-soak one F4-r4 saw. Exactly one place
writes that name:
rigmon/kvbudget.py reset_budget() copy2(path, path + ".bak-<ts>")
os.remove(path)
reached automatically, not by hand, from:
planner/runner.py neutralise_kv_budget() "Run before each [boot]"
for path in lister(cache_dir): resetter(path)
THE BUG IS THE LISTER'S REACH. `list_budget_files` selected
`startswith("kv_budget-") and endswith(".json")`. The seam records are
`kv_budget-<digest>-seam-rank<N>.json`, so they matched a sweep meant for one
file -- and `RunPolicy` says so itself, in the field that drives it:
#: sgl-project#188: clear ``kv_budget-<hash>.json`` before every point.
Intent and implementation disagreed; that is what makes this a defect rather
than a policy. It fires on the DEFAULT policy (`reset_kv_budget=True`,
`pin_token_vector=None`), which is why it recurred three times in one day.
WHY IT COSTS MORE THAN A FILE. The two artifacts have different lifecycles. The
token-vector budget is per-boot and MEANT to be cleared so an A/B arm
re-measures. The seam records are the two-boot protocol's memory: clearing them
does not make the next boot re-measure, it makes it size with no measurement at
all. That is the 12:04 cold boot taking the oversized 550000 pin into an OOM
while loading the NEXTN weights.
FIXED BY SHAPE, NOT BY NAME. A budget file is `kv_budget-<digest>.json` with
nothing between digest and suffix; a sub-artifact keyed off the same digest
carries a further `-`. Excluding "-seam-" specifically would fix today's
artifact and miss the next one; requiring the digest to be the whole stem
excludes every `kv_budget-<digest>-<anything>.json` without having to know it
exists. The digest is a hash and never contains a `-`.
SECOND LOCK: `reset_budget` now refuses a path that is not a budget file. The
lister no longer offers them, but a caller can still hand one over and the cost
of being wrong is a cold boot rather than a re-measurement.
FOREIGN CLEANERS RULED OUT, since the brief asked: no systemd-tmpfiles unit
covers /root/.cache, no cron entry touches sglang or the cache, and all 56
.bak files carry our own signature. No inotify watch was needed -- the
filenames convicted it.
TESTS. Red-first: 5 of 9 fail, including the end-to-end case on the real
callsite with the real lister and resetter -- "kv_budget-...-seam-rank0.json
was swept by the planner's pre-boot budget reset", which is sgl-project#697 reproduced
hermetically. Covered: the budget file still listed and still reset; the seam
records neither listed nor reset; nothing else in the cache claimed; a
hypothetical FUTURE sub-artifact excluded by the same shape rule; the direct
`reset_budget` refusal with its reason; and the pinned policy still touching
nothing.
rigmon + planner/test_runner 381 passed. planner 2574 passed, 2 failed (the
same pre-existing test_rejected_evidence_pins pair). Hermetic. NO DEPLOY.
OPERATIONAL, not fixed by this commit: the live records are STILL GONE right
now, so the next boot sizes cold. The newest backup of each rank is from one
consistent 15:06:10 sweep and restoring is a copy; that is a filesystem action
on the serving line, so it is left to whoever owns the next boot rather than
taken here.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… dry run, and the vouch The notes were written at 3c984ad and had gone stale in three ways. Chain: 18 commits at e21e87f -> 22 at 5939d0e. Added sgl-project#697 (8fb86ef), sgl-project#441 (5e0fa1e), sgl-project#524 (5939d0e) and the revision-1 docs commit. Runtime-touching is now 8 of 22, and sgl-project#441 is recorded as NOT runtime despite its ticket -- it adds a handover script and one test guard. Target: integration/r2 has moved a73a0d8 -> a157bf1, so the dry run was re-run rather than carried over. Still zero conflicts. Scope shrank with the target's advance: 115 -> 109 commits, 129 -> 124 files, +22731 -> +22196. Merged-tree tests: managers 2093 passed / 0 failed; planner plus the three new test files 2594 passed / 2 failed. Those 2 are PpWithSpecEvidenceTest in test_rejected_evidence_pins.py and were re-verified as pre-existing by aborting the merge and running that file on clean a157bf1 -- same 2 failed, 2 passed. The chain introduces no new failure. Because the target tip moved, this was re-checked instead of quoting revision 1's finding. Holds: sgl-project#690 (e21e87f) and sgl-project#697 (8fb86ef) are in F4-r4's bundle, so revision 1's objection to landing sgl-project#690 ahead of the sgl-project#694 soak is spent; both lift to group 1 on his boot commit. sgl-project#524 (5939d0e) joins the hold list -- desk-verified only, never on metal, A/B window-gated. sgl-project#685 (5301b94) stays held on the unmade R' decision. Vouch (new section 5): the four F4-r4 cherry-picks are byte-identical to the originals by git patch-id --stable -- f630947/c41645c8c9, 7c58aba/ce6035884d, f1f31d2/658ea3ac11, de92bb6/84b0171fa6 -- so preferring the originals is free and the only real risk is double attribution. 1073702 is excluded: patch-id 8595e66885ac matches nothing here, it is diagnostic scaffolding and not mine to vouch for. Section 6 records that 5af1531 (sgl-project#696) and c738ef5 (sgl-project#689) are F4-r4's line, not this branch, and are listed only because they lift holds; sgl-project#696 is what was DoSing the lanes. Docs only. No merge performed; the dry-run worktree was removed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…cords
THE SIGNATURE NAMED THE DELETER. The cache directory held no live seam records
and 56 `.bak-YYYYMMDD-HHMMSS` copies whose timestamps matched the
disappearances -- 15:06:10 for the mid-soak one F4-r4 saw. Exactly one place
writes that name:
rigmon/kvbudget.py reset_budget() copy2(path, path + ".bak-<ts>")
os.remove(path)
reached automatically, not by hand, from:
planner/runner.py neutralise_kv_budget() "Run before each [boot]"
for path in lister(cache_dir): resetter(path)
THE BUG IS THE LISTER'S REACH. `list_budget_files` selected
`startswith("kv_budget-") and endswith(".json")`. The seam records are
`kv_budget-<digest>-seam-rank<N>.json`, so they matched a sweep meant for one
file -- and `RunPolicy` says so itself, in the field that drives it:
#: sgl-project#188: clear ``kv_budget-<hash>.json`` before every point.
Intent and implementation disagreed; that is what makes this a defect rather
than a policy. It fires on the DEFAULT policy (`reset_kv_budget=True`,
`pin_token_vector=None`), which is why it recurred three times in one day.
WHY IT COSTS MORE THAN A FILE. The two artifacts have different lifecycles. The
token-vector budget is per-boot and MEANT to be cleared so an A/B arm
re-measures. The seam records are the two-boot protocol's memory: clearing them
does not make the next boot re-measure, it makes it size with no measurement at
all. That is the 12:04 cold boot taking the oversized 550000 pin into an OOM
while loading the NEXTN weights.
FIXED BY SHAPE, NOT BY NAME. A budget file is `kv_budget-<digest>.json` with
nothing between digest and suffix; a sub-artifact keyed off the same digest
carries a further `-`. Excluding "-seam-" specifically would fix today's
artifact and miss the next one; requiring the digest to be the whole stem
excludes every `kv_budget-<digest>-<anything>.json` without having to know it
exists. The digest is a hash and never contains a `-`.
SECOND LOCK: `reset_budget` now refuses a path that is not a budget file. The
lister no longer offers them, but a caller can still hand one over and the cost
of being wrong is a cold boot rather than a re-measurement.
FOREIGN CLEANERS RULED OUT, since the brief asked: no systemd-tmpfiles unit
covers /root/.cache, no cron entry touches sglang or the cache, and all 56
.bak files carry our own signature. No inotify watch was needed -- the
filenames convicted it.
TESTS. Red-first: 5 of 9 fail, including the end-to-end case on the real
callsite with the real lister and resetter -- "kv_budget-...-seam-rank0.json
was swept by the planner's pre-boot budget reset", which is sgl-project#697 reproduced
hermetically. Covered: the budget file still listed and still reset; the seam
records neither listed nor reset; nothing else in the cache claimed; a
hypothetical FUTURE sub-artifact excluded by the same shape rule; the direct
`reset_budget` refusal with its reason; and the pinned policy still touching
nothing.
rigmon + planner/test_runner 381 passed. planner 2574 passed, 2 failed (the
same pre-existing test_rejected_evidence_pins pair). Hermetic. NO DEPLOY.
OPERATIONAL, not fixed by this commit: the live records are STILL GONE right
now, so the next boot sizes cold. The newest backup of each rank is from one
consistent 15:06:10 sweep and restoring is a copy; that is a filesystem action
on the serving line, so it is left to whoever owns the next boot rather than
taken here.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
…an exact fixed point
Identical argv on an identical commit produced three different capacities on
the binding rank of cut (31,16,17) / attn (7,4,5):
bootA 13:38 seam record COLD -> 430000 (pin; profiled 658150)
bootA2 13:45 record written 13:42:40 -> 146390
standS 13:50 record written 13:49:28 -> 161378
The state carried between them is ~/.cache/sglang/kv_budget-<digest>-seam-
rank<N>.json. max_total_tokens is not in the digest (uneven_perf.
measured_kv_budget_fingerprint_fields), so all three boots read and overwrote
the same three files -- this is the sgl-project#188 complaint arriving through the seam
record instead of the measured-budget registry, on a path sgl-project#697 deliberately
made permanent.
ROOT CAUSE. floor_allowed_tokens solves the id space from the free column the
PREVIOUS boot measured, and this boot measures its own and records it for the
next one, so capacity is a first-order recurrence in itself:
T(n+1) = T(n) + (free_measured(n) - floor) / cell
whose fixed point is "the boot measured exactly the floor it solved for". The
model is good but not exact, so the residual never reaches zero: the last
observed step was 6 MiB on a 4808 MiB baseline. That is not convergence, it is
a ratchet -- with the residual reproduced, the untreated sequence continues
161378 -> 161992 -> 162913, growing every boot for reasons absent from the
command line.
FIX. A DIRECTIONAL noise latch: a growth smaller than
POOL_LATCH_TOLERANCE_BYTES (64 MiB) is not taken and the id space stays where
the last boot put it, which turns the asymptotic fixed point into an exact one.
A SHRINK is always taken however small, because a lower re-solve means the rank
can no longer hold its arming floor at the current size -- refusing a small
growth cannot breach a floor, refusing a small shrink can. 64 MiB is an order
of magnitude over the measured per-boot allocator noise (6-7 MiB) and a factor
of three under the corridor band's half-width (205 MiB); it is a noise floor,
not a rig-specific pin.
The latched solve logs every input it used -- free column, id space, record
provenance, floor target, cell -- so a capacity that does not move is as
readable as one that does.
Also states the cold-boot hazard where it is observable rather than in a
ticket: with no record the arena tail prices at 0, so the binding rank's
arming floor is understated (1523 vs 3226 MiB) and its pool is 4.4x too large.
A cold boot's pool is not the configuration's capacity and the log now says so.
That cliff is NOT fixed here; it is boot 1 of the two-boot protocol and remains
open.
The STORED provenance message no longer prescribes pinning --max-total-tokens
for a reproducible pool, because it is no longer needed for the noise case.
Tests: test/registered/unit/managers/test_pool_latch_determinism_782.py, built
from the rig's own recorded numbers rather than constructed ones. 7 pass.
Can-fail proven by neutralising the tolerance to 0, which restores the pre-fix
arithmetic: 4 of 7 fail, including "assert 161992 == 162913" -- the ratchet.
No regressions: 63 pass across test_phase_flip_seam_reserve,
test_seam_fingerprint_and_margin_656, test_seam_records_not_swept_697,
test_seam_holdback_707, test_flip_seam_guard_760. ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.