Skip to content

Fix trt benchmark - #697

Merged
zhyncs merged 3 commits into
mainfrom
fix-trt-bench
Jul 22, 2024
Merged

zhyncs merged 3 commits into
mainfrom
fix-trt-bench

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

No description provided.

Comment thread python/sglang/bench_serving.py Resolved

@zhyncs zhyncs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we use min_length, does it have the same effect as we expected with ignore_eos, in terms of the number of output tokens from the end?

@zhyncs
zhyncs merged commit df7c4c1 into main Jul 22, 2024
@zhyncs
zhyncs deleted the fix-trt-bench branch July 22, 2024 13:32
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
* Create sync-models-datasets.yml

* Create pre-downloaded-datasets.ini

* Create pre-downloaded-models.ini

* Update sync-models-datasets.yml
cherryblo added a commit to cherryblo/sglang-project that referenced this pull request Jul 2, 2026
Update test_npu_performance_utils.py
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…cords

THE SIGNATURE NAMED THE DELETER. The cache directory held no live seam records
and 56 `.bak-YYYYMMDD-HHMMSS` copies whose timestamps matched the
disappearances -- 15:06:10 for the mid-soak one F4-r4 saw. Exactly one place
writes that name:

    rigmon/kvbudget.py  reset_budget()   copy2(path, path + ".bak-<ts>")
                                         os.remove(path)

reached automatically, not by hand, from:

    planner/runner.py  neutralise_kv_budget()   "Run before each [boot]"
                         for path in lister(cache_dir): resetter(path)

THE BUG IS THE LISTER'S REACH. `list_budget_files` selected
`startswith("kv_budget-") and endswith(".json")`. The seam records are
`kv_budget-<digest>-seam-rank<N>.json`, so they matched a sweep meant for one
file -- and `RunPolicy` says so itself, in the field that drives it:

    #: sgl-project#188: clear ``kv_budget-<hash>.json`` before every point.

Intent and implementation disagreed; that is what makes this a defect rather
than a policy. It fires on the DEFAULT policy (`reset_kv_budget=True`,
`pin_token_vector=None`), which is why it recurred three times in one day.

WHY IT COSTS MORE THAN A FILE. The two artifacts have different lifecycles. The
token-vector budget is per-boot and MEANT to be cleared so an A/B arm
re-measures. The seam records are the two-boot protocol's memory: clearing them
does not make the next boot re-measure, it makes it size with no measurement at
all. That is the 12:04 cold boot taking the oversized 550000 pin into an OOM
while loading the NEXTN weights.

FIXED BY SHAPE, NOT BY NAME. A budget file is `kv_budget-<digest>.json` with
nothing between digest and suffix; a sub-artifact keyed off the same digest
carries a further `-`. Excluding "-seam-" specifically would fix today's
artifact and miss the next one; requiring the digest to be the whole stem
excludes every `kv_budget-<digest>-<anything>.json` without having to know it
exists. The digest is a hash and never contains a `-`.

SECOND LOCK: `reset_budget` now refuses a path that is not a budget file. The
lister no longer offers them, but a caller can still hand one over and the cost
of being wrong is a cold boot rather than a re-measurement.

FOREIGN CLEANERS RULED OUT, since the brief asked: no systemd-tmpfiles unit
covers /root/.cache, no cron entry touches sglang or the cache, and all 56
.bak files carry our own signature. No inotify watch was needed -- the
filenames convicted it.

TESTS. Red-first: 5 of 9 fail, including the end-to-end case on the real
callsite with the real lister and resetter -- "kv_budget-...-seam-rank0.json
was swept by the planner's pre-boot budget reset", which is sgl-project#697 reproduced
hermetically. Covered: the budget file still listed and still reset; the seam
records neither listed nor reset; nothing else in the cache claimed; a
hypothetical FUTURE sub-artifact excluded by the same shape rule; the direct
`reset_budget` refusal with its reason; and the pinned policy still touching
nothing.

rigmon + planner/test_runner 381 passed. planner 2574 passed, 2 failed (the
same pre-existing test_rejected_evidence_pins pair). Hermetic. NO DEPLOY.

OPERATIONAL, not fixed by this commit: the live records are STILL GONE right
now, so the next boot sizes cold. The newest backup of each rank is from one
consistent 15:06:10 sweep and restoring is a copy; that is a filesystem action
on the serving line, so it is left to whoever owns the next boot rather than
taken here.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… dry run, and the vouch

The notes were written at 3c984ad and had gone stale in three ways.

Chain: 18 commits at e21e87f -> 22 at 5939d0e. Added sgl-project#697
(8fb86ef), sgl-project#441 (5e0fa1e), sgl-project#524 (5939d0e) and the revision-1 docs
commit. Runtime-touching is now 8 of 22, and sgl-project#441 is recorded as NOT runtime
despite its ticket -- it adds a handover script and one test guard.

Target: integration/r2 has moved a73a0d8 -> a157bf1, so the dry run was
re-run rather than carried over. Still zero conflicts. Scope shrank with the
target's advance: 115 -> 109 commits, 129 -> 124 files, +22731 -> +22196.
Merged-tree tests: managers 2093 passed / 0 failed; planner plus the three new
test files 2594 passed / 2 failed. Those 2 are PpWithSpecEvidenceTest in
test_rejected_evidence_pins.py and were re-verified as pre-existing by
aborting the merge and running that file on clean a157bf1 -- same 2 failed,
2 passed. The chain introduces no new failure. Because the target tip moved,
this was re-checked instead of quoting revision 1's finding.

Holds: sgl-project#690 (e21e87f) and sgl-project#697 (8fb86ef) are in F4-r4's bundle, so
revision 1's objection to landing sgl-project#690 ahead of the sgl-project#694 soak is spent; both
lift to group 1 on his boot commit. sgl-project#524 (5939d0e) joins the hold list --
desk-verified only, never on metal, A/B window-gated. sgl-project#685 (5301b94) stays
held on the unmade R' decision.

Vouch (new section 5): the four F4-r4 cherry-picks are byte-identical to the
originals by git patch-id --stable -- f630947/c41645c8c9,
7c58aba/ce6035884d, f1f31d2/658ea3ac11, de92bb6/84b0171fa6 -- so
preferring the originals is free and the only real risk is double attribution.
1073702 is excluded: patch-id 8595e66885ac matches nothing here, it is
diagnostic scaffolding and not mine to vouch for.

Section 6 records that 5af1531 (sgl-project#696) and c738ef5 (sgl-project#689) are F4-r4's
line, not this branch, and are listed only because they lift holds; sgl-project#696 is
what was DoSing the lanes.

Docs only. No merge performed; the dry-run worktree was removed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…cords

THE SIGNATURE NAMED THE DELETER. The cache directory held no live seam records
and 56 `.bak-YYYYMMDD-HHMMSS` copies whose timestamps matched the
disappearances -- 15:06:10 for the mid-soak one F4-r4 saw. Exactly one place
writes that name:

    rigmon/kvbudget.py  reset_budget()   copy2(path, path + ".bak-<ts>")
                                         os.remove(path)

reached automatically, not by hand, from:

    planner/runner.py  neutralise_kv_budget()   "Run before each [boot]"
                         for path in lister(cache_dir): resetter(path)

THE BUG IS THE LISTER'S REACH. `list_budget_files` selected
`startswith("kv_budget-") and endswith(".json")`. The seam records are
`kv_budget-<digest>-seam-rank<N>.json`, so they matched a sweep meant for one
file -- and `RunPolicy` says so itself, in the field that drives it:

    #: sgl-project#188: clear ``kv_budget-<hash>.json`` before every point.

Intent and implementation disagreed; that is what makes this a defect rather
than a policy. It fires on the DEFAULT policy (`reset_kv_budget=True`,
`pin_token_vector=None`), which is why it recurred three times in one day.

WHY IT COSTS MORE THAN A FILE. The two artifacts have different lifecycles. The
token-vector budget is per-boot and MEANT to be cleared so an A/B arm
re-measures. The seam records are the two-boot protocol's memory: clearing them
does not make the next boot re-measure, it makes it size with no measurement at
all. That is the 12:04 cold boot taking the oversized 550000 pin into an OOM
while loading the NEXTN weights.

FIXED BY SHAPE, NOT BY NAME. A budget file is `kv_budget-<digest>.json` with
nothing between digest and suffix; a sub-artifact keyed off the same digest
carries a further `-`. Excluding "-seam-" specifically would fix today's
artifact and miss the next one; requiring the digest to be the whole stem
excludes every `kv_budget-<digest>-<anything>.json` without having to know it
exists. The digest is a hash and never contains a `-`.

SECOND LOCK: `reset_budget` now refuses a path that is not a budget file. The
lister no longer offers them, but a caller can still hand one over and the cost
of being wrong is a cold boot rather than a re-measurement.

FOREIGN CLEANERS RULED OUT, since the brief asked: no systemd-tmpfiles unit
covers /root/.cache, no cron entry touches sglang or the cache, and all 56
.bak files carry our own signature. No inotify watch was needed -- the
filenames convicted it.

TESTS. Red-first: 5 of 9 fail, including the end-to-end case on the real
callsite with the real lister and resetter -- "kv_budget-...-seam-rank0.json
was swept by the planner's pre-boot budget reset", which is sgl-project#697 reproduced
hermetically. Covered: the budget file still listed and still reset; the seam
records neither listed nor reset; nothing else in the cache claimed; a
hypothetical FUTURE sub-artifact excluded by the same shape rule; the direct
`reset_budget` refusal with its reason; and the pinned policy still touching
nothing.

rigmon + planner/test_runner 381 passed. planner 2574 passed, 2 failed (the
same pre-existing test_rejected_evidence_pins pair). Hermetic. NO DEPLOY.

OPERATIONAL, not fixed by this commit: the live records are STILL GONE right
now, so the next boot sizes cold. The newest backup of each rank is from one
consistent 15:06:10 sweep and restoring is a copy; that is a filesystem action
on the serving line, so it is left to whoever owns the next boot rather than
taken here.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
…an exact fixed point

Identical argv on an identical commit produced three different capacities on
the binding rank of cut (31,16,17) / attn (7,4,5):

    bootA   13:38  seam record COLD        -> 430000  (pin; profiled 658150)
    bootA2  13:45  record written 13:42:40 -> 146390
    standS  13:50  record written 13:49:28 -> 161378

The state carried between them is ~/.cache/sglang/kv_budget-<digest>-seam-
rank<N>.json. max_total_tokens is not in the digest (uneven_perf.
measured_kv_budget_fingerprint_fields), so all three boots read and overwrote
the same three files -- this is the sgl-project#188 complaint arriving through the seam
record instead of the measured-budget registry, on a path sgl-project#697 deliberately
made permanent.

ROOT CAUSE. floor_allowed_tokens solves the id space from the free column the
PREVIOUS boot measured, and this boot measures its own and records it for the
next one, so capacity is a first-order recurrence in itself:

    T(n+1) = T(n) + (free_measured(n) - floor) / cell

whose fixed point is "the boot measured exactly the floor it solved for". The
model is good but not exact, so the residual never reaches zero: the last
observed step was 6 MiB on a 4808 MiB baseline. That is not convergence, it is
a ratchet -- with the residual reproduced, the untreated sequence continues
161378 -> 161992 -> 162913, growing every boot for reasons absent from the
command line.

FIX. A DIRECTIONAL noise latch: a growth smaller than
POOL_LATCH_TOLERANCE_BYTES (64 MiB) is not taken and the id space stays where
the last boot put it, which turns the asymptotic fixed point into an exact one.
A SHRINK is always taken however small, because a lower re-solve means the rank
can no longer hold its arming floor at the current size -- refusing a small
growth cannot breach a floor, refusing a small shrink can. 64 MiB is an order
of magnitude over the measured per-boot allocator noise (6-7 MiB) and a factor
of three under the corridor band's half-width (205 MiB); it is a noise floor,
not a rig-specific pin.

The latched solve logs every input it used -- free column, id space, record
provenance, floor target, cell -- so a capacity that does not move is as
readable as one that does.

Also states the cold-boot hazard where it is observable rather than in a
ticket: with no record the arena tail prices at 0, so the binding rank's
arming floor is understated (1523 vs 3226 MiB) and its pool is 4.4x too large.
A cold boot's pool is not the configuration's capacity and the log now says so.
That cliff is NOT fixed here; it is boot 1 of the two-boot protocol and remains
open.

The STORED provenance message no longer prescribes pinning --max-total-tokens
for a reproducible pool, because it is no longer needed for the noise case.

Tests: test/registered/unit/managers/test_pool_latch_determinism_782.py, built
from the rig's own recorded numbers rather than constructed ones. 7 pass.
Can-fail proven by neutralising the tolerance to 0, which restores the pre-fix
arithmetic: 4 of 7 fail, including "assert 161992 == 162913" -- the ratchet.
No regressions: 63 pass across test_phase_flip_seam_reserve,
test_seam_fingerprint_and_margin_656, test_seam_records_not_swept_697,
test_seam_holdback_707, test_flip_seam_guard_760. ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants