Skip to content

Auto adjust new ratio - #708

Merged
hnyls2002 merged 4 commits into
mainfrom
auto-new-ratio
Jul 24, 2024
Merged

hnyls2002 merged 4 commits into
mainfrom
auto-new-ratio

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jul 24, 2024 •

Copy link
Copy Markdown
Collaborator
  • Retract decode is bad, nothing good with it.
  • When the server is idle, get a conservative new token ratio.
  • And decay the new token ratio step by step.
  • Each retraction will leave space for 20 steps(a global config) for other requests.
  • We adjust the new token ratio according to the space left.

@hnyls2002
hnyls2002 merged commit 4cd64b8 into main Jul 24, 2024
@hnyls2002
hnyls2002 deleted the auto-new-ratio branch July 24, 2024 05:06
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…rom a string

The BOTH-BLOCKED decline stated "the binding resource is KV, not the layout"
as fixed text. It was RIGHT on all three live specimens -- 19,004 avail vs
97,922 pending; 107,881 vs 160,514; 7,085 vs 217,048 -- available well under
pending every time. But a claim that cannot come out the other way is not a
diagnosis. The identical sentence would have printed with the pool standing
empty, sending the next reader to hunt the pool while the real bound sat
elsewhere.

SCOPE, corrected from my own filing: this is a ROBUSTNESS fix, not a
wrong-verdict fix. I first reported the policy as contradicting my relief line;
re-reading all three specimens showed the policy was right each time and MY
relief text was the wrong half (fixed in db07714). Filing it as a wrong
verdict over-claimed, and the ledger should carry the narrower version.

The ROUTING conclusion is deliberately untouched. When the target cannot admit
either, changing layout cannot help, so this is never a flip whichever resource
binds -- only the diagnosis is derived. test_the_routing_conclusion_never_
changes pins that across every availability value including None, because a
fix that let the diagnosis steer the routing would be a far worse bug than the
one being repaired.

RANK-UNIFORM BY CONSTRUCTION. Every PhasePolicyInputs field is replicated by
contract, and under uneven DCP local availability differs per rank -- feeding a
local value in would make the decline text, and anything later keyed on it,
rank-dependent. That is the #616g divergence class this codebase already pays
to avoid. The new input takes the group MIN via the existing accessor
uniform_avail_for_evict, and the helper pins that: it asserts the accessor is
used and that allocator.available_size() is NOT.

None means "not measured", and the line says so instead of guessing.

Tests, hermetic (CUDA_VISIBLE_DEVICES=""):
  test_the_live_specimens_still_name_KV ....... all three real specimens reach
      the same conclusion as before, now with the numbers quoted
  test_kv_not_binding_says_so_and_redirects ... CAN-FAIL, the synthetic
      KV-not-binding specimen the fix exists for; the old string fails it
  test_unmeasured_admits_it_rather_than_guessing
  test_the_routing_conclusion_never_changes
  test_boundary_equal_available_and_pending
  wiring pins: the scheduler passes the input, and the helper uses the group
      MIN accessor rather than the local pool
  -> 8 passed + 7 subtests in the file; 95 passed + 19 subtests across the
     708 + 701 + 698 + flip-runtime suites. ruff clean.

CAN-FAIL PROVEN BY MUTATION: restoring the hardcoded string failed 2 of 8;
reverted and re-verified 8 passed.

Fixture note, because it cost a red first: PhasePolicyConfig refuses to
construct without a positive flip threshold, so the first version of these
tests got "policy disabled" and then a PhasePolicyError rather than the branch
under test. The config now sets enabled=True and flip_tokens=1000 -- a test
that never reaches its branch proves nothing.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…e-key its own pin

Merging Slot-3's sgl-project#713 admission-intake work surfaced two defects of mine, both
of which I had already learned once in this same series.

1. THE STAND-IN TRAP, third time. sgl-project#708 wired the policy input as a direct call,
   self._uniform_kv_available(). Slot-3's scheduler STAND-INS carry only the
   fields the policy reads, so the merge produced nine failures with
   "'S' object has no attribute '_uniform_kv_available'". _idle_locked_inputs
   and the both-blocked relief were both given getattr defaults earlier for
   exactly this reason; I then wrote new code without applying my own lesson.
   Now getattr(self, "_uniform_kv_available", lambda: None)() -- a stand-in
   without the probe has MEASURED NOTHING, and "not measured" is a state the
   sgl-project#708 branch already reports honestly, so the degradation is free.

2. MY OWN PIN PUNISHED THE FIX. test_scheduler_passes_kv_available_tokens
   asserted the literal string "kv_available_tokens=self._uniform_kv_available()"
   and so went red on the legal refactor above -- the same way the sgl-project#698 pin
   broke when that call was made defensive. Re-keyed on the IDENTIFIER plus the
   parameter name rather than the call form. A pin must survive a legal
   refactor of the thing it pins, or it punishes the repair instead of the
   regression.

Also recorded honestly: I pushed the merge before reading its test result,
because I chained the push after pytest with ';' rather than '&&'. The branch
carried nine failures for a few minutes. Sequence the push behind the tests.

Tests: 190 passed + 21 subtests across admission-intake, phase-policy, 708,
713, 699 and flip-runtime. ruff clean in the changed region.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…d conflict map

PREPARATION ONLY. No branch pointer was moved and nothing was merged. Every
conflict is MEASURED by trial merges in a throwaway worktree
(throwaway/merge-train-probe, removed after measurement), not predicted.

Built from git rather than from the handed list, and git contradicted the list
twice:
* fix/713-admission-intake is ALREADY in the serving line (+0 commits, 0
  files) -- not a train item, and scheduling it would be a no-op step that
  looks like progress.
* fix/728-max-bytes-uniform points at 79216e6, byte-identical to
  feat/706-phase-uniform-hicache-keys, and has NO REMOTE. It must not be
  merged as a separate item, and its identity has to be settled first: a
  local-only branch is the one kind that disappears with its worktree.

DIVERGENCE. The 4222976 class turned out not to be the risk it looked like:
that commit is PRESENT in integration/r2 and absent only from upstream main,
which is true of every fork commit. The useful axis is the merge target -- 167
commits are on the serving line and not in integration/r2, and exactly SIX of
them are on no other train branch (the sgl-project#677 range c4bc982..5fed8a6, sgl-project#708
fc6f97b, and the merge 761d0d7). All six are reachable from
feat/677-park-wiring, so that branch carries them; if any step drops or
rewrites it, those six are the loss. The serving line is also NOT an orphaned
detached head -- 5fed8a6 is exactly feat/677-park-wiring, checked, because
unbranched commits on a serving head are how a train loses a fix.

ORDER. All eleven candidates are independent siblings (a containment check
found no branch containing another), so order is a conflict question. The
measured result is two clusters and a clean set:
* Cluster A, scheduler_teardown.py: all four sgl-project#673 thread-stop branches add
  their stop logic to the same file this lane created for sgl-project#673. They conflict
  with EACH OTHER, not with the trunk, so whichever lands first sets the file's
  shape. The barlink one also touches scheduler.py and belongs to live sgl-project#722.
* Cluster B, planner/seam: fix/602-fill-side, fix/701-ledger-wiring and
  feat/704-prefill-ladder each rewrote seam_slope.py, planner/pp_cut.py and
  test_pp_cut_prefill_speed_702.py. Three lanes editing one model of the same
  thing -- the conflict is SEMANTIC, and resolving it by taking hunks would
  produce a seam model nobody designed.
* The clean six-step train (621, 699, 673-lockstep, 706, 717, 677) merged in
  sequence with no conflicts at all and can run without either cluster.

TEST MATRIX, measured on the merged probe state: mem_cache 940 failed / 973
passed and distributed 21 failed / 2716 passed, both matching the standing
baselines. managers came out at 12 failed / 2274 passed against a standing
figure of 14 -- and the point is that this suite's count is
TRAIN-COMPOSITION-DEPENDENT: this lane's branch alone shows 4, the merged probe
12, with the extra 8 arriving from the sgl-project#677/sgl-project#713/sgl-project#631 lanes rather than from
the merge. The failing classes are listed so they can be attributed, and each
owner must record their own baseline before the train, or "the branch shipped
it" becomes "the merge broke it".

HELD OUT, with the separability question answered: 2ce1ed7 is NOT docs-only
-- it changes managers/phase_flip_runtime.py, which is exactly why it earns its
review boot. The two later sgl-project#441 commits touch no file it touches, so they are
cherry-pickable without it; the only entanglement on that branch is
be1fcec -> 2ce1ed7 (same NOTE). But 4512136 carries
sgl-kernel/csrc/kvcacheio/transfer.cu, so riding it means a kernel rebuild --
its own boot-gated risk, not something to smuggle in behind a docs-and-tests
framing.

Also in this push: the sgl-project#568 ledger commit (79216e6), which the audit found
was this lane's only unpushed work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants