Skip to content

Adjust max prefix len - #980

Merged
hnyls2002 merged 2 commits into
mainfrom
max-prefix-len-adjust
Aug 8, 2024
Merged

hnyls2002 merged 2 commits into
mainfrom
max-prefix-len-adjust

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

There is no need to adjust the image_offset:

  • The image offset is calculated by later batching operations, so the previous implementation is wrong.
  • When extend_input_len < 2, there could not be an image input, so the adjust is also a dead code.

This PR also adjusts extend_input_len into a separate function. We do not adjust the prefix indices after the request matches the radix tree, as there is a bug mentioned in #948. Instead, we adjust the token_ids to be matched.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@hnyls2002
hnyls2002 enabled auto-merge (squash) August 8, 2024 00:32
@hnyls2002
hnyls2002 merged commit 2b8257f into main Aug 8, 2024
@hnyls2002
hnyls2002 deleted the max-prefix-len-adjust branch August 8, 2024 00:41
@hnyls2002 hnyls2002 mentioned this pull request Aug 8, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
qybnb pushed a commit to qybnb/sglang that referenced this pull request Jul 21, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…e mutants because the module is new

sgl-project#980 shipped 1202 lines across seven files with no tests, including a new
482-line module (distributed/pp_object_recv.py). A red-first against the
previous commit is the weak proof already reported twice in this window --
every arm would die on ImportError, equally red against a module whose
functions are all 'pass'. So each invariant is paired with a mutant that
breaks that invariant and nothing else, and the arm must go red for it.

What is asserted is NOT 'does it time out'. The module's own law is that once
the size header is received the payload is on the wire and must be taken off
it, because a receiver that gives up mid-frame and later re-posts reads a
payload AS a size and every later message on that stream is garbage. An
abandoning bound is therefore worse than no bound: the unbounded wait hangs
one rank, the abandoning one silently corrupts the stream. Every arm counts
irecv posts -- exactly one size and one payload per object, however many
steps expire and however many times the caller is handed a stall and returns.

Covered: resumption across expired steps in both halves; a stall that raises
AND leaves the frame resumable, with the same receive then completing; a
transport error propagating as itself rather than as ObjectRecvStalled (sgl-project#734,
so a dead peer stays distinguishable from a slow one); abort_after_s <= 0
restoring the exact pre-sgl-project#980 unbounded wait so the default path gains only a
log line.

MY OWN FIRST STRUCTURAL ARM WAS A FALSE POSITIVE, recorded in place rather
than quietly rewritten. It asserted 'wait(timeout' not in the module source
and went red on the module's DOCSTRING, which explains at length why it does
not do that -- so the better the hazard was documented, the more surely the
arm failed. Replaced by an AST walk for a real .wait(...) call carrying a
timeout keyword, plus a can-fail proving that detector sees a genuine one.
The prohibition matters because no runtime arm can observe it: a timed
Work.wait closes the gloo pair (sgl-project#829) and would pass hermetically while
killing a real boot on both sides.

Test: 11 defs / 11 passed at 4ea93b8 (count probe), 1.84s, hermetic --
dist.irecv and ParkedWait stubbed in the module namespace, no gloo, no CUDA.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…boot 8, at the mutation

sgl-project#988 shipped 56 lines with no tests. Unlike sgl-project#980 this one admits a genuine
behavioural red-first: the arms name no symbol it introduces (set_extend_range
predates it) and assert sgl-project#965's co-derived invariant itself --
extend_range.start == len(prefix_indices), which prepare_for_extend asserts at
the batch boundary.

The specimen is boot 8's: PrefillAdder's host load-back grows prefix_indices
in place, and before sgl-project#988 went on to the budget checks without touching
extend_range, so every early return below that line left the previous visit's
range behind a moved prefix.

The arms drive a bail-out that happens AFTER the load-back and assert the
invariant on the way out, deliberately not pinned to one specific return --
the fix's argument is that re-deriving at the mutation makes every current AND
FUTURE exit inherit a consistent shape, and per-branch patches are how sgl-project#965
was paid for twice. An arm pinned to one exit goes stale the moment another
is added.

PICKING THAT BAIL-OUT TOOK A WRONG TURN, recorded in the file. add_one_req
carries TWO near-identical 'rem_chunk_tokens is None and input_tokens >=
rem_input_tokens' returns, one before the _lock_node block and one after.
Configuring for the input budget hits the FIRST, so the prefix never moves and
the file measures nothing -- caught only by the precondition assert, which is
exactly what it is for. Nor can budget separate them: the post-load-back
input_tokens is SMALLER than the pre one, so any budget tripping the second
has already tripped the first. The truncation path (chunk budget 0) is
reachable only after the load-back, and that is the one used.

MEASURED, same file:
  4ea93b8 (pre-sgl-project#988): 2 failed, 2 passed
  8380555 (post)    : 4 passed
The reds read 'the prefix advanced to 4096 and extend_range still starts at 0'
and '(0, 8422) != (4096, 4096)'. The two pre-fix PASSES are load-bearing: the
prefix-preservation arm and the can-fail hold on the old tree too, so the
failures are the defect and not a broken harness.

Chain at 8380555: 164 defs / 164 passed across eleven suites.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 2, 2026
… group STOP, not a void

Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d):
scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused
('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0
named, batch_full_break) with a rank-local compensation:
_pp_admission_pass_voided = True + emptied decision dicts -> mixin
_pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0
pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy
while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797
return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay)
-> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's
label (log 65119) -> one output ahead for the rest of the boot -> the
pp_to_tp arm at 21:42:05 turned the debt into an unproducible output ->
sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form
sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the
second writer of the same flag.

F1 ROOT: the compensation is deleted, the detector is kept, the refusal
is a group STOP. scheduler.py get_new_batch_prefill's except now does
`raise self._pp_forwarded_schedule_stop(refusal) from refusal`;
_pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home)
is removed; the new _pp_forwarded_schedule_stop logs the kept
'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns
a RuntimeError formatted by pp_admission_congruence
.forwarded_schedule_stop_message:
  'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot}
   told=[{told}] reached=[{reached}] census={census} local={local}
   limiter={limiter} running_bs={running_bs} parked={parked}
   r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit}
   batch_full_setter={batch_full_setter}
   batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}'
Every probe is guarded (n/a on an unreadable value); the reached rids
are recorded after the admission loop before any of the three raises;
the batch_is_full setter site is recorded at each writer of the pass
(no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail,
add_one_req_NO_TOKEN). Group stop mechanism (existing, no new
collective): the RuntimeError leaves run_event_loop; run_scheduler_process
(scheduler.py 'except Exception') logs 'Scheduler hit an exception' and
parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under
SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 /
sgl-project#1071 receives or the barlink dead-peer probe.

F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's
rank-local seat-count veto is not a verdict. pp_admission_congruence
.rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on
every non-PP boot (unchanged expression) and False on a rank > 0
executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and
the three pre-loop count gates of the same arithmetic
(batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay,
no_allocatable_reqs) are gated on it. The physical allocator still
refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers).

F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the
flag is provably unreachable (pp_upstream_void_pending returns False on
every path; its final statement is `return False`) -- left with the
proof in a comment. The row-authority _row_skip_plan exit now nulls a
slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020
'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is
factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by
both sites.

Matched check (error class: a follower still ends a PP0-launched pass
silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt
-> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain
scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) ->
except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop
(:9472) -> RuntimeError (:9623); the flag's only readers are
_event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py
void guard (:8955), neither reachable from the refusal.

Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new
test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the
count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless
null, + the mixin:9389 unreachability proof): 17 passed after; on the
parent ca4c6b7 (git worktree) it is a collection error (new names).
test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the
old compensation INVERTED with the withdrawal named in each docstring
(24 passed after; the inverted 12 are red on the parent: 16 failed /
8 passed / 1 error across both files). Mutants: M1 flag write restored
in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red;
M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from
_pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers
files + test/registered/scheduler): before 103 failed / 527 passed /
1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped /
27 errors -- per-file identical except the new file (+17). ruff check
clean on all touched files; ruff format --diff hunk count unchanged vs
parent (15/14/3, pre-existing) and the two test files formatted.

Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F
'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as
'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero
'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants