Repository navigation
[AMD][DSR1] feat(dp): backpressure (lazy) DP dispatch to avoid forced-eager tail - #2456
Conversation
Offline batch inference hands the engine all requests at once; CoreManager sprayed them across DP ranks up front, so a rank that drew shorter sequences drained early, went idle, and — because DP runs in lockstep — forced the whole group onto the slow eager path (observed as a large end-of-run throughput collapse). Add an optional backpressure dispatch mode gated by ATOM_DP_DISPATCH_DEPTH (default 0 = unchanged legacy spray). When >0 it is the per-rank in-flight cap: load-balanced seqs wait in a manager-side pending pool and are released to the lightest rank only up to the cap, then re-fed on each completion. A fast rank keeps getting work instead of running dry, so all ranks drain together and the eager tail is eliminated. - _bp_pending deque + _drain_pending_locked + _bp_dispatch_or_refill - backpressure branch in _dispatch_to_dp_ranks (plain load-balanced batches only; explicit hints / sticky sessions unchanged) - refill hook on offline completion path; pending cleared in reset_dp_router - _input_send_lock serializes input_sockets sends (refill now runs on the per-rank output threads too; a ZMQ socket is not thread-safe) - class-level _bp_depth/_input_send_lock defaults so bare __new__ test objects and _init_shared_state-skipping subclasses keep working Measured on DeepSeek-R1 MXFP4 Offline, 4x MI455X (gfx1250), depth 1860: +39% tokens/sec (38.4k -> 53.5k), ~34 min faster, VALID; accuracy unchanged. tests: add tests/test_dp_backpressure.py (cap / refill / full-drain); existing tests/test_dp_load_balance.py unaffected (38 passed). Co-authored-by: Cursor <cursoragent@cursor.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
|
Please fix format issue. |
Black wraps the refill print and wants a blank line after the module docstring. Ruff RUF017 rejects sum(mgr.sent, []) as a quadratic flatten. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The approach is right and the end-to-end numbers are unusually solid for a scheduling change (+39.4% tok/s, −34 min, both arms LoadGen VALID). I checked the concurrency carefully and it holds up: lock order is acyclic, sends happen outside 1. The stated mechanism is inverted
unified = not sync.any_rank_has_prefill
replayable = (not is_prefill) and unified and not (enforce_eager or step_needs_eager) and on_ladder
use_cudagraph = replayable and shape_capturedwith its own comment: "One meaning and only one: every rank is decoding … A prefilling peer makes the step ragged across the group." So decode-only across the group is the condition that permits replay, not what breaks it. And an idle rank doesn't break it either: _execute_dummy_batch → dummy_execution (model_runner.py:1261) builds a decode-shaped batch (seq.type = SequenceType.DECODE, only total_tokens_num_decode set), so is_prefill is False and any_rank_has_prefill stays clear — that docstring says it outright, "an idle rank's dummy is unified with them". What actually takes the group off the graph is a prefill on any rank. Reading the code, the tail chain looks more like: each rank has its own long queue, so steady state is mostly decode steps (replayed) punctuated by prefill steps (whole group eager); a rank that finishes its share early goes to dummies — wasting its capacity but not hurting replay — while the remaining ranks still have deep queues and keep opening new prefills, each of which puts the whole group, dummy rank included, on the eager path. The fix works because holding work back makes every rank finish prefilling at roughly the same time, so the final stretch can be all-decode and stay on the graph. Worth fixing the wording, since operators will read that sentence to decide whether their workload benefits and how to size depth — under the stated model one would conclude "just avoid idle ranks" and set it too low. Related: if the 7× could be split into "share of eager steps" vs "drop in average batch width", it would be much easier to tell how much of the +39.4% comes from eliminating eager steps and how much from simply refilling the batch. 2. _drain_pending_locked hardcodes least_requests and bypasses _select_dp_rank_lockedbest = min(range(n), key=lambda r: (self._rank_reqs[r], r)) _select_dp_rank_locked is the repo's one rank-selection entry point and supports three strategies, plus a rotating cursor whose docstring states its purpose: "Fully-tied ranks are resolved by a rotating cursor so selection does not always fall on rank 0." The new function drops all of it — so with depth > 0, ATOM_DP_LB_STRATEGY=least_tokens and round_robin become no-ops with no warning, the default strategy loses its token tie-break, and ties always land on the lowest rank index. Using request count as the primary key is right (your docstring's "keeps the lockstep DP ranks in phase"), but that doesn't require discarding the other two strategies. Could _drain_pending_locked call _select_dp_rank_locked() and layer an "already at cap, skip" filter on top? Failing that, a warning when the configured strategy is being overridden. 3. ATOM_DP_DISPATCH_DEPTH is not in envs.py and not documentedEvery other ATOM_DP_* variable is in the environment_variables registry and has a row in docs/environment_variables.md; this PR touches neither file. Three consequences:
4. _send_request's docstring is now falseIt still opens with "Deliberately unsynchronized: input_sockets carries nothing but add_request, which runs on a single thread" and closes with "never send to input_sockets from anywhere but here", calling that separation "load-bearing, not stylistic" and spelling out the failure mode (interleaved frames → the engine's input thread dies on UnpicklingError, nothing recovers, nothing is logged). This PR adds a second class of writer. The code is safe — the lock serializes it — but the correction appears only as a comment inside the body, and the original invariant wasn't "take a lock", it was "one socket per writer class" (which is why _send_control exists). Choosing one socket plus a lock is a legitimate alternative, but please rewrite that docstring to state the new constraint and why a separate socket wasn't used. The next person adding a third writer will read the old text and skip the lock. Smaller
|
The tail leaves CUDA-graph replay because a prefill on any rank makes the lockstep group eager, not because an idle rank's dummy batch does. Say that in the comments, register ATOM_DP_DISPATCH_DEPTH, and warn on a non-integer, negative, or below-max_num_seqs value. Rewrite _send_request for the output thread refill under _input_send_lock, warn when reset drops a non-empty pending pool, and drop the test prints and __main__ runner. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the backpressure deque import together with main's contextmanager import.
|
Thanks for the review. The wording, the env registration, the send docstring, and the small cleanups are in 6b55087, with main merged on top.
Smaller items. reset_dp_router warns when it drops a non-empty pending pool. The test prints and the main runner are gone. |
|
Verified all three on #2 looks like it got skipped (your reply goes 1 → 3 → 4), but I'd narrow it — and a refactor isn't the right fix. The drain loop fills every rank to the cap, so the picker has almost no freedom: initial drain starts all ranks at zero and round-robins by construction, and on refill the completing rank is strictly lightest with no tie. Per-rank counts come out identical whatever the tie-break — so my rotating-cursor point was overstated, withdraw it. So the only real issue is that nothing says the knob was taken over. One line at init when |
…tegy ATOM_DP_DISPATCH_DEPTH balances on in-flight request count, so round_robin and least_tokens do not apply. Say so at init when depth is positive and the configured strategy is not least_requests. Co-authored-by: Cursor <cursoragent@cursor.com>
ZhangLirong-amd
left a comment
There was a problem hiding this comment.
Thanks for the work — the approach and the E2E numbers look good, and the earlier review items are addressed. Three correctness issues on the new dispatch path need fixing before merge (found by reading the code on 90b0a7c; not reproduced on hardware).
1. Partial send failure in _bp_dispatch_or_refill leaks charges on later ranks
_drain_pending_locked charges seqs for every rank under _lb_lock, then the send loop does:
except Exception:
for seq in rank_seqs:
self._release_seq_load(seq.id) # only the failing rank
if raise_on_error:
raise # ranks after it are skippedOn the add path, if rank r fails, ranks r+1..n-1 stay charged but are never sent and never released, so _rank_reqs skews permanently and those seqs are lost. Also, the rest of new_seqs is still in _bp_pending and will be dispatched later by refills, even though the caller got an exception and thinks the add failed.
The legacy _dispatch_to_dp_ranks handles this with dispatched[]. Suggest: try every rank, and for any rank that didn't go out, release its charge and put its seqs back at the head of _bp_pending (extendleft(reversed(rank_seqs))) instead of dropping them.
2. Refill-path send failure drops seqs → offline generate() hangs
With raise_on_error=False the seqs are only logged as "dropped". LLMEngine.generate() loops on not is_finished() and (is_alive() or is_rest()). The engine is still alive and the dropped seqs never finish, so the call spins forever with no error. Re-queueing them as in #1 fixes this too. If a seq really can't be sent, it needs a terminal output (or the run should fail), not a silent drop.
3. Online requests also take this path, and abort_request can't reach pending seqs
The comment says "which is the offline path", but the gate only checks "no rank hints and no session affinity". OpenAI API-server requests carry no data_parallel_rank and affinity is off by default, so with the env set, online traffic goes through _bp_pending too. Then:
abort_request(req_id)on a pending seq:_release_seq_loadis a no-op (not charged yet), and the broadcast abort reaches no engine that holds it. The seq is later dispatched and runs to completion for a disconnected client.- Requests in the pool are invisible to engine-side waiting metrics and the scheduler.
Please either (a) gate the feature explicitly to offline generate() (e.g. a flag/entry point the offline path sets), or (b) make abort_request remove the seq from _bp_pending and deliver a terminal output via _deliver_terminal_output. Either way, the docs should match the actual behavior.
Tests
Please add coverage for these: a mid-loop send failure (counters return to zero, no seq lost or duplicated), and abort of a still-pending seq. The concurrent add + release/refill test requested earlier would also be good to include.
Minor, non-blocking: the per-refill logger.info("bp-dispatch ...") fires on every ADD completion on the output-thread hot path (tens of thousands of lines for a 70k-sample run). Consider debug or rate-limiting.
A partial send charged later ranks and then raised, and a refill failure dropped seqs so offline generate spun forever. Re-queue unsent seqs, pause refills after a failed add, and fail the run when nothing is left in flight. Admit the pending pool only from offline generate, and abort a request that is still waiting there. Co-authored-by: Cursor <cursoragent@cursor.com>
The send handler re-raises, so the BLE001 suppression was unused, and the pending-head check should use next() instead of a one-element slice. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep ATOM_DP_DISPATCH_DEPTH beside the new prefill-balance rows, and use main's session-affinity header order. Co-authored-by: Cursor <cursoragent@cursor.com>
Motivation
Offline batch inference hands the engine all requests at once.
CoreManagerdispatches them across the DP ranks up front, and once a sequence is assigned to
a rank it never moves. A rank that happens to draw shorter sequences therefore
drains its share early and goes idle while peers are still busy. Because DP
attention runs in lockstep (every rank must step together), one idle/decode-only
rank forces the whole group off the captured CUDA-graph path onto eager
execution — producing a large end-of-run throughput collapse (observed ~7×
slower decode for the tail of the run).
The goal of this PR is to keep every DP rank fed until the global workload is
nearly exhausted, so all ranks drain together and the forced-eager tail is
eliminated — without changing any default behavior.
Technical Details
Adds an optional backpressure (lazy) dispatch mode to
CoreManager, gated bythe env var
ATOM_DP_DISPATCH_DEPTH(default0= unchanged legacy up-frontspray). When
> 0, it is the per-rank cap on in-flight (charged-but-unfinished)requests:
_bp_pending)instead of being sprayed immediately.
_drain_pending_locked()releases seqs to the lightest rank (fewest in-flightrequests) only up to the cap, keeping the lockstep ranks in phase.
_bp_dispatch_or_refill()runs on add (request thread) and after everycompletion (per-rank output thread), re-feeding the rank that just freed a
slot — so a fast rank keeps pulling work instead of running dry.
data_parallel_rankhints and sticky-session affinity are routed exactly as before.
reset_dp_router()clears the pending pool between offline batches._input_send_lockserializes allinput_socketssends, because refill nowalso sends from the per-rank output threads and a ZMQ socket is not
thread-safe. Uncontended (zero cost) when the feature is disabled.
_bp_depth/_input_send_lockdefaults so bare__new__instances (unit tests) and any subclass that skips
_init_shared_statecontinue to work.
Changed file:
atom/model_engine/engine_core_mgr.py(+129 / −3).Test Plan
tests/test_dp_load_balance.py— existing DP routing suite, to confirm noregression to
least_requests/least_tokens/ round-robin / sessionaffinity / load accounting.
tests/test_dp_backpressure.py(new) — CPU-only unit tests building a bareCoreManagerwith a stubbed_send_request, asserting: (1) initial dispatchrespects the per-rank cap and holds the remainder in the pending pool,
(2) a completion re-feeds the rank that drained so a fast rank processes more
total requests, (3) every seq is dispatched exactly once with nothing stuck.
samples,
ATOM_DP_DISPATCH_DEPTH=1860(=max_num_seqs1536 + headroom),compared against the identical config with the feature disabled.
Test Result
Unit tests: 38 passed (35 existing + 3 new backpressure).
End-to-end (depth 1860 vs. legacy spray, same config):