Skip to content

[AMD][DSR1] feat(dp): backpressure (lazy) DP dispatch to avoid forced-eager tail - #2456

Merged
valarLip merged 8 commits into
ROCm:mainfrom
karverma-amd:feat/dp-backpressure-dispatch
Oct 27, 2026
Merged

valarLip merged 8 commits into
ROCm:mainfrom
karverma-amd:feat/dp-backpressure-dispatch

Conversation

@karverma-amd

Copy link
Copy Markdown
Contributor

Motivation

Offline batch inference hands the engine all requests at once. CoreManager
dispatches them across the DP ranks up front, and once a sequence is assigned to
a rank it never moves. A rank that happens to draw shorter sequences therefore
drains its share early and goes idle while peers are still busy. Because DP
attention runs in lockstep (every rank must step together), one idle/decode-only
rank forces the whole group off the captured CUDA-graph path onto eager
execution — producing a large end-of-run throughput collapse (observed ~7×
slower decode for the tail of the run).

The goal of this PR is to keep every DP rank fed until the global workload is
nearly exhausted, so all ranks drain together and the forced-eager tail is
eliminated — without changing any default behavior.

Technical Details

Adds an optional backpressure (lazy) dispatch mode to CoreManager, gated by
the env var ATOM_DP_DISPATCH_DEPTH (default 0 = unchanged legacy up-front
spray). When > 0, it is the per-rank cap on in-flight (charged-but-unfinished)
requests:

  • Load-balanced seqs are held in a manager-side pending pool (_bp_pending)
    instead of being sprayed immediately.
  • _drain_pending_locked() releases seqs to the lightest rank (fewest in-flight
    requests) only up to the cap, keeping the lockstep ranks in phase.
  • _bp_dispatch_or_refill() runs on add (request thread) and after every
    completion (per-rank output thread), re-feeding the rank that just freed a
    slot — so a fast rank keeps pulling work instead of running dry.
  • Only plain load-balanced batches use this path; explicit data_parallel_rank
    hints and sticky-session affinity are routed exactly as before.
  • reset_dp_router() clears the pending pool between offline batches.
  • _input_send_lock serializes all input_sockets sends, because refill now
    also sends from the per-rank output threads and a ZMQ socket is not
    thread-safe. Uncontended (zero cost) when the feature is disabled.
  • Class-level _bp_depth / _input_send_lock defaults so bare __new__
    instances (unit tests) and any subclass that skips _init_shared_state
    continue to work.

Changed file: atom/model_engine/engine_core_mgr.py (+129 / −3).

Test Plan

  • tests/test_dp_load_balance.py — existing DP routing suite, to confirm no
    regression to least_requests / least_tokens / round-robin / session
    affinity / load accounting.
  • tests/test_dp_backpressure.py (new) — CPU-only unit tests building a bare
    CoreManager with a stubbed _send_request, asserting: (1) initial dispatch
    respects the per-rank cap and holds the remainder in the pending pool,
    (2) a completion re-feeds the rank that drained so a fast rank processes more
    total requests, (3) every seq is dispatched exactly once with nothing stuck.
  • End-to-end: DeepSeek-R1 MXFP4 MLPerf Offline, 4× MI455X (gfx1250), 70,208
    samples, ATOM_DP_DISPATCH_DEPTH=1860 (= max_num_seqs 1536 + headroom),
    compared against the identical config with the feature disabled.

Test Result

  • Unit tests: 38 passed (35 existing + 3 new backpressure).

  • End-to-end (depth 1860 vs. legacy spray, same config):

    Metric Legacy (disabled) Backpressure Δ
    Tokens/sec 38,381 53,515 +39.4%
    Samples/sec 9.75 13.55 +39.0%
    Wall time (70,208) ~120 min ~86 min −34 min
    LoadGen result VALID VALID ✓

Offline batch inference hands the engine all requests at once; CoreManager
sprayed them across DP ranks up front, so a rank that drew shorter sequences
drained early, went idle, and — because DP runs in lockstep — forced the whole
group onto the slow eager path (observed as a large end-of-run throughput
collapse).

Add an optional backpressure dispatch mode gated by ATOM_DP_DISPATCH_DEPTH
(default 0 = unchanged legacy spray). When >0 it is the per-rank in-flight cap:
load-balanced seqs wait in a manager-side pending pool and are released to the
lightest rank only up to the cap, then re-fed on each completion. A fast rank
keeps getting work instead of running dry, so all ranks drain together and the
eager tail is eliminated.

- _bp_pending deque + _drain_pending_locked + _bp_dispatch_or_refill
- backpressure branch in _dispatch_to_dp_ranks (plain load-balanced batches
  only; explicit hints / sticky sessions unchanged)
- refill hook on offline completion path; pending cleared in reset_dp_router
- _input_send_lock serializes input_sockets sends (refill now runs on the
  per-rank output threads too; a ZMQ socket is not thread-safe)
- class-level _bp_depth/_input_send_lock defaults so bare __new__ test objects
  and _init_shared_state-skipping subclasses keep working

Measured on DeepSeek-R1 MXFP4 Offline, 4x MI455X (gfx1250), depth 1860:
+39% tokens/sec (38.4k -> 53.5k), ~34 min faster, VALID; accuracy unchanged.

tests: add tests/test_dp_backpressure.py (cap / refill / full-drain);
existing tests/test_dp_load_balance.py unaffected (38 passed).

Co-authored-by: Cursor <cursoragent@cursor.com>
@karverma-amd karverma-amd changed the title feat(dp): backpressure (lazy) DP dispatch to avoid forced-eager tail [AMD][DSR1] feat(dp): backpressure (lazy) DP dispatch to avoid forced-eager tail Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2456 --add-label <label>

@zufayu
zufayu requested a review from yitingw1 October 3, 2026 00:33
@yitingw1

yitingw1 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Please fix format issue.

Black wraps the refill print and wants a blank line after the module
docstring. Ruff RUF017 rejects sum(mgr.sent, []) as a quadratic flatten.

Co-authored-by: Cursor <cursoragent@cursor.com>
@yitingw1

Copy link
Copy Markdown
Contributor

The approach is right and the end-to-end numbers are unusually solid for a scheduling change (+39.4% tok/s, −34 min, both arms LoadGen VALID). I checked the concurrency carefully and it holds up: lock order is acyclic, sends happen outside _lb_lock, _release_seq_load (:735) runs before the refill (:753), a failed send rolls the charge back, and make_zmq_socket sets SNDHWM = 0 on the ROUTER — so the new output-thread send can neither block on a high-water mark nor be silently dropped. Four things below.

1. The stated mechanism is inverted

one idle/decode-only rank forces the whole group off the captured CUDA-graph path onto eager execution

ForwardMode.decide (atom/utils/forward_context.py:248):

unified = not sync.any_rank_has_prefill
replayable = (not is_prefill) and unified and not (enforce_eager or step_needs_eager) and on_ladder
use_cudagraph = replayable and shape_captured

with its own comment: "One meaning and only one: every rank is decoding … A prefilling peer makes the step ragged across the group."

So decode-only across the group is the condition that permits replay, not what breaks it. And an idle rank doesn't break it either: _execute_dummy_batch → dummy_execution (model_runner.py:1261) builds a decode-shaped batch (seq.type = SequenceType.DECODE, only total_tokens_num_decode set), so is_prefill is False and any_rank_has_prefill stays clear — that docstring says it outright, "an idle rank's dummy is unified with them".

What actually takes the group off the graph is a prefill on any rank. Reading the code, the tail chain looks more like: each rank has its own long queue, so steady state is mostly decode steps (replayed) punctuated by prefill steps (whole group eager); a rank that finishes its share early goes to dummies — wasting its capacity but not hurting replay — while the remaining ranks still have deep queues and keep opening new prefills, each of which puts the whole group, dummy rank included, on the eager path. The fix works because holding work back makes every rank finish prefilling at roughly the same time, so the final stretch can be all-decode and stay on the graph.

Worth fixing the wording, since operators will read that sentence to decide whether their workload benefits and how to size depth — under the stated model one would conclude "just avoid idle ranks" and set it too low. Related: if the 7× could be split into "share of eager steps" vs "drop in average batch width", it would be much easier to tell how much of the +39.4% comes from eliminating eager steps and how much from simply refilling the batch.

2. _drain_pending_locked hardcodes least_requests and bypasses _select_dp_rank_locked

best = min(range(n), key=lambda r: (self._rank_reqs[r], r))

_select_dp_rank_locked is the repo's one rank-selection entry point and supports three strategies, plus a rotating cursor whose docstring states its purpose: "Fully-tied ranks are resolved by a rotating cursor so selection does not always fall on rank 0." The new function drops all of it — so with depth > 0, ATOM_DP_LB_STRATEGY=least_tokens and round_robin become no-ops with no warning, the default strategy loses its token tie-break, and ties always land on the lowest rank index.

Using request count as the primary key is right (your docstring's "keeps the lockstep DP ranks in phase"), but that doesn't require discarding the other two strategies. Could _drain_pending_locked call _select_dp_rank_locked() and layer an "already at cap, skip" filter on top? Failing that, a warning when the configured strategy is being overridden.

3. ATOM_DP_DISPATCH_DEPTH is not in envs.py and not documented

Every other ATOM_DP_* variable is in the environment_variables registry and has a row in docs/environment_variables.md; this PR touches neither file. Three consequences:

  • a typo (186O) hits except ValueError → 0 → feature off, no log — and since the "ON" line only prints when enabled, there's nothing to notice;
  • a negative value parses fine and also silently disables it;
  • "must stay >= max_num_seqs" lives only in a code comment. Setting 8 against max_num_seqs=1536 caps every rank at 8-request batches — a catastrophic throughput loss in exactly the metric this feature exists to improve, with no hint. _init_shared_state already takes config, so one warning line would catch it.

4. _send_request's docstring is now false

It still opens with "Deliberately unsynchronized: input_sockets carries nothing but add_request, which runs on a single thread" and closes with "never send to input_sockets from anywhere but here", calling that separation "load-bearing, not stylistic" and spelling out the failure mode (interleaved frames → the engine's input thread dies on UnpicklingError, nothing recovers, nothing is logged).

This PR adds a second class of writer. The code is safe — the lock serializes it — but the correction appears only as a comment inside the body, and the original invariant wasn't "take a lock", it was "one socket per writer class" (which is why _send_control exists). Choosing one socket plus a lock is a legitimate alternative, but please rewrite that docstring to state the new constraint and why a separate socket wasn't used. The next person adding a third writer will read the old text and skip the lock.

Smaller

  • Only the ADD branch triggers a refill (:753); the STREAM branch releases the slot at :693 without one. I traced it and nothing gets stuck — online completions still reach the ADD branch — but the slot sits free from STREAM until ADD arrives.
  • _input_send_lock = Lock() as a class attribute is shared by every instance that skips _init_shared_state. Fine with one manager per process, acknowledged in the comment, but class-level mutable state is an easy trap — unlike _bp_depth = 0, which is immutable and safe.
  • reset_dp_router() clears _bp_pending silently. Harmless today (the pool is empty between offline batches), but a mid-batch reset would drop requests with no log and no signal to the caller.
  • The pending pool extends the host-side lifetime of un-dispatched Sequence objects — ~62k of the 70,208 in your run. The caller probably holds the list anyway, but given the scale it would be worth a peak-RSS number alongside the throughput table.
  • All three tests are single-threaded, while the riskiest part of this PR is the output thread refilling concurrently with the request thread adding. One test with two threads interleaving add and release-plus-refill, asserting every seq is dispatched exactly once and the per-rank charges end at zero, would cover the actual risk surface. (Also: the print()s and the main block are CI noise.)
  • The description still says "Changed file: atom/model_engine/engine_core_mgr.py (+129 / −3)" — it's 2 files, +263/−3, since the test was added.

karverma-amd and others added 2 commits October 12, 2026 15:10
The tail leaves CUDA-graph replay because a prefill on any rank makes the
lockstep group eager, not because an idle rank's dummy batch does. Say that
in the comments, register ATOM_DP_DISPATCH_DEPTH, and warn on a non-integer,
negative, or below-max_num_seqs value. Rewrite _send_request for the output
thread refill under _input_send_lock, warn when reset drops a non-empty
pending pool, and drop the test prints and __main__ runner.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the backpressure deque import together with main's contextmanager import.
@karverma-amd

Copy link
Copy Markdown
Contributor Author

Thanks for the review. The wording, the env registration, the send docstring, and the small cleanups are in 6b55087, with main merged on top.

  1. Mechanism wording. Decode-only across the group, including an idle rank's dummy batch, is what permits CUDA-graph replay. A prefill on any rank takes the whole lockstep group off the graph onto eager execution. The comments now say that. Holding work back keeps every rank fed until the global workload is nearly exhausted, so ranks finish prefilling together and the final stretch can stay all-decode on the graph. I have not split the 7× into eager-step share versus batch width in this update.

  2. ATOM_DP_DISPATCH_DEPTH. It is registered in envs.py through _nonnegative_int_env, so a non-integer or negative value warns and reads as 0. There is a row in docs/environment_variables.md. _init_shared_state warns when 0 < depth < max_num_seqs.

  3. _send_request docstring. Rewrote it. The request thread and the output-thread refill both write input_sockets under _input_send_lock. Refill reuses that socket because it sends the same ADD payload; a second ROUTER would need another engine reader. Control stays on _send_control. The docstring says every input_sockets write goes through this path.

Smaller items. reset_dp_router warns when it drops a non-empty pending pool. The test prints and the main runner are gone.

@yitingw1

Copy link
Copy Markdown
Contributor

Verified all three on 6b55087 — good. Nice touches I didn't ask for: reusing the existing _nonnegative_int_env instead of adding a helper, and fixing the stale cross-reference in build_internode_dp_socket_plan's docstring.

#2 looks like it got skipped (your reply goes 1 → 3 → 4), but I'd narrow it — and a refactor isn't the right fix.

The drain loop fills every rank to the cap, so the picker has almost no freedom: initial drain starts all ranks at zero and round-robins by construction, and on refill the completing rank is strictly lightest with no tie. Per-rank counts come out identical whatever the tie-break — so my rotating-cursor point was overstated, withdraw it. least_tokens would still pick different sequences, but overriding it is defensible: request-count balance is what lockstep needs, and continuous refill corrects imbalance dynamically.

So the only real issue is that nothing says the knob was taken over. One line at init when depth > 0 and the strategy isn't least_requests covers it. I'd skip reusing _select_dp_rank_locked — it can't express "skip ranks at the cap".

…tegy

ATOM_DP_DISPATCH_DEPTH balances on in-flight request count, so
round_robin and least_tokens do not apply. Say so at init when depth
is positive and the configured strategy is not least_requests.

Co-authored-by: Cursor <cursoragent@cursor.com>

@ZhangLirong-amd ZhangLirong-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work — the approach and the E2E numbers look good, and the earlier review items are addressed. Three correctness issues on the new dispatch path need fixing before merge (found by reading the code on 90b0a7c; not reproduced on hardware).

1. Partial send failure in _bp_dispatch_or_refill leaks charges on later ranks

_drain_pending_locked charges seqs for every rank under _lb_lock, then the send loop does:

except Exception:
    for seq in rank_seqs:
        self._release_seq_load(seq.id)   # only the failing rank
    if raise_on_error:
        raise                             # ranks after it are skipped

On the add path, if rank r fails, ranks r+1..n-1 stay charged but are never sent and never released, so _rank_reqs skews permanently and those seqs are lost. Also, the rest of new_seqs is still in _bp_pending and will be dispatched later by refills, even though the caller got an exception and thinks the add failed.

The legacy _dispatch_to_dp_ranks handles this with dispatched[]. Suggest: try every rank, and for any rank that didn't go out, release its charge and put its seqs back at the head of _bp_pending (extendleft(reversed(rank_seqs))) instead of dropping them.

2. Refill-path send failure drops seqs → offline generate() hangs

With raise_on_error=False the seqs are only logged as "dropped". LLMEngine.generate() loops on not is_finished() and (is_alive() or is_rest()). The engine is still alive and the dropped seqs never finish, so the call spins forever with no error. Re-queueing them as in #1 fixes this too. If a seq really can't be sent, it needs a terminal output (or the run should fail), not a silent drop.

3. Online requests also take this path, and abort_request can't reach pending seqs

The comment says "which is the offline path", but the gate only checks "no rank hints and no session affinity". OpenAI API-server requests carry no data_parallel_rank and affinity is off by default, so with the env set, online traffic goes through _bp_pending too. Then:

  • abort_request(req_id) on a pending seq: _release_seq_load is a no-op (not charged yet), and the broadcast abort reaches no engine that holds it. The seq is later dispatched and runs to completion for a disconnected client.
  • Requests in the pool are invisible to engine-side waiting metrics and the scheduler.

Please either (a) gate the feature explicitly to offline generate() (e.g. a flag/entry point the offline path sets), or (b) make abort_request remove the seq from _bp_pending and deliver a terminal output via _deliver_terminal_output. Either way, the docs should match the actual behavior.

Tests

Please add coverage for these: a mid-loop send failure (counters return to zero, no seq lost or duplicated), and abort of a still-pending seq. The concurrent add + release/refill test requested earlier would also be good to include.

Minor, non-blocking: the per-refill logger.info("bp-dispatch ...") fires on every ADD completion on the output-thread hot path (tens of thousands of lines for a 70k-sample run). Consider debug or rate-limiting.

karverma-amd and others added 2 commits October 16, 2026 00:57
A partial send charged later ranks and then raised, and a refill failure
dropped seqs so offline generate spun forever. Re-queue unsent seqs, pause
refills after a failed add, and fail the run when nothing is left in flight.
Admit the pending pool only from offline generate, and abort a request that
is still waiting there.

Co-authored-by: Cursor <cursoragent@cursor.com>
The send handler re-raises, so the BLE001 suppression was unused, and
the pending-head check should use next() instead of a one-element slice.

Co-authored-by: Cursor <cursoragent@cursor.com>
yitingw1
yitingw1 previously approved these changes Oct 21, 2026
Keep ATOM_DP_DISPATCH_DEPTH beside the new prefill-balance rows, and use main's session-affinity header order.

Co-authored-by: Cursor <cursoragent@cursor.com>
@valarLip
valarLip merged commit b88c520 into ROCm:main Oct 27, 2026
71 of 86 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants