Skip to content

feat(ep/v2): AsyncLL dispatch/combine on cco, with cross-node and gfx125x support - #768

Merged
yangyuhuiling merged 40 commits into
mainfrom
xiangch/ep-asyncll-cco-v2
Oct 27, 2026
Merged

yangyuhuiling merged 40 commits into
mainfrom
xiangch/ep-asyncll-cco-v2

Conversation

@yangyuhuiling

@yangyuhuiling yangyuhuiling commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Ports mori's AsyncLL dispatch/combine family from mori-shmem (EPv1) to cco
(EPv2), adds the cross-node leg over ccoGda, and enables it on gfx125x.

Select it with async_ll=True on EpDispatchCombineConfig (hip backend); the
transport is async_ll_transport="sdma" (default) or "lsa".

What AsyncLL is for

It splits each phase into a send half and a recv half, so a caller can put
compute between them:

out, _, _, _, total, routing = op.dispatch_send(x, w, None, idx, return_routing=True)
#  ... expert compute here is the point of the whole design ...
op.dispatch_recv()

Nine kernels back those four halves. op.dispatch() / op.combine() remain as
the synchronous wrappers.

Behaviour a caller will notice

  • Calls must alternate: dispatch, then combine. dispatch_send -> dispatch_recv -> combine_send -> combine_recv, with dispatch() /
    combine() taking both steps of their half. AsyncLL's recv counts and
    combine epoch pair each dispatch with the next combine, so an unpaired call
    used to corrupt a LATER round: dispatch twice gave wrong data, combine twice
    faulted every GPU, and cross-node the epoch drifted into a permanent hang.
    Out-of-order calls now raise RuntimeError before anything launches; a call
    refused for its arguments leaves the state where it was; reset() returns to
    the start.
  • Refused at construction rather than misbehaving later:
    • async_ll=True on the FlyDSL backend (it would silently run the ordinary
      kernels);
    • async_ll_transport="sdma" on a communicator with no SDMA queues (a
      library built BUILD_CCO_SDMA=OFF, MORI_ENABLE_SDMA unset, or no peer
      reporting SDMA -- every put would be a no-op and the receiver would spin);
    • dispatch(..., snapshot=True) with async_ll (a routing handle cannot
      outlive the next dispatch here);
    • a combine weight fold (weights= on combine), refused at the call, before
      anything is staged.
  • SDMA queues per pair default to one per peer, capped at 3 on MI308X
    (keyed on its device id, so MI300X/MI325X are not capped), and
    MORI_SDMA_NUM_CHANNELS now overrides the request instead of being ignored.
  • Early notification (SDMA) queues each ready word right behind its payload
    in the send phase, so the count lands while the caller computes. On by
    default for single-node gfx1250 SDMA; MORI_ASYNCLL_EARLY_NOTIFY=1 opts in
    elsewhere, MORI_ASYNCLL_FUSED_RECV=1 folds the receive waits into the copy
    passes. LSA and multi-node keep the original protocol.
    MORI_ASYNCLL_SKIP_RESET is still accepted but no longer does anything: the
    host total_recv reset it skipped was removed for every family by fix(EPv2): give routing handles their own buffers on request (snapshot=True) #733.

Correctness

gfx950 (wave64), EP16 cross-node 30/30 rounds
gfx950, EP8 single node (CI, every PR) 5 shapes x 3 iterations x 2 APIs, 2 graph scenarios x 32 replays, 6 back-to-back rounds, call order -- sdma and lsa
gfx1250 (wave32), EP4 single node same suite, both transports
8x MI355X and 8x MI308X (reviewer) the suite, plus back-to-back rounds with every row bit-exact and graph replay with new inputs, early notify off and on

test_asyncll.py checks arrival count, origin, received hidden / indices /
weights row by row (including a rank sending zero tokens), the identity round
trip, graph replay with fresh inputs, six rounds with no host sync between them,
and every out-of-order call. The two protocol scenarios were checked to have
teeth: removing the call-order check, or the received-weights fix, makes them
fail. It runs in the CCO unit test job on both transports.

Shapes include top-k 6/9/12, where warpSize % numExpertPerToken != 0 and the
remainder differs between wave widths (32 % 9 = 5 against 64 % 9 = 1).

Performance

Overlap, the reason it exists (reviewer, EP8, hidden 7168, top-k 8,
dispatch_send -> GEMM -> dispatch_recv and the same for combine, against the
same work run back to back):

  • MI355X, 512 tokens: sdma saves 58-148 us per exchange (15-22%), about 24%
    faster than native v2 plus the same GEMMs run sequentially.
  • MI355X, 128 tokens: 4-12 us -- the transfers are too short to be worth
    hiding; native v2 is the better choice there.
  • lsa saves nothing, as expected: it spends the grid on the copy.

Early notification on top of that, same setup, dispatch + combine us, off -> on:

off on
MI355X, 512 tok 526-527 450-462
MI355X, 128 tok 223-236 181-189

Cross-node EP16, hidden 7168, top-k 8, bf16, total us:

tok/rank asyncll v1 async_ll vs v1
128 419.9 430.6 -2.4%
256 684.0 694.0 -1.4%
1024 2618.8 2441.2 +7.3%
2048 4559.6 4545.2 +0.3%

Parity with v1, which is the bar: the two are the same protocol over different
transports, and RDMA bandwidth agrees digit for digit at 2048 (25.2/26.2 GB/s
against 25.7/25.8). v2_ll is 1.3-2.6x faster cross-node and that is
topological, not a tuning gap: its proxy model puts the traffic on one fat
same-rail link while AsyncLL's per-PE addressing spreads it over eight thinner
cross-rail QPs.

Single-node EP4 on gfx1250, same shape, empty seam, pair_us = dispatch +
combine, median of interleaved rounds:

tok/rank v2 native asyncll sdma asyncll lsa
64 65.7 90.9 676.4
256 74.3 122.0 677.7
512 95.2 155.8 717.6
2048 246.1 355.4 972.9

With an empty seam AsyncLL can only lose to native -- native writes straight
into the peer, AsyncLL stages for an RDMA leg a single node does not have.
These numbers are its cost; the overlap section above is its benefit.

LSA's peer copies run at Unroll=4 (from 1): -28.6% pair at 2048, the dispatch
half 202 -> 438 GB/s. Moving them onto the gfx1250 TDM engine was measured and
rejected (a wash -- AsyncLL's per-warp slices are in the latency regime where
TDM has no edge), as was padding the slot stride to 128 B (~3% on dispatch,
not worth changing the shared arena layout). Both are written up in the README
with three earlier rejected optimisations.

Deliberate non-features

  • AsyncLL combine does not fold output weights. Its combine payload carries
    no per-token metadata, and widening it would add topk*4 bytes to the one
    transfer this family exists to make cheap. v1 makes the same trade.
  • LSA is single-node only, and on gfx1250 it is 2.7-7.4x slower than sdma,
    so sdma stays the default.

Known limitations

  • DPX-partitioned gfx1250: reported failing downstream (sglang#44492 --
    sdma hangs, lsa combine mismatches); SPX is fine. Not reproduced here yet.
  • Cross-node has not been re-run since the review changes (call order,
    queue sizing, the underflow guards). The EP16 numbers above predate them.
  • One default transport for every arch. The reviewer's numbers favour lsa on
    MI308X and sdma on MI355X and gfx1250; a per-arch default is a follow-up.

Notable commits

  • feat(ep/v2): port the AsyncLL kernel family to cco -- the nine kernels
  • feat(ep/v2): implement the AsyncLL cross-node leg on ccoGda
  • feat(ep/v2): run AsyncLL on wave32 (gfx1250)
  • fix(ep/v2): clear AsyncLL LSA recv counts before announcing the combine epoch -- a count-reuse race; 1/8 against 8/8 pass on an idle gfx1250
  • perf(ep/v2): keep four peer stores in flight on AsyncLL's LSA transport
  • perf(ep/v2): submit AsyncLL SDMA notifications in the send phase
  • fix(ep/v2): keep AsyncLL recv topk weights on dispatch_recv and empty ranks
    (ChengYM5685) -- received weights were zero on the split API
  • From review: enforce AsyncLL's dispatch -> combine call order on the host,
    stop an unpaired AsyncLL combine from wrapping its byte count,
    bound AsyncLL's send slots to the slice they are allocated from,
    refuse AsyncLL sdma when the communicator got no SDMA queues,
    cap AsyncLL's SDMA queues on MI308X only,
    fix(anvil): evaluate a KFD call once in CHECK_HSAKMT_SUCCESS
  • ci(cco): run the v2 AsyncLL test on both transports

On the measurements

Numbers here come from interleaved A/B runs on nodes confirmed idle, with arms
that differ only in MORI_SOURCE_ROOT proven distinct by a planted #error.
That discipline is not ceremony: the same LSA baseline recorded 0 hangs in 55
runs on a busy box and 7 in 8 on an idle one, and several conclusions in this
work were drawn from single runs and were wrong.

Test plan:

  • MORI_ENABLE_SDMA=1 torchrun --nproc_per_node=8 tests/python/ops/dispatch_combine_v2/test_asyncll.py
    (TRANSPORT=lsa for the other transport)
  • tools/run_internode_test.sh --rank {0,1} ... --kernel-type asyncll for the
    cross-node leg
  • tools/bench_ep_asyncll_compare.sh for the four-way comparison

🤖 Generated with Claude Code

xiangch and others added 20 commits October 14, 2026 09:39
EPv2 had intranode, v2 (InterNodeV1) and v2_ll (InterNodeV1LL), but not
AsyncLL. What AsyncLL uniquely brings is the send/recv split: dispatch and
combine become two launches each, with a host-visible seam the caller can put
its own work into while the transfers drain. Every other family does send and
recv in one kernel and has no such seam. It also deduplicates per destination
RANK, where v2_ll does not deduplicate at all.

Thirteen kernels in src/ops/dispatch_combine_v2/ep_asyncll_kernel.hpp: the nine
ported from src/ops/dispatch_combine/low_latency_async.cpp, plus four
alternative transfer passes. The Cfg, the Request and the Args surface are the
internode family's unchanged -- same shape, same launch struct -- so only the
kernel enum, the entry names and the per-pass geometry are new.

Two transports, chosen with async_ll_transport:

  sdma  (default) ccoSdma, the copy engine. The transfer occupies no CUs, so it
        drains under whatever runs in the seam, and it is the shape the
        cross-node ccoGda leg will reuse.
  lsa   peer stores over the flat symmetric VA. Lower latency on one node,
        but it spends the grid on the copy and leaves nothing to overlap.

Two deliberate departures from v1, both measured on MI355X:

  * a peer's payload goes out as ONE copy instead of being sliced per QP. An
    SDMA queue is not a QP -- queues buy parallelism between warps, not inside
    a transfer -- and 600 KB in one op runs at ~46 GB/s against ~35 as two.
  * the token count rides one slot per source rank rather than one per QP.

EP8, hidden 7168, top-k 8, bf16, dispatch+combine as a CUDA graph:

              64 tok/rank            128 tok/rank
  v1 async_ll  125.2 us               150.1 us
  v2 sdma      103.0 us  (-18%)       133.8 us  (-11%)
  v2 lsa        96.4 us  (-23%)       135.0 us  ( -11%)

The transport is not what dominates: AsyncLL stages each payload three times so
an RDMA leg can reuse the protocol, which is also why it is not the fastest way
to dispatch on a single node -- the ordinary v2 intranode kernels do the same
shape in 59.5 / 89.6 us by writing straight into the peer. AsyncLL is off by
default for that reason.

Cross-node is not implemented. The branches are present and marked
TODO(internode), naming the ccoGda helpers that already serve v2/v2_ll, and the
host gate refuses a config spanning nodes rather than failing silently.

Supporting changes:

  * EpInterNodeArgs gains dispReceiverIdxMap (recv slot -> the send slot it
    arrived in) and offSendCount. The latter exists because ccoSdma moves window
    to window and has no immediate form, so a count word needs a real source
    address; v1 carried the same number in the WQE.
  * asyncll_regions() sizes the send buffers per RANK. v1/v1_ll aggregate a node
    behind a proxy rank and size them per NODE, which is a factor of
    gpu_per_node too small here, and the overrun would land in a neighbouring
    region of the same window.
  * the combine input gets its own region. It cannot share inter_combine_inp,
    where peers land their combine payloads: the two address ranges overlap and
    a peer's transfer would clobber the local expert output mid-read.
  * Toolchain::Flags() defines BUILD_CCO_SDMA. cco.hpp hides ccoSdma behind it,
    so a JIT TU that calls it does not merely lose an optimisation -- the name
    is undeclared. Flags() is hashed into the cache key, so this cannot serve a
    stale module.

tests/python/ops/dispatch_combine_v2/test_asyncll.py covers both transports over
five shapes and two iterations on eight ranks, checking arrival count, origin
decode and an identity-expert round trip. The shapes include top-k 6/9/12 --
where warpSize is not a multiple of top-k, the case v1's lane-tiling regression
was found in -- and per-rank token counts that vary per iteration, including
ranks handed none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The AsyncLL port's numbers were collected piecemeal on a shared box, and the
question the sweep was meant to settle is still open: at 64-128 tokens per rank
the transfer is a small part of the time, so the sdma and lsa transports land
within a few microseconds of each other, but at 1024+ the copy engine saturates
(~60 GB/s per link, measured) while peer stores do not. Nothing has measured
that crossover, and the comment in ep_asyncll_kernel.hpp only claims what was
observed at the two low points.

One script, one token sweep, four implementations -- the two AsyncLL transports,
EPv2's ordinary intranode kernels as a floor, and EPv1 AsyncLL as the thing the
port had to beat. Output is one results.json plus a table.

Three things it does that matter more than the sweep itself:

  * refuses to run when other processes hold the GPUs, unless --force. That is
    exactly how the existing numbers became unciteable, and it is invisible
    after the fact.
  * records provenance next to the results -- host, kernel, ROCm, git commit and
    whether the tree was dirty -- because two numbers from two machines are not
    comparable without it.
  * settles known-issues Issue 1 rather than reporting it as unknown. /boot is
    not mounted in the container, so it builds and runs vmm_peer_probe instead;
    an affected node reports about a ninth of the real cco bandwidth and says
    nothing about it, which would void the whole run.

--dry-run does the preflight, the provenance and the plan without touching a
GPU. On a borrowed node, finding out an argument was wrong after forty minutes
of benchmarking is the expensive failure.

Latency is tabulated as the cross-implementation number and bandwidth as the
within-implementation one, because the two harnesses' bandwidth numerators are
close but not identical: bench_ep.py uses recv_tokens*hidden*elem/time, while
the v1 harness reports its own XGMI column. Recomputing the latter by hand lands
within ~3% at the same point, which is why they are tabulated side by side at
all, and the definition travels with every row in the JSON.

The v1 scrape reads the table's Average row rather than the JSON that harness
records internally: that record pairs best-bandwidth with average-latency, and
the two cannot honestly share a column.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… data

The sdma-vs-lsa comment was written from two points measured on a shared box,
and the extrapolation in it was wrong in SIGN. Reasoning from the small end
said lsa should pull ahead once the copy engine saturates near 60 GB/s per
link. A full sweep on an idle, reserved MI355X node says lsa loses, by a
widening margin, from 128 tokens per rank up:

  tok/rank      64     128     256     512    1024    2048   (us, pair)
  sdma       102.0   136.8   194.3   292.1   523.1   982.0
  lsa         97.7   138.9   201.5   320.3   581.4  1094.2
  delta      -4.2%   +1.6%   +3.6%   +9.7%  +11.1%  +11.4%

Two sweeps on the same lease agreed within 0.3 percentage points at every
point, so this is not instance spread.

What the prediction left out: the transfer is a kernel of its own, so nothing
overlaps it either way and the engine's separate hardware is pure gain; and a
grid doing the copy contends with its own HBM traffic where the engines do not.
The ordinary v2 intranode dispatch reaches 856.9 us on the same point, so the
grid is not short of bandwidth -- it is short of bandwidth while also running
the staging copies this protocol adds.

sdma therefore stays the default on evidence rather than only because it is the
shape the cross-node leg will reuse. lsa is kept for the small-batch case, where
it is ~4% ahead, and because it leaves the copy engines free.

The v1 comparison holds across the whole sweep, though the margin narrows from
18% at 64 tokens to ~4% at 2048:

  v1_asyncll 124.7   146.2   201.4   314.1   540.8  1019.2

Peak per-rank payload bandwidth at 2048 tokens: 299 GB/s dispatch / 338 GB/s
combine for sdma, against 337 / 395 for the non-staging v2 intranode path.

Also removes the "lsa is EXPECTED to pull ahead" note from the benchmark
script's header, which was the hypothesis this run falsified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dispatch and combine both staged into inter_staging, at different strides over
the same bytes. That is correct today but couples the phases: dispatch's
transfer reads the buffer asynchronously and combine's SendCopy overwrites it,
so the only thing between them is dispatch's quiet.

The quiet was there for ORDERING -- keeping the count word behind the payload --
and was silently doing a second job, source-buffer safety, because of the
sharing. Two jobs in one call is why it cannot move: it is pinned to "before the
next phase writes", which is the point in the pipeline AsyncLL most wants to be
free to defer past. Splitting the buffers leaves it with only the ordering job,
which turns removing it into a question about SDMA queue ordering alone.

No quiet is removed here, so the critical path is unchanged and the expectation
was that this would measure as nothing. It measures as slightly better. Four
interleaved arms (split, shared, split, shared) in one lease on one idle
reserved MI355X node, EP8 hidden 7168 bf16, pair latency:

   tok/rank     64    128    256    512   1024   2048
   sdma       -0.5%  -0.3%  -0.4%  -0.6%  -0.1%  -0.6%
   lsa        -0.8%  -1.9%  -1.4%  -1.0%  -0.4%  -0.3%

Negative is split being faster, and all twelve points have that sign against a
worst-case rep-to-rep spread of 0.3%. Small, but not noise. With no quiet
removed there is no protocol mechanism for it, so the likely cause is just
locality: dispatch's staging lines are no longer evicted by combine writing the
same addresses at a different stride.

What it is NOT is the writeup's result. The same split on the EPv1 RDMA path was
worth 19us of a 60us improvement there, but that gain comes from dropping an
RDMA CQE drain, which is far more expensive than ccoSdma's quietQueue. That
number should not be expected to transfer, and it did not.

Costs one hidden-sized symmetric buffer per rank: tens of MB at EP8 / 128
tokens / 7168, against an arena already in the hundreds.

Verified on 8x MI355X, both transports, five shapes x two iterations including
top-k 6/9/12 and ranks handed no tokens; plus test_op.py on both backends and
the JIT binding and region unit tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ROCm 7.0-7.2.x ships a clr hipMemSetAccess defect that breaks cco's window
setup CROSS-NODE while leaving a single node working (known-issues skill,
Issue 2). Everything built on cco internode then fails at construction with
`ccoMemAlloc: hipMemSetAccess failed ... (invalid argument)`, and the traceback
points at whichever op happened to call it rather than at the runtime. The same
failure turned up twice while porting AsyncLL -- once in test_ep_backend_parity
on an untouched tree -- and both times cost a round of debugging before the
cause was recognised.

Learning this from a full EP internode test costs a build plus minutes per
attempt. This costs seconds and names the cause in its failure message.

Two details are what make it representative rather than decorative:

  * it registers SEVERAL windows of DIFFERENT sizes. The defect is triggered by
    sub-buffer geometry -- hipMemSetAccess walks the parent reservation from
    sub-buffer 0 regardless of the pointer it was given -- so a single-window
    probe comes back clean on a broken runtime.
  * it asks for GDA_CONNECTION_CROSSNODE, not the RAIL set the current
    internode kernels use. A per-PE protocol such as AsyncLL needs QPs to every
    cross-node peer, and that is the connection set worth proving.

Verified on 2x8 MI355X (AAC mi355-gpu-50/51), ROCm 7.14, podman rootless:

  # cco devcomm probe: PASS (world=16, nodes~2, 0 ranks failed)
    ok (lsa_size=8 lsa_rank=0 world=16)

Two environment notes worth recording, both of which cost time today. On these
hosts the docker group holds only service accounts, so containers come from
podman rootless instead; and an AMD GPU needs `--group-add keep-groups` to carry
the host's video/render membership in, without which KFD filters every GPU out
of the topology and torch reports zero devices while /dev/kfd and /dev/dri are
plainly visible. pyxis/enroot has no AMD hook and hits exactly that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fills the four TODO(internode) branches. Same protocol as the single-node path
-- same slots, same count word, same epoch -- with the transport chosen per peer
on `destPe / gpuPerNode != myNode`, which is how v1's AsyncLL has always worked
(shmem resolves the same test through its per-peer transport table).

Two things differ by transport, both on purpose:

  * a cross-node payload is STRIPED over numQpPerPe QPs, where a same-node one
    goes whole. Splitting an SDMA copy only makes each piece smaller on the one
    link a peer has; splitting an RDMA transfer spreads it over QPs that really
    do progress in parallel, and the op README records that num_qp_per_pe=1
    starves that path by about 1.5x.
  * the remote count and the barrier epoch are atomic ADDs, because an add at an
    arbitrary window offset is the only remote update ccoGda offers. Both are
    exactly equivalent to the stores they replace: combine clears the count slot
    every round so adding tokenNum+1 to zero lands v1's value, and each rank
    writes a given peer's barrier slot once per round, so N increments produce
    the epoch N that the existing wait already compares against.

Host side: a multi-node AsyncLL takes GDA_CONNECTION_CROSSNODE rather than the
RAIL set v2/v2_ll ask for. Those aggregate through a same-rail proxy and need
only same-rail QPs; AsyncLL addresses peers individually, so rank i sends to
rank j directly and needs a QP to every cross-node peer. The "lsa" transport is
refused across nodes -- peer stores reach only the flat LSA window.

That connection requirement has a consequence worth stating where someone will
hit it: AsyncLL cannot run across nodes on a RAIL-ONLY fabric, and the failure
is silent. QP setup is a TCP exchange and succeeds; no data moves. Measured on
two fabrics today: crusoe crsuse2 is rail-only (same-rail 41.6 GiB/s, cross-rail
dead across 12 SL/TC combinations), while AAC mi355-gpu-50/51 routes cross-rail
at full speed (28.1 GiB/s against 27.5 same-rail). Confirm with an ib_write_bw
between two DIFFERENT rails before trusting a multi-node run.

NOT VERIFIED ON TWO NODES. Single node is unaffected and still passes: both
transports over five shapes x two iterations including top-k 6/9/12 and ranks
handed no tokens, test_op.py on both backends, and all thirteen kernels compile.
The two-node run is blocked on hardware rather than on code -- the only
cross-rail-capable pair available is mi355-gpu-[49-51], which is running mori's
own self-hosted CI under a long-lived Slurm allocation, and a control run of the
EXISTING v2_ll there took a memory fault that is most likely collision with it.
Two prerequisites were cleared along the way and are recorded so the next
attempt starts further along: cco multi-node devcomm works on ROCm 7.14
(tools/cco_devcomm_probe.py, world=16, PASS) and ci_cco.yml's matrix names the
RDMA device list, SL and TC this host pair needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…king it

The combine branch of _wrap_asyncll passed a REGION name to _region(), which
takes a VIEW name and resolves it to one. Any caller that asked for output
weights got KeyError: 'dispatch_out_weights' rather than weights.

Resolving the name is not the fix, though -- it only moves the failure. The
fold the internode family performs (EpCombineAll's second WarpAccum, in
ep_internode_kernel.hpp) reads the weights back out of the COMBINE staging,
which carries topk floats per token alongside the hidden row. AsyncLL's
combine payload is hidden-only by design, so there is nothing to accumulate
and the kernels never write combine_out_weights at all. Pointing the base at
the right region would hand callers a zero tensor that looks like an answer.

Widening the payload is the wrong trade: topk*4 bytes per token on the one
transfer this kernel family exists to make cheap. v1's AsyncLL makes the same
call -- low_latency_async.cpp writes weights on dispatch and never on combine
-- so refusing is also parity, not a new limitation.

So: raise NotImplementedError, naming the alternative.

Why no test caught it: on this API `weights` is a REQUEST, not an input, and
every combine_send in test_asyncll.py passed None, which skips the branch
entirely. The cross-node harness asks for weights unconditionally and found it
on the first run. The test now asserts the request is refused.

That check runs last in the iteration and only on the first one. The raise
comes from inside _combine_impl AFTER it has staged `input` into the out_tok
region, so running it before the round trip would have it write through a
buffer the measured path reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ghts

The harness asked for output weights unconditionally, in both the round-trip
check and _verify_once, and compared what came back against
unique_pes * wts. AsyncLL cannot serve that request -- its combine payload
carries no per-token metadata -- so with the refusal in place every asyncll
run died in argparse's successor: the first combine.

Three places now take `out_w is None` as a valid answer rather than a bug:

  * both combine() calls pass weights only when the family folds them;
  * weights_ok is vacuously true when none were requested, so the hidden
    check -- the half that can actually be wrong -- still decides the verdict;
  * the failure dump skips the ratio, the region reads and the weights diff,
    all of which dereference the tensor that was never produced.

_verify_once keeps its "always fold here" rule for every other family. That
rule exists because an unchecked weight path shipped a silent bug twice; the
exception is narrow and the hidden half still runs.

With this, --kernel-type asyncll passes 30/30 rounds at EP16 across two nodes
(hidden 7168, topk 8, bf16, mi355-gpu-50 + gpu-51).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MakeEpInterNodeCfg refused any arch whose wavefront is not 64 lanes, so
AsyncLL died at plan construction on gfx1250:

  mori jit [ep_asyncll_dispatch_slotassign]: mori ep internode v2:
  waveSize 32 is unsupported; the internode kernels are wave64 only

The gate is right about the internode family and wrong about AsyncLL. Its own
comment names the reason: DispatchInterNodeSend builds an intra-warp prefix
count as __popcll(mask << (warpSize - laneId)) over a uint64_t, and on wave32
the shift width no longer matches the container, so the discarded lanes stay
inside the value and the send slot comes out wrong.

AsyncLL does not run that kernel, and nothing in the nine it does run is
written against a lane-mask width: there is no __ballot and no __popc anywhere
in ep_asyncll_kernel.hpp, laneId and warpSize come from DEF_COMMON_VARS (the
HIP builtins), and the geometry already sizes blocks from cfg.waveSize. It
shares the factory, not the kernels.

So the gate is narrowed rather than removed -- v2 and v2_ll still refuse
wave32 until that idiom is rewritten (mask & ((1ull << laneId) - 1) would do
it, independent of container width, but it needs its own validation).

The one real wave dependency in AsyncLL is DispatchRecvCopy's per-PE offset
prefix sum, one lane per PE across a single wavefront. Its static_assert said
worldSize <= 64, which is the absolute bound and not this arch's; waveSize is
host-side state and not part of the rendered cfg, so the kernel cannot state
the exact one. The precise check moves to EpAsyncLLRenderSource, the funnel
every compile goes through including a hand-initialised Cfg, which does know
it. Left unchecked the failure is silent: __shfl past the end returns the
issuing lane's own value rather than faulting, so PEs would quietly share an
offset.

Verified on an MI455X (4x gfx1250, wave32), EP4, test_asyncll.py:

  5 shapes x 2 iterations x 4 ranks, 0 failed checks

including top-k 6/9/12, where warpSize % numExpertPerToken != 0 and the
remainder differs from wave64's (32 % 9 = 5 against 64 % 9 = 1) -- the
trailing-group drop in SlotAssign was written for exactly that and this is the
first time it has been exercised at a second wave width.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nging

The peer-store transfer bodies do not complete on wave32. Measured at EP4 on
an MI455X (4x gfx1250), hidden 7168, topk 8, bf16, CUDA graph: the sdma
transport sweeps 64..2048 in about two minutes and verifies every point, while
lsa never finishes the FIRST one -- all four GPUs pinned at 100% for half an
hour, until the harness timeout killed it.

Undiagnosed, and not diagnosed here. What is worth recording is the shape of
it: unlike sdma, these bodies move the bytes from the grid itself over the flat
peer VA, so they are the half of AsyncLL whose correctness rests on the memory
model rather than on a copy engine. The MI355X numbers in the LSA section
comment were all wave64 and still stand for that arch.

test_asyncll.py drives the sdma transport only, which is why 6e22cb66 could
turn wave32 on and pass without this surfacing. It surfaced in the four-way
comparison benchmark, where lsa is its own arm.

A gate is the whole fix available right now: the failure produces no message of
its own, and a caller who picks lsa on this arch otherwise gets a wedged job
with nothing to read. For the comparison benchmark it turns a 30-minute hang
into an immediate refusal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The numbers existed only in a session's scrollback and in /tmp on two
reservations that will be reclaimed. Both sweeps are here with the conditions
that produced them, because several of the entries are misleading read alone.

Cross-node, EP16 on a pair of MI355X: AsyncLL lands within +-7% of v1 from 128
tokens up and +0.3% at 2048, which is the result that matters -- the two are
the same protocol over different transports, so tracking that closely is what
says the cco port preserved it. The -43% at 64 tokens is recorded as NOT a win:
v1 autotunes its launch geometry and picks badly in that band, and is slower at
64 tokens than at 128. Reading that row as a speedup is the obvious mistake and
the table says so.

Against v2_ll AsyncLL is 1.3-2.6x slower, with the reason: v2_ll's proxy model
puts cross-node traffic on one fat same-rail link (64/71 GB/s at 2048) while
AsyncLL's per-PE addressing spreads it over eight thinner cross-rail QPs
(25/26). Topology, not tuning.

Single-node, EP4 on an MI455X (gfx1250, wave32): AsyncLL is 34-67% behind the
native intranode path, the same shape MI355X shows at EP8, for the same reason
-- native writes straight into the peer and AsyncLL stages, which on one node
buys nothing because there is no RDMA leg to feed. The LSA arm is absent and
why is stated rather than left as a gap in the table.

Carried at the top of the section, because it governs how all of it reads: the
benchmark calls send then recv with nothing in between, which is exactly the
seam AsyncLL exists to fill. It measures the protocol's cost and none of its
benefit. Whether AsyncLL pays for itself needs an end-to-end MoE run with real
expert compute in the gap, and that is not measured here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LSA transport moved the dispatch payload twice. SendCopy gathered scattered
tokens into local staging at SendBufSlotOffset(destPe, slot); SendTransferLsa
then bulk-copied that contiguous range into the peer. The staging hop is there
for ccoSdma, which needs a local contiguous source to hand the engine -- a peer
store has no such requirement, because the grid is already doing the move.

So for this transport the two passes become one: the gather addresses the peer
directly. DispatchSendCopyLsa also absorbs the count publication, which was all
SendTransferLsa did besides copying, which drops the LSA send half from three
kernels to two. Self still lands in staging, because RecvCopy reads this rank's
own slice from there rather than from dispatchInp.

Measured EP8 on an MI355X, hidden 7168, top-k 8, bf16, CUDA graph, six
interleaved pairs in one container, median:

  tok     unfused    fused    delta
   64       118.4    110.2    -7.0%
  128       147.2    138.2    -6.1%
  256       209.2    194.1    -7.2%
  512       316.9    306.5    -3.3%
 1024       574.6    565.5    -1.6%
 2048      1067.2   1047.9    -1.8%

Every pair agreed in sign at every token count, with under a point of spread
inside each.

Smaller than removing a whole pass over the payload suggests, and shrinking
with batch, because of what the pass that remains turned into: a local gather
plus one contiguous remote copy became one SCATTERED remote write, a token at a
time into arbitrary peer slots. The pass that went away was the cheap one. At
small batch the fixed costs it also removed dominate and it is worth 6-7%; by
2048 the two effects have nearly cancelled.

MORI_ASYNCLL_LSA_FUSED=0 selects the old sequence. Kept because it is how the
table above was produced -- the pre-fusion numbers in the LSA section comment
are from a different box, and on these parts that is enough to move a
single-digit effect either way.

EP8 correctness passes on both paths, 5 shapes x 2 iterations x 8 ranks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three follow from the same picture -- AsyncLL stages the payload three times
and runs nine kernels, so the obvious targets are the extra passes and the extra
launches -- and all three are plausible enough to be proposed again. Two were
measured and rejected; one never got correct. Written down so the next reader
does not spend the afternoon rediscovering them.

Pipelining the send half over PE groups: +20% (2048 tokens) to +61% (64), six
interleaved pairs, every pair agreeing in sign. The SDMA engine's copy is not on
the stream, so starting it on group 0 while the grid stages group 1 should be
free overlap; what actually happened is that each SendCopy launch still walks
every entry and copies only its share, so half the grid idles through the scan.

Fusing SlotAssign and SendCopy behind a grid rendezvous: +12% to +104%, same
method. A spin barrier waits for the slowest block and SlotAssign's work is
uneven by nature, where the kernel boundary it replaced was synchronised by the
hardware scheduler for less.

Those two share a mistake worth naming, because it is the reason the second was
attempted after the first failed: both assumed launch overhead was the fixed
cost that matters at small batch. It is not. The first experiment's 61% penalty
at 64 tokens was load imbalance, and reading it as launch cost is what made
removing a launch look promising.

Fusing the cross-node count onto the payload's WQE never passed EP16: 16 of 16
ranks disagreed, max|diff| around 24 with got_nonzero at or near full -- every
output position written, some holding the wrong value, which is a wrong offset
rather than data that has not arrived. Abandoned rather than chased, because the
cross-node leg is bandwidth-bound on per-PE cross-rail QPs (~25 GB/s against
v2_ll's 64-71) and a CQE drain is not what is costing it.

The one that did work is already in: fc4c9285, the LSA gather landing in the
peer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
76ecd0a3 said the peer-store transfer "does not complete on this arch". That is
wrong in a way that matters to anyone reading it: on gfx125x the LSA transport
works fine in eager mode and verifies. What hangs is the combination with CUDA
graph capture, and only sometimes.

Measured at EP4 on an MI455X, hidden 7168, 64 tokens, one configuration, clean
between runs:

  lsa  + eager          passes every run, output verifies
  lsa  + graph capture  hung 6 runs out of 8
  sdma + graph capture  passes every run

So neither the transport nor the arch is broken on its own. The original claim
came from a single run of a benchmark that always captures, read against a
single-node test that never does -- which made an execution-mode problem look
like an architecture one.

The gate stays on the arch rather than moving to capture, because the op is
built before the caller has said whether it will capture and a hang is a worse
answer than a refusal. That is a real cost -- it refuses a working eager path --
so MORI_ASYNCLL_LSA_FORCE=1 unlocks it for a caller who knows their mode, and
for working on the hang.

Still undiagnosed, but the shape is worth recording: these are the one half of
AsyncLL that moves bytes from the GRID over the flat peer VA instead of handing
them to an engine, so their ordering rests on the memory model rather than on a
queue. Eager puts a kernel-launch boundary between the peer stores and the
receiver's spin; a replayed graph does not. A missing release there would look
exactly like this -- intermittent, capture-only, sdma unaffected.

The comments now also say to measure the RATE before believing anything about
it. Four conclusions were drawn here from single runs -- that the kernels were
broken on wave32, that the LSA fusion caused it, that the iteration count was a
threshold, and that a lane guard was the fix -- and all four were wrong. At a
75% hit rate a passing run carries no information.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ne epoch

CombineRecvTransferLsa published its epoch, waited for every peer, and only
then cleared the dispatch receive counts. A peer that cleared the handshake
could publish its next round's dispatch count before our late zero landed on
it, and the next dispatch then waited on a count that would never be written
again. SDMA clears in CombineSendTransfer, ahead of its epoch announcement,
which is why only LSA ever hung. LSA now clears each peer's slot before the
release store that frees it, paired with an acquire on the waiting side.

This is what was behind the intermittent gfx125x hang under CUDA graph capture.
Eight interleaved pairs at EP4, hidden 7168, 64 tokens, on an idle MI455X:

  baseline   1 pass / 7 hang
  fixed      8 pass / 0 hang

no contaminated samples, arms alternating rather than run in blocks. Graph E2E
latency is unchanged (1069us against 1071us), and three further graph runs on
the final tree all passed.

So the gfx125x gate from 76ecd0a3 goes. It was refusing a transport that now
works, including in eager mode where it always worked.

Two things about the measurement, because an earlier 55-run matrix on the same
baseline recorded 0 hangs and concluded nothing:

The arms shared one host library and differed only in MORI_SOURCE_ROOT. That
this actually compiled different kernels was checked, not assumed -- an #error
planted in the baseline tree fired when the root pointed there and stayed
silent when it pointed at the fixed tree. Without that, "both arms passed"
cannot be distinguished from "both arms ran the same code".

And the node has to be idle. The 55-run matrix ran on a box with other jobs
coming and going; an interfering job widens the gap this race needs and hides
it completely. Freeing the machine first is what made a 7-in-8 failure visible
in sixteen runs.

A note on the reasoning, since it cost a round of disbelief: I argued this race
could not be the cause because it does not need graphs, yet eager passed 8/8.
That was wrong. The window is the gap between a peer clearing the handshake and
our clear landing; eager's host round trip between kernels is wide enough that
ranks rarely sit in it. "Eager passes" was evidence that the window was wide,
not that the race was absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gfx1250 table had two columns and a paragraph explaining why LSA had none.
With the count-reset fix in, it has four, measured as a median of three
interleaved rounds at EP4, hidden 7168, top-k 8, bf16, under graph capture:

  tok    native   sdma   lsa fused   lsa unfused
    64     65.7   90.9       676.5        1071.3
   256     74.3  122.0       714.5        1108.8
  2048    246.1  355.4      1370.2        1706.9

Two findings are worth more than the numbers.

LSA is usable on this arch and not useful: 2.9-7.4x slower than sdma, 5.6-10.3x
slower than native. On MI355X the two transports trade places -- lsa 4% ahead
at 64 tokens, 11% behind at 2048 -- so whatever makes a peer store competitive
on wave64 does not hold here. That is a real difference between the parts, not
a tuning gap, and sdma stays the default by a wide margin.

The LSA fusion (fc4c9285) is worth five times more here than it was on MI355X:
-36.9% at 64 tokens against -7.0% there, same sign throughout. Consistent with
the staging pass being a larger share when the pass that remains is this
expensive. It was committed on a 1.8%-at-2048 result; this is the stronger case
for it.

The 512 row comes from a separate six-repeat run. The three-round sweep put it
at 331us, which is non-monotonic between 256's 122 and 1024's 222 and did not
match the 156.8us measured on another node. Six repeats give 155.8us with 1.2us
of spread -- and the neighbouring 384, included as a control, showed 810us of
spread over the same six. This box throws occasional large outliers that a
median of three does not absorb. 256 and 1024 reproduced to within 0.5% across
both runs, which is what says the rest of the column stands rather than needing
the same treatment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The AsyncLL notes and perf tables referred to the gfx1250 box by its product
name. Everything measured there is a property of the ISA and the wave width,
which is what the surrounding text already reasons about, so the arch name is
both the accurate label and the one that does not carry an unreleased part.
MI355X is left as-is: it is a shipping part and the gfx950 rows name it the
same way the file's existing MI308X rows do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both are line breaks the formatter wants and nothing else: the spec macro now
fits on one line, and the barrier wait wraps before `flag`. No behaviour
change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… guard

The guard's DEVICE_ONLY_HEADERS list is both the set of headers no host source
may include AND, via the path filter under it, the set of files exempt from
that rule because they are device headers themselves. AsyncLL's kernel header
was added without being put on the list, so its own include of
ep_internode_kernel.hpp read as a host source reaching into a device one:

  LEAK  a host source #includes a device kernel header:
        src/ops/dispatch_combine_v2/ep_asyncll_kernel.hpp:69:#include ".../ep_internode_kernel.hpp"

Nothing was actually leaking -- ep_asyncll_spec.cpp is the host side and never
includes the kernel header -- so this is the list being incomplete rather than
the property being violated. Caught by "CCO unit test (MI355X_AINIC)" on #768;
the step is registered by no other job, which is why it only surfaced there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three peer-facing copies on the "lsa" transport -- the fused dispatch
gather and the two bulk SendTransfers -- used core::WarpCopy's default
Unroll=1. That leaves each lane with exactly ONE 16 B store outstanding, so a
warp walks a 14 KB token as 28 dependent round trips to the peer. Unroll=4
keeps four in flight.

Measured on an idle gfx1250 node at EP4, hidden 7168, topk 8, bf16, graph;
five interleaved rounds per arm, medians, arms differing only in
MORI_SOURCE_ROOT. us for the dispatch+combine pair:

  tok/rank        128      512     2048
  Unroll=1      646.5    813.6   1364.1
  Unroll=4      644.3    715.4    974.5
  delta         -0.3%   -12.1%   -28.6%

The dispatch half alone is -53.9% at 2048, 202 -> 438 GB/s in bench_ep.py's
accounting. Unroll=8 ran in the same matrix and is not better -- it ties at
512 and 2048 and loses 1.4% at 128 -- so 4 is the setting rather than a
direction to keep pushing.

EpCombineAsyncLLSendCopy_body, whose copy is LOCAL, was already at Unroll=4.
Only the peer-facing sites were left at 1, which is the wrong way round: a peer
store is the one with latency worth hiding.

Note which end this moves. At 64 tokens it is worth nothing (+0.3%), so the
remaining 7.4x gap to sdma at the small end is a different problem from the
2.7x at the top; the README's EP4 table is re-measured accordingly, with the
unmodified arm reproducing the column it replaces to within ~1% at five of six
points.

Two things measured in the same matrix and rejected, both written up under
"Tried and rejected": moving the transfer onto the gfx1250 TDM engine (a wash,
and the reason is that AsyncLL's per-warp slices sit in the latency regime
where benchmark/cco/ualoe_metric already shows TDM has no edge), and padding
the slot stride to a 128 B row (worth ~3.4% on dispatch, but it changes the
arena layout shared with the v2/v2_ll families).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… ranks

AsyncLL's recv copy pass unpacks the topk weights only when weightsBuf is
non-null, and _wrap_asyncll set it from the weights handed to the current
launch. Two launches got a null pointer, so the recv weights view was never
written and read back as zeros:

  * dispatch_recv(): it launches the recv half with only input / dest_map /
    num_tokens, and in doing so also reset the stored indices to None.
    Reuse the weights and indices the send half was handed instead.
  * a rank with no tokens of its own: its weights tensor is empty, so
    data_ptr() is 0, yet it still receives its peers' tokens. Keep
    weightsBuf non-null there (the send half reads nothing with 0 tokens).

Downstream this zeroes every routed expert's contribution: SGLang
DeepSeek-V4-Flash EP4 on gfx1250 produced garbage with AsyncLL (sdma and
lsa, with or without TBO), where idle DP ranks hit the second case and the
split dispatch_send / dispatch_recv path hits the first.

test_asyncll did not catch it: it checks arrival, origin and a weights=None
round trip, never the received weights. With this change, native EPv2 and
AsyncLL (sdma, lsa) give bit-identical recv hidden / indices / weights and
combine output on EP4 with one rank sending 0 tokens, and test_asyncll
still passes.

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and tested on 8×MI308X (ROCm 7.14) and 8×MI355X (CI image rocm/mori:ci-cco), PR head a0b9dfcb against its base 5f396156.

Summary: AsyncLL works and is worth having on MI355X, but on MI308X the default config cannot build the op, and a couple of misordered calls turn into GPU faults or silent wrong data. Details are inline; the first four are what I would fix before merge:

  1. MI308X: 8 SDMA queues per pair do not fit, so op construction exits (hip_backend.py).
  2. combine without a matching dispatch underflows the count, and the resulting SDMA copy faults all GPUs (reproduced).
  3. dispatch twice without combine gives wrong data (reproduced on both archs); on the cross-node path the epoch drifts into a permanent hang (by reading). A host-side call-order check would cover 2 and 3.
  4. async_ll=True is silently ignored on the default FlyDSL backend.

What passed:

MI308X MI355X
test_asyncll.py sdma / lsa sdma fails at construction; passes with ≤3 queues; lsa passes pass / pass
6 back-to-back rounds, no host sync, every received row checked bit-exactly pass pass
8 graph replays, new inputs each replay pass pass
dispatch twice, then combine wrong data wrong data
combine twice GPU memory fault not run (shared box)
default path (async_ll=False), base vs PR no regression (within ±2%) —

Overlap (the point of the feature). I timed dispatch_send → GEMM → dispatch_recv (and the same for combine) against the same work run back-to-back. The GEMM is [rows×7168]×[7168×4096] bf16; EP8, hidden 7168, top-k 8, 32 experts per rank.

  • MI355X, 512 tokens: sdma saves 58–148 µs (15–22%). Against native v2 plus the same GEMMs run sequentially, it comes out about 24% faster (376 vs ~495 µs at 1024 GEMM rows).
  • MI355X, 128 tokens: it saves only 4–12 µs, because the transfers are too short to be worth hiding.
  • MI308X: it saves 150–640 µs, but mostly by recovering its own slow transfers.
  • lsa: saves nothing, as expected.

So the seam pays off, but mainly at larger batches and on parts with fast SDMA.

Optimization ideas, with what I measured:

  • Fold the count and epoch into the transfer queue (inline on RecvTransfer): this removes the post-seam drain-and-announce step, about 20 µs per exchange by my estimate.
  • MI308X is SDMA-bandwidth-bound: dispatch RecvTransfer takes ~400 µs at 512 tokens, against ~50 µs for all the CU staging passes combined. Seven peers share engines 2/4/6 there, so using engine 0 as well, or a hybrid that splits peers between sdma and lsa, is where the time is.
  • Choose AsyncLL and its transport per arch and per batch size: see the table on async_ll_transport; at 128 tokens native v2 is the better choice on both parts.
  • Not worth it, measured: Unroll=4 on the local SendCopy/RecvCopy gives no change on MI308X (A/B through MORI_SOURCE_ROOT, confirmed with a planted #error). Your README already covers TDM and fusing SlotAssign with SendCopy.

Not covered: cross-node AsyncLL (no two-node run on my side) and gfx1250.

# fewer queues than peers folds two warps onto one queue -- correct,
# but they then serialise on its in-order commit chain. Eight is the
# hardware cap; above it cco clamps with a warning.
requirements.sdma_queue_count = min(8, max(1, cfg.gpu_per_node))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MI308X: the default sdma transport cannot build the op. min(8, gpu_per_node) asks for 8 SDMA queues per GPU pair, but MI308X only fits 3: its xGMI engines fold to the even ids {2,4,6} (8 queue slots each), and ROCr's own blit queues take part of that. On 8×MI308X (ROCm 7.14), test_asyncll.py fails on all 8 ranks with ERROR code: 6 (HSAKMT_STATUS_NO_MEMORY) at anvil.cpp:210 (hsaKmtCreateQueueExt), and the process exits.

With this value overridden to 1, 2 or 3, test_asyncll.py passes (5 shapes × 2 iterations), as do my back-to-back and graph-replay checks. So the protocol is fine; only the count is too high.

Suggest deriving the count from what the device can actually hold (or clamping it per arch), and degrading instead of exiting when queue creation fails. A plain MORI_SDMA_NUM_CHANNELS-style override would also help, since this value currently beats the env default.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 29966fe. Two parts:

  • _asyncll_sdma_queues() caps the request per arch for archs known not to fit eight. Only gfx942 -> 3 is in the table, taken from your MI308X numbers. That cap also applies to MI300X, which may fit more. I have no MI300X/MI308X to check, so I chose the value that is known to build.
  • If MORI_SDMA_NUM_CHANNELS is set, AsyncLL now returns 0 and leaves the count to the env. Before, its request always overrode the env value.

I did not make queue creation degrade. CHECK_HSAKMT_SUCCESS calls exit(), so by the time the request is refused the process is already gone, and changing that touches every anvil caller. While in there I found the macro evaluated call twice: on failure the KFD call ran a second time, and the code it printed was that retry's. So the ERROR code: 6 you saw came from the second hsaKmtCreateQueueExt, not the first. Fixed in a3767ba.

I can't verify this on MI308X. If you can rerun test_asyncll.py there with the default config, that would confirm it.

uint64_t tokenNum = 0;
if (laneId == 0) {
uint64_t* slot = reinterpret_cast<uint64_t*>(recvCounts + asyncll::RecvCountSlot(destPe));
tokenNum = core::AtomicLoadRelaxedSystem(slot) - 1;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A combine with no matching dispatch underflows tokenNum and starts a ~2^64-byte SDMA copy. If the slot is already 0 — combine called twice with the same routing, combine() before any dispatch, or combine_recv() after a full combine() — AtomicLoadRelaxedSystem(slot) - 1 is UINT64_MAX, and the tokenNum == 0 guard below misses it. sdma.put(..., tokenNum * hiddenBytes) is then a wrapped, enormous size.

Reproduced on 8×MI308X: dispatch, combine, then a second combine with the same routing makes all 8 GPUs fault (Memory access fault by GPU node-N, PERMISSION_FAULTS in dmesg) and kills the job.

A user mistake should be an exception, not a GPU fault. A guard here (if (raw == 0) continue; before subtracting) plus the host-side call-order check suggested on dispatch_send would cover it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, kernel side, in 84f6f76 and e24c627. It now tests the raw biased value and skips when it is 0. 84f6f76 only covered the legacy path. The early-notify path that d59607f added (on by default for single-node gfx1250 SDMA) had the same count - 1 in CombineSendCopy; e24c627 guards it too.

This only keeps the mistake from becoming a GPU fault; the round is still wrong. What actually stops it is the host check in 0c0b3ba (see the dispatch_send thread): combine() twice now raises before anything launches.

const int destPe = globalWarpId / warpsPerPe;
if (destPe >= npes) return;

const uint64_t tokenNum =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same underflow as in the SDMA combine send (EpCombineAsyncLLSendTransfer_body): a count of 0 makes tokenNum UINT64_MAX, and LsaShareFor(..., tokenNum * hiddenBytes) then has the grid peer-store far past the buffer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 84f6f76, same shape as the SDMA one. It reads the raw value and only sizes the transfer when announced > 1, so 0 (no dispatch) and 1 (zero tokens) both skip.

)
return table

def dispatch_send(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing enforces the dispatch → combine order, and the protocol depends on it. The recv counts are cleared only by combine's send leg, and destPeTokenCounter only by RecvCopy, so an unpaired call leaves state the next call misreads:

  • dispatch twice, no combine in between: the second RecvTransfer sees last round's non-zero count and passes immediately (see the wait in EpDispatchAsyncLLRecvTransfer_body). It then reads data that may not have arrived yet.
    • Reproduced on both 8×MI308X and 8×MI355X, with both sdma and lsa: wrong received counts (e.g. 560 vs 548) and wrong combine output, intermittently (1–2 of 6 attempts).
  • dispatch_send twice: the slot counter keeps growing past MaxNumTokensToSendPerRank with no bound check, so the second send writes into the next PE's slice.
  • combine twice: GPU fault, see the comment on the combine send transfer.

Plain dispatch() also sets _async_pending_dispatch_tokens, so dispatch(); dispatch_recv() re-runs the recv half without complaint.

Suggest a small state machine on the op (send → recv → combine_send → combine_recv) that raises on any out-of-order call. It costs nothing on the hot path and turns silent corruption or a hang into an error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0c0b3ba: a four-state machine on the op, idle -> sent -> dispatched -> combining -> idle. dispatch() and combine() each take both steps of their half. Every out-of-order call raises RuntimeError on the host before any launch. The error names reset(), which puts it back to idle. It is only armed for async_ll.

That covers each case you listed:

  • dispatch twice: refused
  • dispatch_send twice: refused. The slot counter is also bounded kernel-side now (748b620), so it can't write into the next PE's slice even if reached some other way.
  • combine twice: refused
  • dispatch(); dispatch_recv(): refused

The combine weight-fold refusal is also hoisted to the call. Before, it raised after _combine_impl had already staged input and after the state had advanced.

Tested in 5ad0e06 (run_call_order: 14 out-of-order sequences and the weight-fold refusal, plus recovery by finishing the open round and by reset()). With the check disarmed, every rank reports dispatch twice: was accepted and the run fails.

if (blockId == 0 && warpId == 0 && laneId < npes) {
const uint64_t* slot =
reinterpret_cast<const uint64_t*>(recvCounts + asyncll::RecvCountSlot(laneId));
while (core::AtomicLoadRelaxedSystem(const_cast<uint64_t*>(slot)) == 0) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wait is where an unpaired dispatch goes wrong: 0 means "not arrived", but nothing clears the slot except combine's send leg. After dispatch(); dispatch() the second round finds last round's count still here and proceeds before its own payload lands (reproduced, see the comment on dispatch_send).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The kernel can't tell a stale count from a fresh one here, so the fix is upstream of it: the host check in 0c0b3ba refuses the second dispatch before this wait can run. The new back-to-back scenario (5ad0e06) runs six rounds with no host sync between them, so ranks can run ahead of each other, and it checks every round afterwards.

# what the cross-node leg will reuse. It is therefore NOT the fastest way to
# dispatch on a single node -- the ordinary v2 intranode kernels are, by about
# 2x on MI355X -- and it is off by default for that reason.
async_ll: bool = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

async_ll=True is silently ignored on the default kernel_backend (FlyDSL). The op builds and runs the ordinary kernels; only the split API raises later.

I hit this with test_graph_capture.py plus async_ll=True: it reported backend=flydsl and "passed" without running AsyncLL at all.

Suggest rejecting async_ll=True for a backend that does not implement it, or selecting hip automatically, so a config never means something other than what it says.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b66f409. It's another clause in the flydsl backend's _unsupported, so construction fails with a message pointing at kernel_backend='hip'. I went with rejecting rather than switching to hip automatically, so the config doesn't silently mean something other than what it says.

#
# Pick "lsa" for a small-batch, empty-seam call; keep "sdma" otherwise, when
# the point is the overlap, or when this op will later run across nodes.
async_ll_transport: str = "sdma"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The best default differs by arch. Measured EP8, hidden 7168, top-k 8, 32 experts/rank, dispatch+combine with no compute in the seam (µs):

v2 native asyncll sdma asyncll lsa
MI308X, 128 tok 167 329 (3 queues) 180
MI308X, 512 tok 387 895 (3 queues) 458
MI355X, 128 tok 150 166 161
MI355X, 512 tok 319 316 351

On MI308X sdma is about 2× slower than lsa. A GEMM in the seam hides much of that gap but does not make sdma a better default there; on MI355X sdma is the better choice at 512 tokens. Suggest picking the default per arch rather than one global "sdma".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the table. I haven't changed the default. Data from gfx1250 to go with yours (EP4, hidden 7168, top-k 8, bf16, graph, pair us, empty seam):

tok/rank v2 native asyncll sdma asyncll lsa
64 65.7 90.9 676.4
512 95.2 155.8 717.6
2048 246.1 355.4 972.9

sdma wins at every size there, and by a lot. So "sdma" stays the right default on gfx1250 and on MI355X at 512, and MI308X is the case that wants "lsa". A per-arch default makes sense, but I'd rather do it in a separate change that measures more points than three. For now MI308X users can pass async_ll_transport="lsa". I agree that at 128 tokens native v2 is the better choice everywhere.

if (sameNode) {
// Our payload to this peer is out of the queue before its count word
// goes, so the peer seeing a non-zero count has the tokens too.
sdma.quietQueue<::mori::cco::ccoCoopWarp>(destPe - args.lsaBase,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optimization idea (no change needed for this PR): the count could ride the same SDMA queue right behind the payload, which would remove this drain-then-announce step from the recv half.

cco already relies on copy → ATOMIC ordering within one queue: test_sdma_signal part B checks that a remote signal implies the data has landed. Sending tokenNum + 1 as an atomic add of that value, right after the put in SendTransfer, would put the count on the peer while the caller is still computing, instead of after its seam.

How much is at stake, from my runs: on MI355X at 512 tokens with a seam GEMM long enough to hide the transfers, about 120 µs per dispatch+combine pair stays exposed. That is the send/recv copy passes plus the two count/epoch exchanges, and I estimate the exchanges at roughly 20 µs each.

It needs a cco entry point for "put + remote atomic add of a value at a window offset" (today ccoSdma only increments its own signal pool). The cross-node leg would need a count per QP, as your README notes. The epoch handshake in combine could use the same trick.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d59607f does a version of this. On the SDMA path, the count is queued on the same queue right behind its payload during the send phase, so it lands on the peer while the caller is computing. It's a put of a staged word rather than a remote atomic add: the slot is zeroed every round, so the two are equivalent. It's on by default only for single-node gfx1250 SDMA, where it measured 58.65 -> 44.82 us of extra pair cost (T64, four GEMMs in the seam). On MI355X it's opt-in via MORI_ASYNCLL_EARLY_NOTIFY=1, and I haven't measured it there. LSA and cross-node keep the old protocol.

)
op.dispatch_recv()
torch.cuda.synchronize()
dist.barrier()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test coverage gaps. The new test is not in CI: ci.yml only runs v1's test_dispatch_combine_async_ll.py, so nothing guards v2 AsyncLL after merge. What it does not check today:

  • Received outputs: the received weights, expert ids and scales are never compared. The dispatch_recv weights bug fixed in a0b9dfc lived exactly here, and this test still would not catch a regression of it.
  • Back-to-back rounds: every phase is followed by synchronize() + dist.barrier(), which closes the windows where the count-clear and epoch races live. The LSA race you fixed was found by a benchmark, not by this test.
  • Graph replay with changing inputs: identical inputs every replay cannot tell stale data from fresh.
  • Out-of-order calls (see the dispatch_send comment).

For reference, I ran these on 8×MI308X and 8×MI355X and they all pass with this PR (MI308X with ≤3 SDMA queues):

  • back-to-back 6 rounds with no host sync, checking every received row bit-exactly (payload, weights, expert ids) against its decoded origin;
  • 8 graph replays with new inputs each replay.

One trap for whoever writes the graph check: routing.disp_tok_id_to_src_tok_id_local is cached on first access, and a captured handle lives across replays, so it returns the first replay's map. Read the live arena view instead. Happy to share the harness.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All four gaps are covered now, and the test runs in CI.

  • Received outputs and graph replay with changing inputs: added in d59607f. Hidden, indices and weights are compared row by row, including a zero-input rank receiving its peers' weights. Each replay updates the payload and weights, and the reverse map is read after dispatch_recv inside the captured function, which avoids the caching trap you mentioned.
  • Back-to-back rounds: 5ad0e06. Six rounds with no host sync, with per-round token counts (zero included), alternating split/ordinary APIs and one unweighted round. Results are copied off on the stream and checked after the last round.
  • Out-of-order calls: 5ad0e06, run_call_order.
  • CI: 023ec0c adds a step to the CCO unit test job, EP8, sdma then lsa. It passed on 8x MI355X: # PASS: 0 failed checks across 8 ranks, ... 6 back-to-back rounds, call order for both transports.

To check the new scenarios can fail, I removed each guarded fix from an installed build and reran. With the call-order check disarmed, every rank reports accepted calls. With a0b9dfc reverted, back-to-back fails on exactly the split rounds that carry weights.

Thanks for offering the harness. I ended up not needing it, but it's good to know it exists for the MI308X side.

# a count that never arrives. A hang with no message is the worst way to
# learn about an unset environment variable.
if cfg.async_ll_transport == "sdma" and os.environ.get(
"MORI_ENABLE_SDMA", ""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Untested, from reading: this only checks the MORI_ENABLE_SDMA env var. A host library built with BUILD_CCO_SDMA=OFF (which gemm_ar/op.py checks via _build_flags) would get 0 queues, every SDMA put would be a no-op, and the receiver would spin forever.

The toolchain.cpp comment says the queue count coming back 0 is reported by a host-side gate, but I could not find that gate. Checking _build_flags.BUILD_CCO_SDMA and the handle's actual queue count here would close it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your reading is right, and it's still open. ccoSdmaSetupCommQueues logs "this mori was built BUILD_CCO_SDMA=OFF ... every ccoSdma put on this comm will move no bytes" but refuses nothing, and the receiver then spins. I'll add a construction-time check of the handle's actual queue count for async_ll_transport="sdma", which also catches the env-off case, as a follow-up. For CI, the new step sets MORI_ENABLE_SDMA=1, and a runner that lost SDMA would fail it on the timeout rather than pass.

ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 16, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1. The normal one keeps the
  ordinary kernels and, under SDMA, runs the AsyncLL send and receive halves
  back to back. The new low-latency one sends in dispatch_a / combine_a and
  receives in dispatch_b / combine_b, so with two-batch overlap the sibling
  micro-batch computes while a transfer drains. Both return the normal
  dispatch output, keeping the combine staging view. Under AsyncLL: no comm
  stream and MORI's own launch geometry, fp4 dispatch / quantized combine
  fall back to bf16, the full receive view is kept, and the cco window is
  sized from MORI's asyncll_regions. init_mori_epv2_op takes deepep_mode and
  enable_sdma like init_mori_op. MORI EPv2 reaches SDMA only through the
  AsyncLL halves, so the normal dispatcher names its fallbacks and warnings
  after SDMA / low_latency and keeps AsyncLL to the one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
xiangch and others added 6 commits October 16, 2026 03:25
Some gfx1250 KFD nodes expose peer io_links but no CPU io_link. Fall
back to the union of their recommended SDMA engine masks so loopback
channels can still spread across engines instead of all using engine 0.

Keep the fallback inside the existing gfx1250 loopback path. This is the
common prerequisite used by both arms of the C12 overlap experiments.
Queue each ready word after its payload so receive completion can progress
under the caller's computation instead of waiting for the sender's recv phase.
Enable early notification and omit the redundant receive-count reset by
default for single-node gfx1250 SDMA. Preserve outgoing completion, self
notifications and zero-token peers; keep receive fusion opt-in.

Carry the resolved protocol in each Plan's host request/config so the Python
sequence, generated kernel and JIT cache agree. LSA and multi-node ops retain
the legacy protocol even when the SDMA environment options are enabled.
Fused receive uses one waiting wave per CTA followed by acquire/barrier.

Preserve the latest receive-weights fix and cover exact received metadata,
weighted/unweighted calls, ordinary/split APIs and zero-token graph replays.

Validation: fresh C12 gfx1250 build; 35 C++ tests, host/device split check and
5 Python policy tests; six EP4 GPU variants with 5 shapes x 4 iterations x
2 APIs and two zero-token graph scenarios x 32 replays, all passing. A T64
host-clock smoke with four GEMMs per seam reduced extra pair cost from
58.65 to 44.82 us for the default path (one process, five 256-pair samples).
…hold

AsyncLL wants one SDMA queue per peer, so it set sdmaQueueCount to
min(8, gpu_per_node) unconditionally. On 8xMI308X that is fatal rather than
slow: its xGMI engines fold to the even ids {2,4,6} and ROCr's own blit queues
take part of that, so a pair fits three. The eighth hsaKmtCreateQueueExt
returns HSAKMT_STATUS_NO_MEMORY, and anvil's CHECK_HSAKMT_SUCCESS is an
exit(EXIT_FAILURE) -- every rank leaves the process at anvil.cpp:210 and the op
is never built. Reported with ERROR code: 6 on all 8 ranks (ROCm 7.14); with
the count at 1, 2 or 3 the whole AsyncLL suite passes there, so the protocol
was never the problem, only the number.

Two changes, both on the EP side, because by the time the request is refused
the process is already gone -- there is nothing to recover from:

  * cap per arch for the archs known not to fit eight (gfx942 -> 3). The count
    is a performance knob, not a correctness one: fewer queues than peers folds
    two issuing warps onto one queue, which still commits in order.
  * defer to MORI_SDMA_NUM_CHANNELS when it is set. That env knob already
    existed and this request silently beat it, so an operator who knew what
    their machine could take had no way to say so short of editing the source.

_make_dev_comm takes the arch for this; it stays a staticmethod and the caller
passes self._arch, which is resolved well before.

Reported by jhchouuu on #768, reproduced on 8xMI308X.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ount

The recv-count slot is biased by one so that a rank sending zero tokens still
publishes a value distinguishable from "has not arrived". Both combine send
transfers subtracted that bias before testing it, so a slot that is legitimately
0 -- combine() twice on one routing, combine() before any dispatch,
combine_recv() after a whole combine() -- produced UINT64_MAX, which the
`tokenNum == 0` guard below cannot catch.

What that turns into differs by transport and neither is survivable:

  * sdma: sdma.put is handed tokenNum * hiddenBytes, a wrapped ~2^64-byte copy.
    Reproduced on 8xMI308X -- dispatch, combine, combine again faults every GPU
    in the job (Memory access fault, PERMISSION_FAULTS in dmesg).
  * lsa: LsaShareFor splits that same count across the grid and every warp peer
    stores far past the buffer.

Test the raw announced value instead and skip when it is 0. This does not make
the call correct -- the round is still wrong, and the host now refuses it before
the launch -- it only keeps a caller's mistake from becoming a GPU fault.

Reported by jhchouuu on #768 (combine twice reproduced on 8xMI308X; the lsa
twin found by reading).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AsyncLL's signal slots carry the coupling between calls, and nothing checked
it. An unpaired call does not fail where it is made; it leaves state the NEXT
call misreads:

  * dispatch twice, no combine between: the recv counts are cleared only by
    combine's send leg, so the second RecvTransfer finds the previous round's
    non-zero count and its "0 means not arrived" wait passes at once. It then
    reads a payload that may still be in flight. Reproduced on 8xMI308X and
    8xMI355X, both transports, as wrong received counts (560 against 548) and
    wrong combine output, in 1-2 of 6 attempts.
  * dispatch_send twice: the slot counter climbs past MaxNumTokensToSendPerRank
    unchecked and writes into the next PE's slice.
  * combine twice: the count is already 0 and the biased read underflows (the
    kernels now skip rather than fault, but the round is still wrong).
  * cross-node combine twice: the epoch is an add of 1 per combine while
    crossDeviceBarrierFlag moves once per dispatch, so one unpaired call shifts
    them permanently and the wait never matches -- a hang only reset() clears.

So: a four-state machine on the op, idle -> sent -> dispatched -> combining,
with dispatch() and combine() taking both steps of their half. It raises at the
call that is wrong instead of corrupting a later round, costs one comparison on
a path that launches nine kernels, and is armed only for async_ll -- no other
family couples its calls this way. reset() returns it to idle, which is what
the error tells a caller to do.

Also hoist the combine weight-fold rejection to the call. The backend raised it
from inside the kernel wrapper, after _combine_impl had already staged `input`
into out_tok and after the order check had advanced the state, so a caller who
caught it was left with a half-written buffer; test_asyncll documents having hit
exactly that ordering hazard. The backend's raise stays as a backstop.

Reported by jhchouuu on #768.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only the hip backend compiles the AsyncLL kernel family, and flydsl is the
default. Asking the default backend for async_ll=True built the ordinary
kernels and ran them: test_graph_capture.py with async_ll=True reported
backend=flydsl and passed without AsyncLL being involved at all. Only the split
dispatch_send/dispatch_recv API raised, so a caller using plain
dispatch()/combine() never found out that the config meant something other than
what it said.

_unsupported is the hook this backend already uses to reject a config at
construction (internode, fp4, asymmetric-dtype quant); this is one more clause
in it, pointing at kernel_backend='hip'.

Reported by jhchouuu on #768.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xiangch and others added 2 commits October 16, 2026 06:37
"AsyncLL v2 SDMA overlap" is 23 characters and its ~ underline was 22, which
Sphinx reports as "Title underline too short". build-docs treats warnings as
errors, so it failed. Added with d59607f; that push landed while #768 was in
conflict with main, when no workflows ran, so this is the first build to see it.

Co-Authored-By: Claude <noreply@anthropic.com>
…erflow

84f6f76 stopped an unpaired combine from wrapping its byte count, but only on
the legacy path. The early-notify path that d59607f added -- on by default for
single-node gfx1250 SDMA -- reads the recv count in CombineSendCopy instead, and
did the same `count - 1` into the send-count slot. A 0 there (no dispatch
announced this round) became UINT64_MAX, and SendTransfer multiplies that into
the SDMA copy size.

Keep 0 as 0. The host call-order check already makes this unreachable through
the API; this is the kernel-side backstop 84f6f76 was meant to be, now on both
paths. Every other read of a recv-count slot either waits for it to be non-zero
first or tests the raw value.

test_asyncll (sdma, which takes this path by default on gfx1250) passes on
4x gfx1250.

Co-Authored-By: Claude <noreply@anthropic.com>
@yangyuhuiling

Copy link
Copy Markdown
Contributor Author

@jhchouuu thanks for the thorough review, especially the MI308X runs and the reproductions. Status on everything, with replies inline:

Your four pre-merge items: all addressed

Fix
1. MI308X can't build the op 29966fe: per-arch cap (gfx942 -> 3), and MORI_SDMA_NUM_CHANNELS is respected. a3767ba: CHECK_HSAKMT_SUCCESS evaluated its call twice on failure
2. combine underflow -> GPU fault 84f6f76 (legacy path), e24c627 (the early-notify path)
3. dispatch twice -> wrong data, cross-node drift 0c0b3ba: host-side call-order state machine. 748b620: send slots bounded kernel-side
4. async_ll=True ignored on FlyDSL b66f409: refused at construction

Tests and CI: 5ad0e06 adds back-to-back rounds with no host sync, and out-of-order calls with recovery. 023ec0c runs test_asyncll.py in the CCO unit test job on both transports, green on 8x MI355X.

Still open, as follow-ups:

  • per-arch default transport (gfx1250 numbers inline; sdma wins there at every size)
  • a construction-time check that SDMA queues actually exist (BUILD_CCO_SDMA=OFF or env off)
  • the _async_pending_* retention nit
  • MI308X engine use (engine 0, or an sdma/lsa hybrid)

The count-behind-payload idea is partly in d59607f, default on gfx1250 only.

Not verified by me: anything on MI308X, since I have no access. If you can rerun test_asyncll.py there with the default config, that would confirm item 1. Cross-node is also unverified; I have no two-node run since these changes.

Also on the branch since your review: main merged in (a1c4423), resolving the anvil conflict with #766, which was the same fix landed twice. Plus a one-character docs fix that was failing build-docs.

xiangch and others added 2 commits October 16, 2026 07:37
The only guard was an env check: MORI_ENABLE_SDMA unset is refused before
anything is built. It is not the only way to get zero queues. A library built
BUILD_CCO_SDMA=OFF, or peers that do not report SDMA capability, come back with
sdmaNumQueue == 0 too, and cco only logs it. Every ccoSdma put on that
communicator then moves no bytes and the receiver spins on a count that never
arrives -- a hang, where an error belonged. jhchouuu found this by reading on
#768 and could not find the host-side gate the toolchain comment refers to.

Ask the handle what it actually connected. ccoDevComm already carried
sdma.sdmaNumQueue and the .pxd already declared it; this exposes it as
DevComm.sdma_num_queue / DevCommHandle.sdma_num_queue and refuses the op in
_make_dev_comm when transport is sdma, there are same-node peers, and the count
is 0. A node of one makes no SDMA put, so 0 is correct there and allowed. The
env pre-check stays: it is earlier and names the usual cause.

Verified on 4x gfx1250. With SDMA up the handle reports 4 queues (EP4) and the
op builds. With MORI_ENABLE_SDMA unset and the env pre-check bypassed in-process,
the comm comes back with 0 and every rank refuses construction. test_asyncll
passes on sdma and lsa.

Co-Authored-By: Claude <noreply@anthropic.com>
dispatch() and combine() recorded the state the split recv halves need --
the input tensor, the dest map, and for combine a tuple around them -- on every
call, for every kernel family, and nothing ever cleared it. That held the
previous round's activations alive until the next call. Reported as a nit by
jhchouuu on #768.

Record it only when the call WAS a send half (the kernel table passed in is not
the ordinary one), and have dispatch_recv drop its references once it has
launched, as combine_recv already did for its tuple. Neither recv pass reads the
input after launch, and the dest map is op-owned.

Verified on 4x gfx1250 with weakrefs: a plain dispatch() no longer keeps its
input alive, dispatch_send keeps it until dispatch_recv launches, and
dispatch_recv releases it. test_asyncll passes on sdma and lsa.

Co-Authored-By: Claude <noreply@anthropic.com>
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 16, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
@jhchouuu

Copy link
Copy Markdown
Collaborator

Thanks for the fast turnaround. I re-tested 11483df5 on 8×MI308X (ROCm 7.14) and 8×MI355X.

  • MI308X, default config: test_asyncll.py passes on both transports with no queue or env overrides, which confirms item 1. It also passes on MI355X.
  • Misordered calls (dispatch twice, combine twice) are now refused with RuntimeError on both archs. There is no GPU fault any more.
  • My extra checks pass on both archs, with early notify off and on:
    • back-to-back rounds, with every received row checked bit-exactly;
    • graph replay with new inputs on every replay.
  • Early notify under heavier load on MI308X also held: M=1024, 12 back-to-back rounds, run twice, with every row bit-exact.

Early notify, measured (EP8, hidden 7168, top-k 8, sdma, GEMM in the seam, dispatch+combine µs, interleaved A/B ×2):

off on
MI308X, 512 tok, 1024-row GEMM 1817–1822 1769–1772
MI308X, 128 tok, 256-row GEMM 553 503–504
MI355X, 512 tok, 4096-row GEMM 526–527 450–462
MI355X, 128 tok, 256-row GEMM 223–236 181–189

It saves about 50 µs per pair on MI308X and 40–70 µs on MI355X. On MI355X at 128 tokens it turns the seam from break-even into a 15–19% win, so I'd suggest enabling it by default on gfx942 and gfx950 too, not only gfx1250.

Non-blocking follow-ups:

  1. Early notify relies on copy → copy visibility order within one SDMA queue. cco only tests copy → atomic (test_sdma_signal part B). It would be worth pinning with a copy-then-copy case there, or sending the ready word as an atomic add so it stays inside what cco already guarantees.
  2. The queue cap keys on gfx942, so MI300X and MI325X get 3 queues too, although they have the full set of engines. Keying on MI308X's device id (0x74a2; anvil already has isMi308x) would avoid limiting parts that fit more.
  3. Per-arch default transport, as you planned.

LGTM from my side. Cross-node after these changes is still untested on my end too.

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two follow-up suggestions on how AsyncLL's arena layout is defined. Both are non-blocking and could be a small separate PR; details inline.

return regions


def asyncll_regions(cfg):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up suggestion (non-blocking; fine to land this PR as is): give AsyncLL's arena layout its own, explicit definition.

I understand why it lives here. AsyncLL shares EpInterNodeArgs, and hip_backend.py binds every off* field by region name (offStaging=offset("inter_staging"), ...), so the names have to be the internode ones. Sharing rather than restating also keeps the two families from drifting. But deriving the layout as "take internode_regions(cfg), then override" has a few costs:

  1. Silent coupling. Any later change to internode_regions changes AsyncLL's arena without touching this function: a new region, or a different size formula for one that isn't overridden here. Nothing would flag it, because both families share one window.
  2. Regions allocated but never read. The AsyncLL kernels never read offNodeRecvTokenNum, offDispTokOffset, offChunkFlag, offInpWeights or offCombineOutWeights, yet their regions are still carved. They are small; this is more about the layout saying what the family actually uses.
  3. No unit test. test_internode_regions.py only covers internode_regions. AsyncLL's sizes, which are per-rank SendBufSlotOffset extents and the hidden-only combine stride, are checked only end-to-end on GPU.

Suggested shape for a follow-up PR:

  • Move this to its own module, written as the full list of regions AsyncLL uses with its own formulas, instead of copy-then-override.
  • Bind names to EpInterNodeArgs fields through an explicit name → field map for AsyncLL, so regions can be named for what they hold (see the comment on inter_dispatch_staging below).
  • Add a pure-Python test like test_internode_regions.py: every region is at least as large as the furthest offset the kernels' slot arithmetic can reach, for world_size * max_send_per_rank slots.

I would keep sharing EpInterNodeArgs itself. A separate args struct would duplicate ~40 fields and the whole binding, which brings back the drift risk the sharing avoids. The AsyncLL-only fields there (offSendCount, offCombineStaging, dispReceiverIdxMap) are a cheap price for that.

)
else:
per_rank = max_send_per_rank
regions["inter_dispatch_staging"] = world_size * per_rank * hidden_bytes

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to the follow-up above: here inter_dispatch_staging holds AsyncLL's combine input, which is what combine_in_view() hands the caller's expert. That is a different thing from what the internode family stores under that name. Likewise inter_combine_inp keeps its name but changes stride (hidden-only, per-rank slots). The comments explain it, but the names now mislead anyone reading the backend or the kernel args.

With an explicit name → args-field map for AsyncLL, this could be called e.g. asyncll_combine_input and still bind to the same offset field.

@jhchouuu

Copy link
Copy Markdown
Collaborator

/ci-rerun

@github-actions

Copy link
Copy Markdown
Contributor

Re-running MoRI CI test and CCO CI test against the current main.

xiangch and others added 2 commits October 19, 2026 04:37
29966fe keyed the cap on the arch, gfx942 -> 3. MI300X, MI325X and MI308X are
all gfx942, and only MI308X is short of engines, so MI300X and MI325X were held
to three queues per pair despite having the full set. Pointed out by jhchouuu on
#768.

Key it on the model instead. mori.ops.utils.detect_model() already resolves the
PCI device id (0x74A2 -> "mi308x") from KFD sysfs and hip_tuning_configs uses it
for the same reason; anvil singles MI308X out the same way (AnvilLib::isMi308x).
Calling it inside _make_dev_comm also drops the `arch` parameter 29966fe had
added for this alone, so that signature is back to what main has.

Checked in isolation: mi308x -> 3, mi300x/mi325x/mi355x -> 8, an unknown part
(gfx1250, None) -> its peer count, and MORI_SDMA_NUM_CHANNELS still defers.

Co-Authored-By: Claude <noreply@anthropic.com>
Picks up 11 commits, notably bba3dd8 (#790), which fixes test_peer_pool's
FailingEngine against the new CreatePlanBuilder interface. Without it the PR's
"mori umbp unit test" job fails to compile when run against current main, as
the /ci-rerun on 10-16 did; nothing in this PR touches UMBP.

No conflicts. The only file here near this PR's area is
src/cco/device/cco_device_wrapper.cpp (#786, new SDMA bitcode entry points),
which the AsyncLL kernels do not use.

Co-Authored-By: Claude <noreply@anthropic.com>
@jhchouuu

Copy link
Copy Markdown
Collaborator

@yangyuhuiling #733 has landed on main (eb211db), so this branch now conflicts in .github/workflows/ci_cco.yml and dispatch_combine_op.py. Could you merge main again? The two conflict hunks are simple: keep both CI steps, and in _dispatch_impl keep kernels[disp_spec] and add #733's snapshot branch. But I did a trial merge, and two things outside the hunks break after it:

  1. hip_backend.py:1444, KernelSet(self_resets_counters=skip_reset). fix(EPv2): give routing handles their own buffers on request (snapshot=True) #733 removed self_resets_counters from KernelSet, together with the host-side total_recv.zero_(). After the merge, every AsyncLL op fails at construction with a TypeError. The fix is to drop the kwarg. MORI_ASYNCLL_SKIP_RESET and the skip_reset that _asyncll_options returns then have nothing left to control. AsyncLL never needed the host zero: RecvCopy assigns totalRecvTokenNum (ep_asyncll_kernel.hpp, ~L700).
  2. snapshot is not passed into _dispatch_impl. The auto-merge puts fix(EPv2): give routing handles their own buffers on request (snapshot=True) #733's if snapshot ... checks into _dispatch_impl, which has no snapshot parameter, and dispatch() doesn't pass it. Every dispatch() on every backend then raises NameError. Add a snapshot parameter, pass snapshot=snapshot from dispatch(), and pass False from dispatch_send().

It is also worth adding self.cfg.async_ll to #733's snapshot guard. AsyncLL's combine reads the latest dispatch's arena state (dispReceiverIdxMap, the per-peer recv counts) as well as the handle's dest map, so a snapshot can't outlive the next dispatch. The call-order state machine already turns misuse into a RuntimeError; rejecting dispatch(snapshot=True) up front just gives a clearer message.

Two small things from reading how AsyncLL plugs into the JIT (non-blocking):

  • EpAsyncLLCfgFromFields passes EpInterNodeKernel::Dispatch to MakeEpInterNodeCfg only because the parameter isn't optional. So the Cfg, and with it plan.info, carries the blockNum from the internode dispatch rule, which AsyncLL never launches with. A geometry rule of its own for the AsyncLL family in MakeEpInterNodeCfg, or leaving blockNum unset for it, would keep info accurate.
  • toolchain.cpp now adds -DBUILD_CCO_SDMA=1 to every JIT TU. I checked that this is safe: the macro only gates the device-side ccoSdma class, so the ccoDevComm layout doesn't change; it's in the flags, so it's in the cache key; and the construction-time queue check covers a library built without SDMA. But it differs from cco/device/bitcode.py, which follows the baked build flag, and it invalidates every existing JIT cache entry once on upgrade. Following the baked flag here too would keep the two JIT paths consistent.

Once the merge is in, I'll rerun test_asyncll.py (both transports) and #733's test_routing_snapshot.py on MI355X, and then approve.

xiangch and others added 2 commits October 20, 2026 06:08
Two textual conflicts and two that git merged cleanly but broke.

Textual:
  .github/workflows/ci_cco.yml -- both sides added a step after backend
    parity (this PR's AsyncLL step, main's routing snapshot step); keep both.
  dispatch_combine_op.py -- both edited the dispatch dest-map choice: this PR
    takes the kernel table as a parameter (for the split API), #733 adds the
    snapshot branch. Combined.

Merged without a conflict marker, but broken:
  * dispatch() gained `snapshot` from #733 and _dispatch_impl's body uses it,
    but _dispatch_impl is this PR's function and its signature had no such
    parameter, and dispatch() did not pass it on -- every dispatch would have
    raised NameError. Added the parameter and pass it through.
  * #733 removed KernelSet.self_resets_counters (with the host total_recv
    reset it gated), but the AsyncLL KernelSet still passed it, so every
    async_ll op would have failed construction with a TypeError. Dropped. That
    reset was redundant for AsyncLL in every mode -- its RecvCopy assigns
    totalRecvTokenNum rather than accumulating -- so MORI_ASYNCLL_SKIP_RESET
    no longer selects anything. It is still parsed, so existing settings
    validate, and docs/api/communication.rst now says it has no effect.

Also refuse dispatch(..., snapshot=True) on async_ll. A snapshot handle exists
to be combined after later dispatches; AsyncLL's combine reads
dispReceiverIdxMap, which every dispatch's RecvCopy rewrites, and its call-order
check refuses exactly the dispatch-A, dispatch-B, combine-A pattern anyway.

Co-Authored-By: Claude <noreply@anthropic.com>
0c0b3ba advanced the call-order state at the top of each entry point, before
the body. A call that the body then refused for its ARGUMENTS -- more tokens
than max_num_inp_token_per_rank, routing= together with return_routing=, or the
new snapshot=True on async_ll -- raised after the state had moved on, for a
launch that never happened. The caller's next, correct call was then refused as
out of order:

  dispatch(..., snapshot=True)  -> ValueError (correct)
  dispatch(...)                 -> RuntimeError: AsyncLL is 'dispatched'

Found while validating the snapshot refusal from the main merge.

The check and the advance are now split: _asyncll_call is a context manager
that refuses an out-of-order call up front and advances only when the body
returns, so an exception leaves the state where it was. All six entry points
use it; the combine weight-fold refusal stays ahead of it, as before.

run_call_order gains two argument-level refusals from idle (over capacity, and
snapshot on async_ll) followed by the existing verified round, which is what
catches it: with the old ordering restored in the installed package, every rank
fails there with "AsyncLL is 'dispatched'", rc=1.

On 4x gfx1250 (C14): test_asyncll passes on sdma and lsa.

Co-Authored-By: Claude <noreply@anthropic.com>
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 20, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 20, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 26, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 27, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 27, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Oct 27, 2026
Drive MORI EPv2's AsyncLL kernels (ROCm/mori#768) with the same switches as
EPv1: --deepep-mode low_latency or MORI_ENABLE_SDMA=1. The transport is MORI's
copy engine ("sdma") with MORI_ENABLE_SDMA=1 and peer stores ("lsa")
otherwise.

- moriep: split the EPv2 dispatcher like EPv1: _MoriEPv2DispatcherImplBase
  holds the op setup, dtype handling and dispatch/combine post-processing,
  and the normal and low-latency dispatchers only differ in transport. The
  normal one keeps the ordinary kernels and, under SDMA, runs the AsyncLL
  send and receive halves back to back. The new low-latency one sends in
  dispatch_a / combine_a and receives in dispatch_b / combine_b, so with
  two-batch overlap the sibling micro-batch computes while a transfer
  drains. Both return the normal dispatch output, keeping the combine
  staging view. Under AsyncLL: no comm stream and MORI's own launch
  geometry, fp4 dispatch / quantized combine fall back to bf16, the full
  receive view is kept, and the cco window is sized from MORI's
  asyncll_regions. init_mori_epv2_op takes deepep_mode and enable_sdma, and
  its leading parameters now follow init_mori_op's order and names
  (max_tokens_per_rank -> num_max_dispatch_tokens_per_rank); the EPv2-only
  max_total_recv_tokens and launch_config come last. MORI EPv2 reaches SDMA
  only through the AsyncLL halves, so the normal dispatcher names its
  fallbacks and warnings after SDMA / low_latency and keeps AsyncLL to the
  one _uses_async_ll check.
- moe_hook: low_latency no longer rewrites to normal on EPv2, and SDMA no
  longer forces EPv1. Asking for AsyncLL on a MORI build without EPv2 AsyncLL
  falls back to EPv1 unless SGLANG_MORI_EP_V2=true.
- two_batch_overlap / moe utils: MoeA2ABackend.low_latency_supports_extend()
  (true on MORI EPv2) lets low_latency keep extend-batch TBO.
- docs: describe the AsyncLL switches under SGLANG_MORI_EP_V2.

Needs a MORI build whose dispatch_recv() keeps the send half's weights;
PR sgl-project#768 at 616dfdfa drops them (recv topk weights read back as zero).
@yangyuhuiling
yangyuhuiling merged commit 0304aa6 into main Oct 27, 2026
21 of 25 checks passed
jhchouuu added a commit that referenced this pull request Oct 28, 2026
Resolves the conflicts with #768 (AsyncLL):
- AsyncLL, like internode, sizes its arena from its own formula
  (asyncll_regions), so the op describes no intranode plans for it.
- The TokOffExt decision, taken before the gate, also excludes AsyncLL.
- scale_stride_bytes() returns the packed row on AsyncLL, as on internode.
- tests/cpp/jit/test_jit_core.cpp keeps both the AsyncLL and the describe tests.
zhangfei829 added a commit that referenced this pull request Oct 28, 2026
Conflicts with #768 (AsyncLL on cco), both sides kept: hip_backend._build makes the device communicator for an AsyncLL op too and drops the tok_off slot-word decision, since TokOffExt is a stub on this branch; dispatch keeps the snapshot recv maps next to AsyncLL's pending send-half state; combine passes the snapshot maps to whichever kernel table runs.

ll128_layout returns None for an AsyncLL op: its kernel set holds only AsyncLL's plans, and a gfx1250 bf16 or fp8 AsyncLL op past the table's max_tok would otherwise pick the LL128 spec and find no kernel.
QizhouZhang97 added a commit that referenced this pull request Nov 2, 2026
Brings in AsyncLL (#768) and plan-reported arena/staging sizes (#844).

Conflicts, resolved by keeping both sides:
- EpInterNodeRequest carries combineLoadFirst and asyncLlMode (20 fields).
- hip_backend: the internode path keeps the resolve_schedule KernelConfig
  specs; main's geometry-bucket helpers stay removed. The schedule is now
  resolved in __init__ so _describe() sizes the arena from the same variants
  that are built. AsyncLL is a one-entry schedule, requests every allocated
  QP with no drain count, and rejects packed fp4 and the active-QP knobs.
- internode test: AsyncLL weight handling alongside the active-QP switching;
  _family_that_ran reads kernel_config().

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants