Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/workflows/ci_cco.yml
Original file line number Diff line number Diff line change
Expand Up @@ -427,8 +427,16 @@ jobs:
# MORI_ENABLE_SDMA=1 matters for the sdma leg: without SDMA queues every
# put is a no-op and the receiver spins until the timeout, so a runner
# that quietly lost SDMA shows up as this step timing out, not passing.
for transport in sdma lsa; do
$CT exec -e MORI_ENABLE_SDMA=1 -e TRANSPORT=$transport $CONTAINER bash -c "\
#
# sdma runs once per notification protocol: early notification is the
# gfx950 default, so legacy and fused receive each need an explicit run.
for run in "sdma" "sdma MORI_ASYNCLL_EARLY_NOTIFY=0" "sdma MORI_ASYNCLL_FUSED_RECV=1" "lsa"; do
set -- $run
transport=$1; shift
envs="-e MORI_ENABLE_SDMA=1 -e TRANSPORT=$transport"
for e in "$@"; do envs="$envs -e $e"; done
echo "== test_asyncll $run"
$CT exec $envs $CONTAINER bash -c "\
cd $GITHUB_WORKSPACE/tests/python/ops/dispatch_combine_v2 && \
out=\$(timeout 900 torchrun --standalone --nproc_per_node=8 test_asyncll.py 2>&1) || \
{ echo \"\$out\"; exit 1; }; \
Expand Down
35 changes: 28 additions & 7 deletions docs/api/communication.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ clear win: a two-op microbenchmark gets slower, while DeepSeek-V4 prefill with
two-stream two-batch overlap got faster. It is off by default; measure your
workload before turning it on there. Only the ``"sdma"`` transport uses it.

On a single gfx1250 node, the send phase now posts each payload followed by
On a single gfx950 or gfx1250 node, the send phase now posts each payload followed by
its ready notification on the same SDMA queue. The receive phase still waits
for outgoing copies before the source buffers can be reused. Zero-token peers
also exchange notifications. The redundant dispatch receive-count reset is
Expand All @@ -98,8 +98,9 @@ These options are read once when an op is constructed:
* - Environment variable
- Default and effect
* - ``MORI_ASYNCLL_EARLY_NOTIFY``
- ``1`` on single-node gfx1250 SDMA; ``0`` elsewhere. Posts notifications
in the send phase. Other architectures can opt in with ``1``.
- ``1`` on single-node gfx950 and gfx1250 SDMA; ``0`` elsewhere. Posts
notifications in the send phase. Other architectures can opt in with
``1``.
* - ``MORI_ASYNCLL_SKIP_RESET``
- No longer has an effect. It skipped the host-side ``total_recv`` reset
before each dispatch; that reset has been removed for every kernel
Expand All @@ -109,16 +110,36 @@ These options are read once when an op is constructed:
* - ``MORI_ASYNCLL_FUSED_RECV``
- ``0``. Set ``1`` to fold receive waits into the copy passes. One wave
per CTA waits, then an acquire and CTA barrier release the copy waves.
* - ``MORI_ASYNCLL_FUSED_RECV_CTAS``
- ``0``, meaning half the compute units this process may use: the
device's, or its share under ``HSA_CU_MASK``. Grid ceiling for the
fused receive copies; ignored unless ``MORI_ASYNCLL_FUSED_RECV=1``.

All three variables accept ``0`` or ``1``. Skipping the reset or fusing receive
requires early notification. Set all three to ``0`` for the legacy sequence.
The first three variables accept ``0`` or ``1``. Skipping the reset or fusing
receive requires early notification. Set all three to ``0`` for the legacy sequence.
LSA and multi-node ops retain their existing protocol and ignore these SDMA
options. The resolved notification mode is part of every native Plan's
configuration and generated source, so different ops can coexist without
changing the protocol of an already constructed Plan.

The default targets single-node gfx1250 overlap. Receive fusion remains opt-in
because its additional benefit depends on the token count and compute window.
The default targets single-node gfx950 and gfx1250 overlap. On MI355X (gfx950,
EP8, hidden 7168, top-6), early notification beat the legacy sequence at every
token count from 128 to 4096: by 2-5% on one stream and 3-11% with two
overlapping streams. Receive fusion remains opt-in because its additional
benefit depends on the token count and compute window.

Receive fusion is a single-stream optimisation. A fused receive copy waits for
its peers inside every CTA, so its whole grid stays resident for the wait. When
two micro-batches overlap on two streams (two-batch overlap), that grid can
crowd out the send pass the other stream must run before any peer's wait can
end, and the ranks deadlock. The grid ceiling makes this unlikely but cannot
rule it out: it is sized from the compute units the process may use, not from
those another stream, kernel or process is occupying at the time.

Do not combine receive fusion with micro-batches on two streams, especially
when the device may be shared. On MI355X, two streams gained nothing from
fusion over early notification, so dual-stream callers should use early
notification alone, which is the gfx950 default.

MORI Shmem (Symmetric Memory)
-------------------------------
Expand Down
14 changes: 12 additions & 2 deletions include/mori/ops/dispatch_combine_v2/ep_internode_spec.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ struct EpInterNodeCfg {
// cross-compiling.
int mpCount{64};
int waveSize{64};
// Grid ceiling for the fused receive copies (asyncLlMode 2), resolved by
// MakeEpInterNodeCfg; 0 everywhere else. A fused copy waits for its peers
// inside every CTA, so its whole grid stays resident for the wait. Spanning
// the device, that can leave no room for the send pass another stream must
// run before any peer's wait can end -- a cross-rank deadlock.
int asyncLlFusedRecvCtas{0};
};

template <typename Self, typename Visit>
Expand All @@ -91,11 +97,12 @@ inline void VisitFields(Self& cfg, const EpInterNodeCfg& defaults, Visit&& visit
MORI_FIELD(rdmaBlockNum);
MORI_FIELD(mpCount);
MORI_FIELD(waveSize);
MORI_FIELD(asyncLlFusedRecvCtas);
#undef MORI_FIELD
}

MORI_JIT_ASSERT_FIELD_COUNT(
EpInterNodeCfg, 8, "added an EpInterNodeCfg field -- update VisitFields(EpInterNodeCfg) too");
EpInterNodeCfg, 9, "added an EpInterNodeCfg field -- update VisitFields(EpInterNodeCfg) too");

// The whole Cfg as text, for the plan's `info` and for logs. NOT the source
// text: RenderSource renders the kernel specialisation (and AsyncLL mode), not
Expand Down Expand Up @@ -183,6 +190,8 @@ struct EpInterNodeRequest {
int warpPerBlock = 0;
int rdmaBlockNum = 0;
int mpCount = 0;
// asyncLlMode 2 only; 0 = half the device (mpCount / 2).
int asyncLlFusedRecvCtas = 0;
};

template <typename Self, typename Visit>
Expand All @@ -206,11 +215,12 @@ inline void VisitFields(Self& request, const EpInterNodeRequest& defaults, Visit
MORI_FIELD(warpPerBlock);
MORI_FIELD(rdmaBlockNum);
MORI_FIELD(mpCount);
MORI_FIELD(asyncLlFusedRecvCtas);
#undef MORI_FIELD
}

MORI_JIT_ASSERT_FIELD_COUNT(
EpInterNodeRequest, 18,
EpInterNodeRequest, 19,
"added an EpInterNodeRequest field -- update VisitFields(EpInterNodeRequest) too");

std::string EpInterNodeRequestSchema();
Expand Down
78 changes: 77 additions & 1 deletion python/mori/ops/dispatch_combine_v2/hip_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,47 @@ def _give_back_sdma_queue_offset(device_index: int, offset: int) -> None:
del users[offset]


def _hsa_cu_mask_cus(device_index: int, mp_count: int):
"""CUs HSA_CU_MASK leaves this process on `device_index`, or None if unrestricted.

ROCr's syntax is `<gpus>:<cus>[;<gpus>:<cus>...]`, each side a comma list of
decimal indices or `a-b` ranges, e.g. `0-7:0-63`. GPU indices are ROCr's
enumeration order, which is the HIP/torch order unless HIP_VISIBLE_DEVICES
reorders devices. A setting this cannot read returns None, so the ceiling
falls back to the whole device rather than failing the op.
"""
raw = os.environ.get("HSA_CU_MASK", "").strip()
if not raw:
return None

def indices(text):
out = set()
for part in text.split(","):
part = part.strip()
if not part:
continue
lo, sep, hi = part.partition("-")
lo, hi = int(lo), int(hi) if sep else int(lo)
if lo < 0 or hi < lo:
raise ValueError(part)
out.update(range(lo, hi + 1))
return out

try:
for entry in raw.split(";"):
if not entry.strip():
continue
gpus, _, cus = entry.partition(":")
if device_index in indices(gpus):
found = indices(cus)
if mp_count > 0:
found = {c for c in found if c < mp_count}
return len(found)
except ValueError:
return None
return None


def _asyncll_sdma_queues(model, gpu_per_node: int) -> int:
"""How many SDMA queues per pair to ask cco for, 0 to leave the env default.

Expand Down Expand Up @@ -1413,7 +1454,10 @@ def flag(name, default):
raise ValueError(f"{name} must be '0' or '1', got {value!r}")
return value == "1"

early = flag("MORI_ASYNCLL_EARLY_NOTIFY", arch == "gfx1250")
# On by default where it has been measured to win: gfx1250, and gfx950
# (MI355X), where it beat legacy at every token count, most with two
# overlapping streams.
early = flag("MORI_ASYNCLL_EARLY_NOTIFY", arch in ("gfx950", "gfx1250"))
fused = flag("MORI_ASYNCLL_FUSED_RECV", False)
skip_reset = flag("MORI_ASYNCLL_SKIP_RESET", early)
if (fused or skip_reset) and not early:
Expand All @@ -1424,6 +1468,34 @@ def flag(name, default):
# Must match EpInterNodeCfg::asyncLlMode: legacy, early, fused CTA wait.
return (2 if fused else int(early)), skip_reset

@staticmethod
def _asyncll_fused_recv_ctas(mode, device_index=0, mp_count=0):
# Grid ceiling for the fused receive copies; 0 lets the native factory
# pick half the device. MORI_ASYNCLL_FUSED_RECV_CTAS is only a tuning
# override: the default exists so that fusion cannot starve another
# stream's sends into a deadlock.
name = "MORI_ASYNCLL_FUSED_RECV_CTAS"
value = os.environ.get(name, "0")
try:
ctas = int(value)
except ValueError:
ctas = -1
if ctas < 0:
raise ValueError(f"{name} must be a non-negative integer, got {value!r}")
if mode != 2:
return 0
if ctas > 0:
return ctas
# "Half the device" has to mean half of what this process can use. The
# native default is mpCount / 2, and mpCount is the whole device even when
# HSA_CU_MASK leaves the process a fraction of it: at 64 of 256 CUs two
# 128-CTA fused grids fill every slot, and the ranks deadlock as if there
# were no ceiling at all.
usable = _hsa_cu_mask_cus(device_index, mp_count)
if usable is not None and 0 < usable < mp_count:
return max(usable // 2, 1)
return 0

@classmethod
def _asyncll_seq(cls, cfg, mode=0):
lsa = cfg.async_ll_transport == "lsa"
Expand Down Expand Up @@ -1461,6 +1533,7 @@ def _asyncll_request(self, cfg, dtype_tag, block_num, warp_per_block):
# geometry table supplies grids directly.
request["mpCount"] = self._multi_processor_count
request["asyncLlMode"] = self._asyncll_mode
request["asyncLlFusedRecvCtas"] = self._asyncll_fused_ctas
return request

def _build_asyncll_kernels(self, cfg) -> KernelSet:
Expand All @@ -1480,6 +1553,9 @@ def _build_asyncll_kernels(self, cfg) -> KernelSet:
# KernelSet.self_resets_counters with it), and AsyncLL's RecvCopy
# always assigned the count. Still parsed so existing settings validate.
self._asyncll_mode, _ = self._asyncll_options(cfg, self._arch)
self._asyncll_fused_ctas = self._asyncll_fused_recv_ctas(
self._asyncll_mode, self._device_index, self._multi_processor_count
)
sequences = self._asyncll_seq(cfg, self._asyncll_mode)
for group, names in sequences.items():
for name in names:
Expand Down
10 changes: 10 additions & 0 deletions src/ops/dispatch_combine_v2/ep_asyncll_spec.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,16 @@ mori::jit::v2::LaunchGeometry EpAsyncLLGeometry(const EpInterNodeCfg& cfg, EpAsy
geometry.sharedBytes = static_cast<unsigned>(EpInterNodeCombineSharedBytes(cfg));
break;
}

// A fused copy holds its whole grid resident while it waits for its peers
// (see EpInterNodeCfg::asyncLlFusedRecvCtas). RecvCopy strides over its
// tokens, so a narrower grid only does the same work in fewer CTAs.
const bool fusedCopy = cfg.asyncLlMode == 2 && (kind == EpAsyncLLKernel::DispatchRecvCopy ||
kind == EpAsyncLLKernel::CombineRecvCopy);
if (fusedCopy && cfg.asyncLlFusedRecvCtas > 0) {
const auto cap = static_cast<unsigned>(EpAsyncLLAlignGrid(cfg.asyncLlFusedRecvCtas, worldSize));
if (cap < geometry.gridX) geometry.gridX = cap;
}
return geometry;
}

Expand Down
15 changes: 15 additions & 0 deletions src/ops/dispatch_combine_v2/ep_internode_spec.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@

#include "mori/ops/dispatch_combine_v2/ep_internode_spec.hpp"

#include <algorithm>
#include <stdexcept>
#include <string>

Expand Down Expand Up @@ -129,6 +130,20 @@ EpInterNodeCfg MakeEpInterNodeCfg(const std::string& arch, const EpInterNodeRequ
if (cfg.asyncLlMode != 0 && cfg.kernelCfg.worldSize != cfg.kernelCfg.gpuPerNode) {
throw std::runtime_error("mori ep v2: asyncLlMode > 0 requires a single-node SDMA plan");
}
if (request.asyncLlFusedRecvCtas < 0) {
throw std::runtime_error("mori ep v2: asyncLlFusedRecvCtas must be >= 0");
}
if (request.asyncLlFusedRecvCtas != 0 && cfg.asyncLlMode != 2) {
throw std::runtime_error(
"mori ep v2: asyncLlFusedRecvCtas only applies to asyncLlMode 2 (fused receive)");
}
// Half the device by default. On MI355X it keeps most of fusion's
// single-stream gain and removed the deadlock measured against a second
// stream's sends; a stream that fills most of the device can still starve it.
if (cfg.asyncLlMode == 2) {
cfg.asyncLlFusedRecvCtas = request.asyncLlFusedRecvCtas > 0 ? request.asyncLlFusedRecvCtas
: std::max(cfg.mpCount / 2, 1);
}

// Caught here rather than by hipModuleLaunchKernel, which reports it as a
// generic launch failure with no mention of which knob was too large.
Expand Down
43 changes: 43 additions & 0 deletions tests/cpp/jit/test_jit_core.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -628,6 +628,49 @@ TEST(AsyncLLMode, FactoryRejectsInvalidAndIncompatibleModes) {
EpInterNodeFamily::AsyncLL));
}

TEST(AsyncLLMode, FusedReceiveGridIsCappedAtHalfTheDeviceByDefault) {
const auto kinds = {EpAsyncLLKernel::DispatchRecvCopy, EpAsyncLLKernel::CombineRecvCopy};
// Only the fused mode resolves a ceiling, and only its receive copies use it.
for (int mode : {0, 1}) {
const auto cfg = AsyncLLCfgForMode(mode);
EXPECT_EQ(cfg.asyncLlFusedRecvCtas, 0);
for (const auto kind : kinds) {
EXPECT_EQ(EpAsyncLLGeometry(cfg, kind).gridX, static_cast<unsigned>(cfg.mpCount));
}
}
const auto fused = AsyncLLCfgForMode(2);
EXPECT_EQ(fused.asyncLlFusedRecvCtas, fused.mpCount / 2);
for (const auto kind : kinds) {
EXPECT_EQ(EpAsyncLLGeometry(fused, kind).gridX, static_cast<unsigned>(fused.mpCount / 2));
}
EXPECT_EQ(EpAsyncLLGeometry(fused, EpAsyncLLKernel::CombineSendCopy).gridX,
static_cast<unsigned>(fused.mpCount));

// An explicit ceiling is aligned down to worldSize and never widens the grid.
EpInterNodeRequest request;
request.worldSize = 4;
request.gpuPerNode = 4;
request.asyncLlMode = 2;
for (const auto& [ctas, grid] : std::vector<std::pair<int, int>>{{10, 8}, {1, 4}, {1000, 64}}) {
request.asyncLlFusedRecvCtas = ctas;
const auto cfg = MakeEpInterNodeCfg("gfx1250", request, EpInterNodeKernel::Dispatch,
EpInterNodeFamily::AsyncLL);
EXPECT_EQ(cfg.asyncLlFusedRecvCtas, ctas);
EXPECT_EQ(EpAsyncLLGeometry(cfg, EpAsyncLLKernel::DispatchRecvCopy).gridX,
static_cast<unsigned>(grid));
}

request.asyncLlFusedRecvCtas = -1;
EXPECT_THROW(MakeEpInterNodeCfg("gfx1250", request, EpInterNodeKernel::Dispatch,
EpInterNodeFamily::AsyncLL),
std::runtime_error);
request.asyncLlMode = 1;
request.asyncLlFusedRecvCtas = 16;
EXPECT_THROW(MakeEpInterNodeCfg("gfx1250", request, EpInterNodeKernel::Dispatch,
EpInterNodeFamily::AsyncLL),
std::runtime_error);
}

TEST(AsyncLLMode, ReceiveEntryAndDefinesUseThePlanMode) {
for (const auto kind : {EpAsyncLLKernel::DispatchRecvCopy, EpAsyncLLKernel::CombineRecvCopy}) {
const std::string stem = kind == EpAsyncLLKernel::DispatchRecvCopy ? "EpDispatchAsyncLLRecvCopy"
Expand Down
Loading
Loading