Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,11 @@ Benchmark result on DeepSeek V3 model configurations:
| **Kernels**| **# CUs**| **Dispatch XGMI** |**Dispatch RDMA** |**Combine XGMI**|**Combine RDMA** |
|------------|----------|-------------------|------------------|----------------|-----------------|
|EP8 | 80 | 307 GB/s | x | 330 GB/s | x |
|EP16-V0 | 32 | 75 GB/s | 23 GB/s | 76 GB/s | 23GB/s |
|EP16-V0 | 80 | 79 GB/s | 24 GB/s | 82 GB/s | 25GB/s |
|EP16-V1 | 32 | 185 GB/s | 57 GB/s | 172GB/s | 52GB/s |
|EP16-V1 | 80 | 208 GB/s | 63 GB/s | 161GB/s | 49GB/s |
|EP16-V0 | 32 | 75 GB/s | 23 GB/s | 76 GB/s | 23 GB/s |
|EP16-V0 | 80 | 79 GB/s | 24 GB/s | 82 GB/s | 25 GB/s |
|EP16-V1 | 32 | 185 GB/s | 57 GB/s | 172 GB/s | 52 GB/s |
|EP16-V1 | 80 | 208 GB/s | 63 GB/s | 161 GB/s | 49 GB/s |
|EP32-V1-LL | 32 | 103 GB/s | 57 GB/s | 91 GB/s | 50 GB/s |

**Latency Performance**

Expand All @@ -49,6 +50,7 @@ Benchmark result on DeepSeek V3 model configurations:
|EP8 | 80 | 37 us | 176 GB/s | 99 us | 107 GB/s |
|EP16-V0 | 32 | 226 us | 33 GB/s | 296 us | 49GB/s |
|EP16-V1 | 32 | 115 us | 63 GB/s | 141 us | 110GB/s |
|EP32-V1-LL | 32 | 157 us | 48 GB/s | 280 us | 55GB/s |

**NOTE**: We show best performance values measured from multiple test rounds to eliminate fluctuations.

Expand Down
2 changes: 1 addition & 1 deletion examples/ops/dispatch_combine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Run the following command on each node and replace node_rank to its actual rank.
export GLOO_SOCKET_IFNAME=ens14np0
export MORI_RDMA_DEVICES=^mlx5_0,mlx5_1 # Optional: use `^` prefix to exclude specified devices

torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 --master_addr="10.194.132.29" --master_port=1234 examples/ops/dispatch_combine/test_dispatch_combine_internode.py --bench
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 --master_addr="10.194.132.29" --master_port=1234 examples/ops/dispatch_combine/test_dispatch_combine_internode.py --max-tokens 128 --cmd bench
```

The output of this scripit includes total number of tokens received, total number of RDMA tokens received and total bandwidth(include XGMI and RDMA). To calculate RDMA bandwidth, multiply the total bandwidth with (total # of RDMA tokens / total # of tokens);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
kernel_type_map = {
"v0": mori.ops.EpDispatchCombineKernelType.InterNode,
"v1": mori.ops.EpDispatchCombineKernelType.InterNodeV1,
"v1_ll": mori.ops.EpDispatchCombineKernelType.InterNodeV1LL,
}


Expand Down Expand Up @@ -862,7 +863,7 @@ def test_dispatch_combine(
type=str,
default="v1",
help="Type of kernel to test",
choices=["v0", "v1"],
choices=["v0", "v1", "v1_ll"],
)
args_cli = parser.parse_args()

Expand Down
1 change: 1 addition & 0 deletions include/mori/ops/dispatch_combine/dispatch_combine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ enum KernelType {
IntraNode = 0,
InterNode = 1,
InterNodeV1 = 2,
InterNodeV1LL = 3,
};

inline const char* HipDataTypeToString(hipDataType dtype) {
Expand Down
1 change: 1 addition & 0 deletions python/mori/ops/dispatch_combine.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ def get_dispatch_src_token_pos(self):
if self.config.kernel_type.value in (
EpDispatchCombineKernelType.IntraNode.value,
EpDispatchCombineKernelType.InterNodeV1.value,
EpDispatchCombineKernelType.InterNodeV1LL.value,
):
return self._get_dispatch_src_token_pos_func(self._handle)

Expand Down
5 changes: 4 additions & 1 deletion src/ops/dispatch_combine/dispatch_combine.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,8 @@ void EpDispatchCombineHandle::LaunchDispatch(KernelType kernelType, int blockNum
EpDispatchInterNodeKernel<<<grid, block, sharedMemSize, stream>>>(args);
} else if (kernelType == KernelType::InterNodeV1) {
EpDispatchInterNodeV1Kernel<<<grid, block, sharedMemSize, stream>>>(args);
} else if (kernelType == KernelType::InterNodeV1LL) {
EpDispatchInterNodeV1KernelLowLatency<<<grid, block, sharedMemSize, stream>>>(args);
} else if (kernelType == KernelType::IntraNode) {
EpDispatchIntraNodeKernel<DataT><<<grid, block, sharedMemSize, stream>>>(args);
} else {
Expand All @@ -303,7 +305,8 @@ void EpDispatchCombineHandle::LaunchCombine(KernelType kernelType, int blockNum,
if (kernelType == KernelType::InterNode) {
assert(config.useExternalInpBuffer);
EpCombineInterNodeKernel<<<grid, block, sharedMemSize, stream>>>(args);
} else if (kernelType == KernelType::InterNodeV1) {
} else if ((kernelType == KernelType::InterNodeV1) ||
(kernelType == KernelType::InterNodeV1LL)) {
assert(config.useExternalInpBuffer);
EpCombineInterNodeV1Kernel<<<grid, block, sharedMemSize, stream>>>(args);
} else if (kernelType == KernelType::IntraNode) {
Expand Down
Loading