Skip to content

fix(EPv2): clamp the untuned flydsl combine block count to the xdb flag slots - #876

Merged
kawhil-amd merged 3 commits into
ROCm:mainfrom
kawhil-amd:fix/epv2-clamp-untuned-combine-blocks
Oct 31, 2026
Merged

kawhil-amd merged 3 commits into
ROCm:mainfrom
kawhil-amd:fix/epv2-clamp-untuned-combine-blocks

Conversation

@kawhil-amd

@kawhil-amd kawhil-amd commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Fixes #794. On a device with no tuned table, the flydsl EP v2 lookup() falls back to _cu_scaled_default(), which sets combine_block_num to the CU count. The combine kernel keeps one xdb flag per block in an array of xdb_flag_slots (256), so a part with more than 256 CUs (MI300X has 304) fails at op construction:

AssertionError: combine block_num 304 exceeds xdb_flag_slots 256: the per-block xdb flag array would overflow

#795 adds a tuned MI300X table, which fixes MI300X. It leaves the fallback unchanged, so any other untuned part with more than 256 CUs still hits the assert (#795 says the issue stays open for the clamp). This PR is that clamp. It does not depend on #795 and does not touch the same lines.

Technical Details

  • tuning_configs.py: _cu_scaled_default() now returns combine_block_num=min(cu, _MAX_COMBINE_BLOCKS), with _MAX_COMBINE_BLOCKS = 256. dispatch_block_num is unchanged: only combine has the slot limit.
    - 256 blocks is already enough to saturate P2P bandwidth, so the cap does not limit the peak the fallback can reach on a part with more CUs.
  • The flag array is not sized from the actual block_num, which [EPv2] flydsl backend asserts at op construction on MI300X: CU-scaled combine_block_num 304 exceeds xdb_flag_slots 256 #794 lists as the other option. Block 0 fills the unused tail [block_num, xdb_flag_slots) so that calls with different block_num (the per-token schedule) stay in step, and that needs a fixed size.
  • tuning_configs.py must not import intranode_kernels (it pulls in flydsl), so the limit is a second constant. A test checks that it equals intranode_kernels.xdb_flag_slots.
  • An explicit combine_block_num above 256 still fails the assert. That is correct: the user asked for something the kernel cannot do.

A clamped fallback is only a safe default, not a fast one. #795's table shows the CU fallback with combine capped at 256 is about a third slower than a tuned schedule at large token counts on MI300X.

Test Plan

New host-only test tests/python/ops/dispatch_combine_v2/test_tuning_configs.py (no GPU needed):

  • Untuned device with 80, 256 and 304 CUs: combine blocks are 80, 256 and 256; dispatch blocks stay at the CU count.
  • A known device whose default is None takes the same clamped path.
  • _MAX_COMBINE_BLOCKS == intranode_kernels.xdb_flag_slots.
pytest tests/python/ops/dispatch_combine_v2/test_tuning_configs.py   # 5 passed

Without the change, the 304-CU case and the other two tests that use the constant fail. pre-commit passes on the changed files.

Not covered: I have no MI300X here (the run was on MI355X), so I did not construct a real EpDispatchCombineOp on a 304-CU part. The test checks the geometry that lookup() returns, not the kernel assert itself.

🤖 Generated with Claude Code

…ag slots

On a device with no tuned table, lookup() falls back to _cu_scaled_default(),
which set combine_block_num to the CU count. The combine kernel keeps one xdb
flag per block in an array of xdb_flag_slots (256), so a part with more than 256
CUs (MI300X, 304) failed at op construction with "combine block_num 304 exceeds
xdb_flag_slots 256".

Clamp the fallback's combine_block_num to 256. The flag array is not resized to
the block count: block 0 fills its unused tail so that calls with different
block_num stay in step, which needs a fixed size. dispatch_block_num has no such
limit and is unchanged.

Host-only tests cover the clamp and check that the constant in tuning_configs
equals intranode_kernels.xdb_flag_slots.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kawhil-amd

Copy link
Copy Markdown
Contributor Author

cc @jhchouuu @jeffdaily plz help to review this fix for EPv2 flydsl backend.

@kawhil-amd kawhil-amd self-assigned this Oct 30, 2026

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Tested at 7f4ad5cb, merged onto current main:

  • Host. The new test fails on main (3 failed / 2 passed) and passes with this change (5 passed).
  • MI355X, forced onto the untuned path as a 304-CU part. I set _cu_count to 304 and _device_key to None, then ran bench_ep.py with the flydsl backend and the gather combine.
    • On main the op dies at construction with the #794 assert ("combine block_num 304 exceeds xdb_flag_slots 256").
    • With this change it builds and validates: eager and graph, 128 and 4096 tokens.
  • MI308X (gfx942, 80 CUs). Validates on its tuned table, and also when forced through the untuned fallback. Nothing changes there, since 80 < 256.

Two small things:

  • The license header in test_tuning_configs.py appears twice. The second copy is cut off after "ARISING FROM," and runs straight into the docstring.
  • Two comments now say the opposite of the new code. intranode_kernels.py:558 says "256 == the CU count (the max combine block_num), so block_num never exceeds it", and flydsl_backend.py:177 has "(== CU count, the max combine block_num)". Neither holds on a 304-CU part, which is exactly what this PR handles. Pointing them at _MAX_COMBINE_BLOCKS and the clamp would keep the next reader from undoing it.

kawhil-amd and others added 2 commits October 30, 2026 11:28
…ents

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kawhil-amd

Copy link
Copy Markdown
Contributor Author

LGTM. Tested at 7f4ad5cb, merged onto current main:LGTM。已在 7f4ad5c 版本测试,并已合并到当前主分支:

  • Host. The new test fails on main (3 failed / 2 passed) and passes with this change (5 passed).主机。新测试在主测试中失败(3 个失败/2 个通过),但通过此更改后(5 个通过)。
  • MI355X, forced onto the untuned path as a 304-CU part. I set _cu_count to 304 and _device_key to None, then ran bench_ep.py with the flydsl backend and the gather combine.
    MI355X 被强制设置为未调优路径,因为它是 304 CU 的芯片。我将 _cu_count 设置为 304,将 _device_key 设置为 None,然后使用 flydsl 后端和 gather combine 运行了 bench_ep.py。
  • MI308X (gfx942, 80 CUs). Validates on its tuned table, and also when forced through the untuned fallback. Nothing changes there, since 80 < 256.MI308X(gfx942,80 个计算单元)。在经过调优的表中进行验证,并在强制使用未经调优的回退表时也进行验证。由于 80 < 256,因此结果没有变化。

Two small things:  两件小事:

  • The license header in test_tuning_configs.py appears twice. The second copy is cut off after "ARISING FROM," and runs straight into the docstring.test_tuning_configs.py 文件中的许可证头出现了两次。第二次出现的许可证头在“ARISING FROM”之后被截断,直接进入了文档字符串。
  • Two comments now say the opposite of the new code. intranode_kernels.py:558 says "256 == the CU count (the max combine block_num), so block_num never exceeds it", and flydsl_backend.py:177 has "(== CU count, the max combine block_num)". Neither holds on a 304-CU part, which is exactly what this PR handles. Pointing them at _MAX_COMBINE_BLOCKS and the clamp would keep the next reader from undoing it.现在有两条评论与新代码的说法相反。intranode_kernels.py:558 写道:“256 == CU 数量(最大组合块数),所以块数永远不会超过它”,而 flydsl_backend.py:177 写道:“(== CU 数量,最大组合块数)”。这两条评论都不适用于 304 个 CU 的组件,而这正是此 PR 所要处理的。如果将它们指向 _MAX_COMBINE_BLOCKS 和 clamp 函数,就可以防止后续读者推翻此 PR。

Thanks for review comments, all fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[EPv2] flydsl backend asserts at op construction on MI300X: CU-scaled combine_block_num 304 exceeds xdb_flag_slots 256

2 participants