Skip to content

Enable FP8/MXFP8 Ops with requests and CUDA alignment #2207

New issue
New issue
Open
Task
Open
Enable FP8/MXFP8 Ops with requests and CUDA alignment#2207
Task
Assignees
StonepiaCuiYifengLuFinchxiaowangintel
Labels
agent:needs-humanAgent lifecycle: real bug the agent could not fix on its owndtype: float8fp8 dtype (e4m3/e5m2)dtype: mxfp8MX (microscaling) fp8 (e4m3/e5m2)module: infraOnly for CICD infra used, no action needtest: utUnit-test failure surface
Milestone
PT2.16

Description

@CuiYifeng
CuiYifeng
opened on Oct 23, 2025

Summary

This is a tracking issue to enable FP8 and MXFP8 operations on XPU, covering memory ops (fill, flip, index_put, gather, cat, eq, ne, where, empty/zeros/ones, to, copy, clone, add/sub/mul/div, compare, normal) and GEMM ops (_scaled_mm and _scaled_grouped_mm) across multiple FP8 variants (e4m3fn, e4m3fnuz, e5m2, e5m2fnuz, e8m0fnu). The issue tracks support status with green/yellow/red indicators and links to related PRs, targeting PyTorch releases from 2.10 through 2.16.

Type

  • Category: feature-gap
  • Platform: xpu
  • Related Components: FP8/MXFP8 kernels, memory ops (fill, flip, index_put, index_select, gather, cat, eq, ne, where, empty/zeros/ones, to, copy, clone, add/sub/mul/div, compare, normal), GEMM ops (_scaled_mm, _scaled_grouped_mm)

Objective

Enable FP8/MXFP8 support for a set of memory and GEMM operations across all FP8 dtype variants (e4m3fn, e4m3fnuz, e5m2, e5m2fnuz, e8m0fnu), with CUDA alignment for scaled matrix multiply ops.

Current Status

Memory Op e4m3fn e4m3fnuz e5m2 e5m2fnuz e8m0fnu PR Link
fill/fill_ 🟢 🟢 🟢 🟢 🟢
flip/fliplr/flipud 🟢 🟢 🟢 🟢 🟢 #2190
index_put/index_put_ 🟢 🟢 🟢 🟢 ❌ #2190
index.Tensor/index.Tensor_out 🟢 🟢 🟢 🟢 🟢 #2190
index_select/index_select.out 🟢 🟢 🟢 🟢 🟢
gather/gather.out 🟢 🟢 🟢 🟢 🟢
cat/cat.out 🟢 🟢 🟢 🟢 🟢 #2152
eq/eq_ 🟢 🟢 🟢 🟢 🟢 #2152
ne/ne_ 🟢 🟢 🟢 🟢 🟢
where 🟢 🟢 🟢 🟢 🟢 #2152
empty/zeros/ones 🟢 🟢 🟢 🟢 🟢
to 🟢 🟢 🟢 🟢 🟢
copy 🟢 🟢 🟢 🟢 🟢 #2258
clone 🟢 🟢 🟢 🟢 🟢
add/sub/mul/div 🟡 🟡 🟡 🟡 🟡 #2145
compare 🟡 🟡 🟡 🟡 🟡 #2154
normal 🟡 🟡 🟡 🟡 🟡
GEMM OP Activation&Weight Scale Modeling Scale data type Scale layout Status PR Link Targets
_scaled_mm FP8(E4M3/E5M2) Tensorwise scaling FP32 Scalar 🟢 pt#165978 2.10
FP8(E4M3/E5M2) Channelwise scaling FP32 Vector 🟢 pt#165978 2.10
FP8(E4M3/E5M2) 128-element 1D/128x128-element 2D block scaling FP32 Tensor 🟢 pt#173630 2.12
MxFP8(E4M3/E5M2) 32-element 1D block scaling UE8M0 Tiled Tensor 🟢 pt#181726 2.14
_scaled_grouped_mm FP8(E4M3/E5M2) Tensorwise scaling FP32 Scalar 🟡 2.16
FP8(E4M3/E5M2) Channelwise scaling FP32 Vector 🟡 2.16
FP8(E4M3/E5M2) 128-element 1D/128x128-element 2D block scaling FP32 Tensor 🟡 2.16
MxFP8(E4M3/E5M2) 32-element 1D block scaling UE8M0 Tiled Tensor 🟡 2.16

Context

Labels: dtype: float8. Related PRs: #2190, #2152, #2145, #2154, #2258, pytorch/pytorch#165978, pytorch/pytorch#173630. PyTorch release targets range from 2.10 to 2.16.

Root Cause Analysis

This is a tracking issue (not a bug report) for incrementally enabling FP8 and MXFP8 operations on XPU. Several memory ops (add/sub/mul/div, compare, normal) and GEMM ops (_scaled_mm with block scaling and MXFP8, _scaled_grouped_mm) are partially implemented (🟡) or missing. The gaps are in kernel implementations within torch-xpu-ops for arithmetic/comparison ops on FP8 dtypes, and GEMM ops for advanced scaling modes.

Proposed Fix Strategy

This is a multi-PR tracking issue. Individual sub-tasks are tracked via linked PRs (e.g., #2145 for add/sub/mul/div, #2154 for compare). Each unfinished item requires separate kernel implementations in torch-xpu-ops. No single agent action closes this issue — it is an umbrella tracker that must be resolved incrementally as sub-PRs land.

Action Items

  • 🔍 Issue formatted (Discovery Agent)

    Discovery log

    [2026-05-10 23:05:41]
    Summary: This is a tracking issue to enable FP8 and MXFP8 operations on XPU, covering memory ops (fill, flip, index_put, gather, cat, eq, ne, where, empty/zeros/ones, to, copy, clone, add/sub/mul/div, compare, normal) and GEMM ops (_scaled_mm and _scaled_grouped_mm) across multiple FP8 variants (e4m3fn, e4m3fnuz, e5m2, e5m2fnuz, e8m0fnu). The issue tracks support status with green/yellow/red indicators and links to related PRs, targeting PyTorch releases from 2.10 through 2.16.
    Failed tests: N/A
    Dependency: N/A
    Commit scope: N/A

Original Issue

Original issue body

🚀 The feature, motivation and pitch

Plan to enable the following ops for FP8/MXFP8:
🟢 Supported 🟡 TBD ❌Rejected

Memory Op e4m3fn e4m3fnuz e5m2 e5m2fnuz e8m0fnu PR Link
fill/fill_ 🟢 🟢 🟢 🟢 🟢
flip/fliplr/flipud 🟢 🟢 🟢 🟢 🟢 #2190
index_put/index_put_ 🟢 🟢 🟢 🟢 ❌ #2190
index.Tensor/index.Tensor_out 🟢 🟢 🟢 🟢 🟢 #2190
index_select/index_select.out 🟢 🟢 🟢 🟢 🟢
gather/gather.out 🟢 🟢 🟢 🟢 🟢
cat/cat.out 🟢 🟢 🟢 🟢 🟢 #2152
eq/eq_ 🟢 🟢 🟢 🟢 🟢 #2152
ne/ne_ 🟢 🟢 🟢 🟢 🟢
where 🟢 🟢 🟢 🟢 🟢 #2152
empty/zeros/ones 🟢 🟢 🟢 🟢 🟢
to 🟢 🟢 🟢 🟢 🟢
copy 🟢 🟢 🟢 🟢 🟢 #2258
clone 🟢 🟢 🟢 🟢 🟢
add/sub/mul/div 🟡 🟡 🟡 🟡 🟡 #2145
compare 🟡 🟡 🟡 🟡 🟡 #2154
normal 🟡 🟡 🟡 🟡 🟡
GEMM OP Activation&Weight Scale Modeling Scale data type Scale layout Status PR Link Targets
_scaled_mm FP8(E4M3/E5M2) Tensorwise scaling FP32 Scalar 🟢 pt#165978 2.10
FP8(E4M3/E5M2) Channelwise scaling FP32 Vector 🟢 pt#165978 2.10
FP8(E4M3/E5M2) 128-element 1D/128x128-element 2D block scaling FP32 Tensor 🟡 pt#173630 2.12
MxFP8(E4M3/E5M2) 32-element 1D block scaling UE8M0 Tiled Tensor 🟡 2.14
_scaled_grouped_mm FP8(E4M3/E5M2) Tensorwise scaling FP32 Scalar 🟡 2.16
FP8(E4M3/E5M2) Channelwise scaling FP32 Vector 🟡 2.16
FP8(E4M3/E5M2) 128-element 1D/128x128-element 2D block scaling FP32 Tensor 🟡 2.16
MxFP8(E4M3/E5M2) 32-element 1D block scaling UE8M0 Tiled Tensor 🟡 2.16

Alternatives

No response

Additional context

No response

triage log **Verdict:** NEEDS_HUMAN **Reason:** Umbrella tracking issue spanning multiple in-progress PRs and kernel implementations; not a single fixable bug.

Root Cause: This is a tracking issue (not a bug report) for incrementally enabling FP8 and MXFP8 operations on XPU. Several memory ops (add/sub/mul/div, compare, normal) and GEMM ops (_scaled_mm with block scaling and MXFP8, _scaled_grouped_mm) are partially implemented (🟡) or missing. The gaps are in kernel implementations within torch-xpu-ops for arithmetic/comparison ops on FP8 dtypes, and GEMM ops for advanced scaling modes.

Fix Strategy: This is a multi-PR tracking issue. Individual sub-tasks are tracked via linked PRs (e.g., #2145 for add/sub/mul/div, #2154 for compare). Each unfinished item requires separate kernel implementations in torch-xpu-ops. No single agent action closes this issue — it is an umbrella tracker that must be resolved incrementally as sub-PRs land.

Tokens: model: claude-sonnet-4 | tokens: 37.5K | in: 3 | out: 373 | cache_read: 34.4K | cache_write: 2.7K | cost: $0.0262
Log: agent-issue-2207-triaging-20260511-212839.log

Activity

  1. CuiYifeng
    assigned
    CuiYifeng
    and
    liangan1
    on Oct 23, 2025

  2. liangan1 commented on Oct 23, 2025

    @liangan1
    liangan1
    on Oct 23, 2025
    Contributor

    @CuiYifeng, except for the element data type with e5m2 and e4m3, PT also defined limited ops for UE8M0 for CUDA. we also need to enable this part. An example is when wen print the u8m0 scale, the index will be used.

  3. jikunshang commented on Oct 23, 2025

    @jikunshang
    jikunshang
    on Oct 23, 2025

    can you also add #2017 #2013 to this list?

  4. CuiYifeng commented on Oct 23, 2025

    @CuiYifeng
    CuiYifeng
    on Oct 23, 2025
    ContributorAuthor

    can you also add #2017 #2013 to this list?

    @jikunshang These ops are already included.
    #2013 will be solved in #2152.
    #2017 will be solved in #2190.

  5. chuanqi129
    added this to the PT2.10 milestone on Oct 29, 2025
  6. chuanqi129
    added the
    Task
    issue type on Oct 29, 2025

  7. carsonwang commented on Oct 30, 2025

    @carsonwang
    carsonwang
    on Oct 30, 2025
    Contributor

    Hi, the following are a few requests for implementing scaled_mm MXFP8/MXFP4/NVFP4+ support. The immediate request is to support copy for Float8_e8m0fnu and Float4_e2m1fn_x2.

    • copy support for Float8_e8m0fnu
    • For MXFP4, maybe open a separate ticket to track. First priority is to support copy for Float4_e2m1fn_x2
    • Also need to track the support for NVFP4+, scale will be E5M3

  8. CuiYifeng commented on Oct 30, 2025

    @CuiYifeng
    CuiYifeng
    on Oct 30, 2025
    ContributorAuthor

    @carsonwang Thanks for the request. Will update FP8 ticket and open a new ticket for NVFP4 support.

  9. github-merge-queue
    added 2 commits that reference this issue on Nov 3, 2025

    Enable FP8 concat_xpu and where_xpu (#2152)

    a3efbb3

    Support FP8 in op flip, index_put, and index.Tensor (#2190)

    9aac5a1
  10. RUIJIEZHONG66166
    added 2 commits that reference this issue on Nov 4, 2025

    Enable FP8 concat_xpu and where_xpu (#2152)

    7a25487

    Support FP8 in op flip, index_put, and index.Tensor (#2190)

    9e636f9
  11. github-merge-queue
    added a commit that references this issue on Nov 5, 2025

    Add Float8_e8m0fnu support for copy (#2258)

    58b063e
  12. wincent8
    added a commit that references this issue on Nov 5, 2025

    Enable FP8 concat_xpu and where_xpu (#2152)

    424196a

  13. 31 remaining items

  14. Stonepia
    added
    agent:needs-humanAgent lifecycle: real bug the agent could not fix on its own
    on May 12, 2026
  15. daisyden
    modified the milestones: PT2.13, PT2.14 on May 13, 2026
  16. laifenxiawucha
    unassigned
    laifenxiawucha
    and
    Copilot
    on Jul 28, 2026
  17. CuiYifeng
    modified the milestones: PT2.14, PT2.15 on Jul 29, 2026
  18. Stonepia
    assigned
    xiaowangintel
    on Sep 16, 2026

  19. Stonepia commented on Sep 16, 2026

    @Stonepia
    Stonepia
    on Sep 16, 2026
    Contributor

    Move to 2.16 because scaled_grouped_mm is not target for 2.15

  20. Stonepia
    modified the milestones: PT2.15, PT2.16 on Sep 16, 2026
  21. daisyden
    added
    module: infraOnly for CICD infra used, no action need
    dtype: mxfp8MX (microscaling) fp8 (e4m3/e5m2)
    test: utUnit-test failure surface
    on Nov 11, 2026

  22. daisyden commented on Nov 11, 2026

    @daisyden
    daisyden
    on Nov 11, 2026
    Contributor
    label-issue: #2207

    Root cause: no defect; umbrella tracker to incrementally enable FP8/MXFP8 memory + GEMM ops (add/sub/mul/div #2145, compare #2154, _scaled_grouped_mm) in torch-xpu-ops across PT2.10-2.16.

    axis value reason
    type Task Native Type field set to Task (human); umbrella FP8/MXFP8 enabling tracker.
    test test: ut No e2e/oob/benchmark signal; default UT surface.
    module module: infra Multi-op umbrella tracking issue with no single owning op; catch-all.
    priority Low Feature-gap/enablement request (feature) with no failure or crash.
    dtype dtype: float8 Human label dtype: float8; issue targets FP8 variants e4m3fn/e5m2/e8m0fnu.
    dtype dtype: mxfp8 GEMM table tracks MxFP8(E4M3/E5M2) 32-element block scaling for _scaled_mm/_scaled_grouped_mm.

    Applied to this issue. Generated by a local label-issue run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent:needs-humanAgent lifecycle: real bug the agent could not fix on its owndtype: float8fp8 dtype (e4m3/e5m2)dtype: mxfp8MX (microscaling) fp8 (e4m3/e5m2)module: infraOnly for CICD infra used, no action needtest: utUnit-test failure surface

Type

Task

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions