Skip to content

[GPU] Micro sdpa #24656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
p-durandin merged 7 commits into from
Jul 1, 2024
Merged

[GPU] Micro sdpa #24656

p-durandin merged 7 commits into from
Jul 1, 2024

Conversation

@vladimir-paramuzov
Copy link

@vladimir-paramuzov vladimir-paramuzov commented May 23, 2024 •
edited
Loading

Details:

  • Added SDPA impl based on microkernels using internal onednn API and related infra
  • Current limitations:
    • fused transpose shouldn't change order of innermost dim (head size).
    • is_causal = true is not supported
    • fp16 only
    • num heads dimension must be static
    • no indirect kv support
  • Initial version of KV Cache + SDPA func test
  • Enabled Transpose+SDPA fusion for static shape too

Tickets:

@vladimir-paramuzov vladimir-paramuzov added this to the 2024.3 milestone May 23, 2024
@github-actions github-actions bot added the category: build OpenVINO cmake script / infra label May 23, 2024
@github-actions github-actions bot added the category: IE Tests OpenVINO Test: plugins and common label Jun 4, 2024
@vladimir-paramuzov vladimir-paramuzov force-pushed the micro_sdpa branch 5 times, most recently from 5dbb9c7 to d149dd4 Compare June 13, 2024 13:03
@github-actions github-actions bot removed the category: IE Tests OpenVINO Test: plugins and common label Jun 13, 2024
@vladimir-paramuzov vladimir-paramuzov force-pushed the micro_sdpa branch 2 times, most recently from 5cc640c to 6c596a9 Compare June 14, 2024 11:06
@vladimir-paramuzov vladimir-paramuzov marked this pull request as ready for review June 14, 2024 11:08
@vladimir-paramuzov vladimir-paramuzov requested review from a team as code owners June 14, 2024 11:08
@vladimir-paramuzov vladimir-paramuzov changed the title [WIP][GPU] Micro sdpa draft [GPU] Micro sdpa draft Jun 14, 2024
@vladimir-paramuzov vladimir-paramuzov changed the title [GPU] Micro sdpa draft [GPU] Micro sdpa Jun 14, 2024
@vladimir-paramuzov vladimir-paramuzov force-pushed the micro_sdpa branch 3 times, most recently from 979a0d4 to 9525423 Compare June 29, 2024 11:50
sshlyapn
sshlyapn reviewed Jul 1, 2024
View reviewed changes
src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp Outdated
return false;

// For platforms with DPAS support we don't have any shape-based limitations
if (device_info.supports_immad && cldnn::query_microkernels_supported(m_context->get_engine(), config))
Copy link
Contributor

@sshlyapn sshlyapn Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to kernel selector, micro SDPA kernel supports only f16 data type and bfyx 4D format - should we relax these limitations for kernel, or restrict such cases here? Also, dynamic num_heads dimension is not supported and then sdpa_ref kernel will be used

Copy link
Author

@vladimir-paramuzov vladimir-paramuzov Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjusted callback to decompose those cases

sshlyapn
sshlyapn approved these changes Jul 1, 2024
View reviewed changes
Copy link
Contributor

@sshlyapn sshlyapn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

src/plugins/intel_gpu/src/kernel_selector/kernels/sdpa/sdpa_kernel_micro.h

#ifdef ENABLE_ONEDNN_FOR_GPU

#pragma once
Copy link
Contributor

@sshlyapn sshlyapn Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could be moved to the top of the file probably

@p-durandin p-durandin enabled auto-merge July 1, 2024 13:55
@p-durandin p-durandin added this pull request to the merge queue Jul 1, 2024
Merged via the queue into openvinotoolkit:master with commit 2918322 Jul 1, 2024
@vladimir-paramuzov vladimir-paramuzov deleted the micro_sdpa branch July 3, 2024 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@sshlyapn sshlyapn sshlyapn approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

category: build OpenVINO cmake script / infra category: GPU OpenVINO GPU plugin under_perf_check

Projects

None yet

Milestone

2024.3

Development

Successfully merging this pull request may close these issues.

3 participants

@vladimir-paramuzov @sshlyapn @p-durandin