Skip to content

[Fix] attention: Move torch_native/flex_attention CUDA graph disable after platform fallback - #43358

Open
sunshinejiali wants to merge 1 commit into
sgl-project:mainfrom
sunshinejiali:fix/cuda-graph-fallback-order
Open

sunshinejiali wants to merge 1 commit into
sgl-project:mainfrom
sunshinejiali:fix/cuda-graph-fallback-order

Conversation

@sunshinejiali

@sunshinejiali sunshinejiali commented Oct 9, 2026 •

Copy link
Copy Markdown

Motivation

  • When the attention backend is auto-selected on a CPU platform without AMX support, the platform fallback logic (_attention_backend_platform_fallbacks) rewrites intel_amx to torch_native. However, the CUDA graph disable check for torch_native was placed before this fallback in handle_attention_backend_compatibility, so it saw the pre-fallback value (intel_amx) and skipped the disable logic.
  • This resulted in CUDA graphs remaining enabled for the auto-selected torch_native path, while explicitly-set torch_native correctly disabled them — an inconsistency that violates the expected behavior described in issue [Bug] CUDA graphs stay enabled for torch_native attention when the platform fallback selects it #43142.
  • This PR fixes the ordering so that the CUDA graph disable checks for torch_native and flex_attention run after the platform fallback, ensuring both auto-selected and explicitly-set backends behave identically.

Modifications

  1. pythonattention_hook.py
  • Moved the torch_native and flex_attention CUDA graph disable blocks from before _attention_backend_platform_fallbacks to after it, so the checks see the post-fallback backend value.
  1. test_server_args.py
  • Added TestAttentionBackendCudaGraph with 3 regression tests:

    • test_auto_fallback_and_explicit_torch_native_disable_cudagraph: Both auto fallback and explicit torch_native correctly disable CUDA graphs.
    • test_auto_and_explicit_yield_same_cudagraph_config: Both paths produce identical cuda_graph_config.
    • test_round_trip_stability: ServerArgs round-trip via asdict preserves cuda_graph_config.
  • Tests use _create_model_dir() to avoid the model_path="dummy" early return, and @override_platform(is_cuda=False, has_amx=False) to simulate CPU without AMX.

Accuracy Tests

N/A

Speed Tests and Profiling

N/A

Checklist


CI States

Latest PR Test (Base): ❌ Run #39401972727
Latest PR Test (Extra): ❌ Run #39401972633
Latest PR Test (AMD ROCm 10): ❌ Run #39401972844

@sunshinejiali
sunshinejiali force-pushed the fix/cuda-graph-fallback-order branch from aaa6fe0 to 1f71424 Compare October 9, 2026 10:46
@sunshinejiali

Copy link
Copy Markdown
Author

/tag-and-rerun-ci

@sunshinejiali
sunshinejiali force-pushed the fix/cuda-graph-fallback-order branch from 1f71424 to 476468a Compare October 20, 2026 09:36
@github-actions

This comment has been minimized.

@sunshinejiali sunshinejiali changed the title [Fix] move torch_native/flex_attention CUDA graph disable after platform fallback [Fix] attention: Move torch_native/flex_attention CUDA graph disable after platform fallback Oct 20, 2026
@sunshinejiali

Copy link
Copy Markdown
Author

/tag-and-rerun-ci

@sunshinejiali

Copy link
Copy Markdown
Author

Hi maintainers, could someone help assign an appropriate reviewer for this PR? It modifies attention_hook.py under arg_groups, which doesn't appear to have a matching CODEOWNERS rule. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant