Skip to content

Update AArch64 toolchains to GCC15 - #182016

Closed
Anndrey24 wants to merge 2 commits into
pytorch:mainfrom
Anndrey24:gcc15-enable
Closed

Anndrey24 wants to merge 2 commits into
pytorch:mainfrom
Anndrey24:gcc15-enable

Conversation

@Anndrey24

@Anndrey24 Anndrey24 commented Apr 30, 2026 •

Copy link
Copy Markdown
Contributor

@pytorch-bot

pytorch-bot Bot commented Apr 30, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/182016

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures, 10 Unclassified Failures

As of commit 7a7fe4c with merge base e9ff322 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot Bot added the release notes: releng release notes category label Apr 30, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Apr 30, 2026 •

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: aditew01 / name: Aditya Tewari (0db4157)
  • ✅ login: Anndrey24 / name: Andrei Hutu (0db4157, 12bb589)

@aditew01 aditew01 added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 30, 2026

@Skylion007 Skylion007 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh will this fix the LTO issues I'm having I wonder...

@Skylion007

Copy link
Copy Markdown
Collaborator

Okay, this is why I hit a bunch of bugs when enabling IPO only on AARCH: #180186

}

std::shared_ptr<RpcAgent> RpcAgent::currentRpcAgent_ = nullptr;
std::atomic<std::shared_ptr<RpcAgent>> RpcAgent::currentRpcAgent_{nullptr};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A shared_ptr is implicitly atomic in ref count. Do you have the order swapped and mean a shared_ptr to
An atomic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong, but my understanding is that the distinction here is between the std::shared_ptr control block and the std::shared_ptr object itself. The refcount operations are thread-safe, as you said, but concurrent access/modification of the same std::shared_ptr object still needs synchronisation, which is why the old code used:

std::shared_ptr<RpcAgent> RpcAgent::currentRpcAgent_ = nullptr;
...
std::atomic_load(&currentRpcAgent_)
std::atomic_compare_exchange_strong(&currentRpcAgent_, ...)
std::atomic_exchange(&currentRpcAgent_, ...)

GCC 15 now deprecates those APIs and suggests using std::atomic<std::shared_ptr<T>> instead:

/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:259:26: error: 'std::shared_ptr<_Tp> std::atomic_load(const shared_ptr<_Tp>*) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:263:53: error: 'std::shared_ptr<_Tp> std::atomic_load(const shared_ptr<_Tp>*) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:277:40: error: 'bool std::atomic_compare_exchange_strong(shared_ptr<_Tp>*, shared_ptr<_Tp>*, shared_ptr<_Tp>) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:286:29: error: 'std::shared_ptr<_Tp> std::atomic_exchange(shared_ptr<_Tp>*, shared_ptr<_Tp>) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
cc1plus: all warnings being treated as errors

So I don’t think we want std::shared_ptr<std::atomic<...>> here, since the thing being atomically updated is the std::shared_ptr handle, not the RpcAgent object it points to.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why this ifdef is needed, let's just deprecate all API and use modern one (but as part of separate PR)

TARGET=final
GPU_IMAGE=arm64v8/almalinux:8
DOCKER_GPU_BUILD_ARG=" --build-arg DEVTOOLSET_VERSION=13"
DOCKER_GPU_BUILD_ARG=" --build-arg DEVTOOLSET_VERSION=15"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so this is why is hit a bunch of bugs only AARCH when enabling LTO!

@pytorch-bot pytorch-bot Bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label May 7, 2026
@pytorch-bot

pytorch-bot Bot commented May 7, 2026 •

Copy link
Copy Markdown

Workflows were awaiting approval. CI has now been triggered for the ciflow labels on this PR.

@Anndrey24

Copy link
Copy Markdown
Contributor Author

@aditew01 could you please rerun ciflow/trunk for this new commit? Thanks!

@pytorch-bot pytorch-bot Bot added the no-runner-experiments Bypass Meta/LF runner determinator label May 13, 2026
@aditew01 aditew01 added the ci-no-td Do not run TD on this PR label May 13, 2026
@Anndrey24

Anndrey24 commented May 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Below is a summary of the GCC 15 AArch64 issues encountered in this PR, along with their corresponding fixes:

Compilation/build errors

1. HeaderOnlyArrayRef empty vector construction

GCC 15 reports -Werror=free-nonheap-object from inlined HeaderOnlyArrayRef<T>::vec() while constructing an std::vector<T> from Data to Data + Length. For empty refs, Data may be null, and forming Data + Length still performs pointer arithmetic on that null pointer. The fix returns {} immediately for empty refs to avoid that.

Failure message:

    inlined from 'std::vector<T> c10::HeaderOnlyArrayRef<T>::vec() const [with T = long int]' at /pytorch/torch/headeronly/util/HeaderOnlyArrayRef.h:237:64,
    inlined from 'at::TensorIterator at::native::{anonymous}::_make_unfold_backward_iter_over_grad_out(at::Tensor&, const at::Tensor&, int64_t, int64_t, int64_t)' at /pytorch/aten/src/ATen/native/UnfoldBackward.h:52:65,
    inlined from 'void at::native::{anonymous}::unfold_backward_cpu_kernel(at::Tensor&, const at::Tensor&, int64_t, int64_t, int64_t)' at /pytorch/aten/src/ATen/native/cpu/UnfoldBackwardKernel.cpp:129:41:
/usr/include/c++/15/bits/new_allocator.h:172:66: error: 'void operator delete(void*, std::size_t)' called on pointer '<unknown>' with nonzero offset [8, 9223372036854775800] [-Werror=free-nonheap-object]
  172 |         _GLIBCXX_OPERATOR_DELETE(_GLIBCXX_SIZED_DEALLOC(__p, __n));
      |                                                                  ^
cc1plus: some warnings being treated as errors

Files changed:

  • torch/headeronly/util/HeaderOnlyArrayRef.h

2. RPC current agent atomic shared_ptr operations

GCC 15/libstdc++ deprecates the free std::atomic_* overloads for std::shared_ptr. The fix uses std::atomic<std::shared_ptr<RpcAgent>> and member operations when __cpp_lib_atomic_shared_ptr is available, while also preserving the old free-function path for older libstdc++ versions that do not provide the C++20 atomic shared pointer specialisation.

Failure message:

/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:259:26: error: 'std::shared_ptr<_Tp> std::atomic_load(const shared_ptr<_Tp>*) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
  259 |   return std::atomic_load(&currentRpcAgent_) != nullptr;
      |          ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:263:53: error: 'std::shared_ptr<_Tp> std::atomic_load(const shared_ptr<_Tp>*) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
  263 |   std::shared_ptr<RpcAgent> agent = std::atomic_load(&currentRpcAgent_);
      |                                     ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:277:40: error: 'bool std::atomic_compare_exchange_strong(shared_ptr<_Tp>*, shared_ptr<_Tp>*, shared_ptr<_Tp>) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
  277 |     std::atomic_compare_exchange_strong(
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/pytorch/torch/csrc/distributed/rpc/rpc_agent.cpp:286:29: error: 'std::shared_ptr<_Tp> std::atomic_exchange(shared_ptr<_Tp>*, shared_ptr<_Tp>) [with _Tp = torch::distributed::rpc::RpcAgent]' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Werror=deprecated-declarations]
  286 |         std::atomic_exchange(&currentRpcAgent_, std::move(rpcAgent));
      |         ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: some warnings being treated as errors

Files changed:

  • torch/csrc/distributed/rpc/rpc_agent.cpp
  • torch/csrc/distributed/rpc/rpc_agent.h

3. OperatorName namespace string insertion overlap

OperatorName::setNamespaceIfNotSet() used std::string::replace() to copy from name back into an overlapping region of the same name buffer. GCC 15 inlines that path down to memcpy and reports -Werror=restrict. The fix uses std::char_traits<char>::move() for the overlapping move and std::char_traits<char>::copy() for the non-overlapping namespace prefix.

Failure message:

    inlined from 'constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::replace(size_type, size_type, const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, size_type, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]' at /usr/include/c++/15/bits/basic_string.h:2434:29,
    inlined from 'bool c10::OperatorName::setNamespaceIfNotSet(const char*)' at /pytorch/aten/src/ATen/core/operator_name.h:44:19,
    inlined from 'bool c10::OperatorName::setNamespaceIfNotSet(const char*)' at /pytorch/aten/src/ATen/core/operator_name.h:38:8,
    inlined from 'virtual void OperatorNameTest_SetNamespaceIfNotSetWithExistingNamespace_Test::TestBody()' at /pytorch/aten/src/ATen/test/operator_name_test.cpp:17:58:
/usr/include/c++/15/bits/char_traits.h:429:56: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' accessing 28 bytes at offsets 11 and 0 overlaps 17 bytes at offset 11 [-Werror=restrict]
  429 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
      |                                        ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
cc1plus: some warnings being treated as errors

Files changed:

  • aten/src/ATen/core/operator_name.h

Test failures

4. SVE BF16 transpose strict aliasing

The previous transpose_mxn<BFloat16>() used BFloat16[] scratch tiles but accessed the same storage through uint16_t* SVE loads/stores. That mixed incompatible access types, so under strict aliasing GCC could treat the stores as unrelated and keep stale tile values. The fix uses bfloat16_t[] scratch tiles with svld1_bf16/svst1_bf16 internally, leaving casts only at the external BFloat16* boundary.

Failure message:

======================================================================
FAIL: test_transpose_copy (__main__.CPUReproTests)
----------------------------------------------------------------------
...
AssertionError: Tensor-likes are not close!

Mismatched elements: 8 / 289 (2.8%)
Greatest absolute difference: 2.0625 at index (16, 15) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (16, 8) (up to 0.016 allowed)

Files changed:

  • aten/src/ATen/cpu/vec/sve/vec_bfloat16.h

5. SVE PCH internal compiler errors

GCC reports an internal compiler error while compiling AArch64 SVE vector helpers through the Inductor precompiled-header path, matching GCC PR target/123457.

The affected code is valid, but GCC 14/15 can ICE when those SVE intrinsics are compiled with TORCH_INDUCTOR_PRECOMPILE_HEADERS enabled. The workaround adds narrow __attribute__((optimize("O0"))) guards around the specific helper functions that trigger the compiler bug for the affected GCC versions.

Representative failure:

torch._inductor.exc.InductorError: CppCompileError: C++ compile error

Output:
during GIMPLE pass: ccp
... In member function 'at::vec::CPU_CAPABILITY::Vectorized<float> at::vec::CPU_CAPABILITY::Vectorized<float>::tanh() const':
... internal compiler error: Segmentation fault

Affected tests:

  • test/inductor/test_compile_subprocess.py::CpuTests::test__dyn_quant_matmul_4bit_bf16_input_cpu
  • test/inductor/test_compile_subprocess.py::CpuTests::test__dyn_quant_matmul_4bit_fp32_input_cpu
  • test/inductor/test_cpu_repro.py::CPUReproTests::test_issue122380
  • test/inductor/test_cpu_repro.py::CPUReproTests::test_linear_buffer_reuse
  • test/inductor/test_cpu_select_algorithm.py::TestSelectAlgorithmCPU::test_bmm_with_y_storage_offset_bs_1_Mdim_192_Kdim_196_Ndim_385_cpu_bfloat16
  • test/inductor/test_torchinductor.py::CpuTests::test__dyn_quant_matmul_4bit_bf16_input_cpu
  • test/inductor/test_torchinductor.py::CpuTests::test__dyn_quant_matmul_4bit_fp32_input_cpu
  • test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test__dyn_quant_matmul_4bit_bf16_input_dynamic_shapes_cpu
  • test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test__dyn_quant_pack_4bit_weight_fp32_dynamic_shapes_cpu
  • test/inductor/test_torchinductor_dynamic_shapes.py::DynamicShapesCpuTests::test_float32_to_int32_dynamic_shapes_cpu

Files changed:

  • aten/src/ATen/cpu/vec/sve/vec_bfloat16.h
  • aten/src/ATen/cpu/vec/sve/vec_float.h
  • aten/src/ATen/cpu/vec/sve/vec_int.h

6. test_upsampling_bfloat16 stale xfail

The test now passes on this configuration, so the existing expected-failure marker turns the pass into an unexpected-success failure.

unexpected success: test_upsampling_bfloat16

Potential fix: #182034

@Anndrey24 Anndrey24 changed the title [WIP] Enable GCC15 on AArch64 Update AArch64 toolchains to GCC15 May 26, 2026
@Anndrey24
Anndrey24 marked this pull request as ready for review May 26, 2026 10:43
@Anndrey24
Anndrey24 requested review from a team and janeyx99 as code owners May 26, 2026 10:43
@aditew01
aditew01 requested review from Skylion007 and malfet May 26, 2026 10:45
@aditew01

Copy link
Copy Markdown
Collaborator

@claude review

@claude

claude Bot commented May 26, 2026 •

Copy link
Copy Markdown

Claude finished @aditew01's task in 2m 36s —— View job


Code Review: PR #182016 — Update AArch64 toolchains to GCC15

  • Gather context and read changed files
  • Review CI/Docker toolchain changes
  • Review HeaderOnlyArrayRef.h fix (empty vector construction)
  • Review operator_name.h fix (overlapping memcpy)
  • Review rpc_agent.cpp/h fix (atomic shared_ptr)
  • Review SVE vector files (strict aliasing + PCH ICE workaround)
  • Post final review

Overall Assessment

This is a well-structured PR that upgrades the AArch64 CI from GCC 13 to GCC 15 and fixes the resulting compilation issues. The fixes are correct and appropriately scoped. Below are detailed notes per change.


1. CI/Docker Changes (.ci/docker/, .github/workflows/)

Verdict: LGTM

Straightforward version bumps: GCC_VERSION=13 → 15, GCCTOOLSET_VERSION=13 → 15, and the corresponding Docker image name renames in workflow files. No issues here.

One minor note: the comment in Dockerfile_cuda_aarch64 still reads # Cuda ARM build needs gcc 11 (line 3) which is now stale — it should probably say GCC 15 (or just be removed). This is cosmetic and pre-existing.


2. HeaderOnlyArrayRef.h — Empty vector early return

File: torch/headeronly/util/HeaderOnlyArrayRef.h:236-241

std::vector<T> vec() const {
  if (this->empty()) {
    return {};
  }
  return std::vector<T>(this->Data, this->Data + this->Length);
}

Verdict: LGTM

Clean fix. When empty() is true, Data may be null, and Data + Length (even with Length=0) performs pointer arithmetic on a null pointer which is UB. The early return avoids this entirely without any performance cost.


3. operator_name.h — Overlapping memcpy fix

File: aten/src/ATen/core/operator_name.h:38-54

bool setNamespaceIfNotSet(const char* ns) {
  if (!getNamespace().has_value()) {
    const auto ns_len = strlen(ns);
    const auto old_name_size = name.size();
    name.resize(ns_len + 2 + old_name_size);
    auto name_data = name.data();
    std::char_traits<char>::move(
        name_data + ns_len + 2, name_data, old_name_size);
    std::char_traits<char>::copy(name_data, ns, ns_len);
    name[ns_len] = ':';
    name[ns_len + 1] = ':';
    return true;
  } else {
    return false;
  }
}

Verdict: LGTM

The old code used std::string::replace() which internally uses memcpy, but the source and destination regions overlap within the same buffer. GCC 15 inlines this and catches the -Werror=restrict violation.

The fix correctly uses:

  • char_traits::move for the overlapping shift of old content rightward
  • char_traits::copy for the non-overlapping namespace prefix write

The order of operations is correct: resize first (to ensure capacity), then move existing content right, then write the namespace prefix and :: separator.


4. rpc_agent.cpp/h — std::atomic<std::shared_ptr<T>>

Files: torch/csrc/distributed/rpc/rpc_agent.h:13-16, 282-286 and torch/csrc/distributed/rpc/rpc_agent.cpp:256-309

Verdict: LGTM

GCC 15 deprecates the free-standing std::atomic_load/atomic_exchange/atomic_compare_exchange_strong overloads for shared_ptr. The fix uses std::atomic<std::shared_ptr<RpcAgent>> behind a feature-test macro:

#if defined(__cpp_lib_atomic_shared_ptr) && \
    __cpp_lib_atomic_shared_ptr >= 201711L
#define TORCH_RPC_HAS_ATOMIC_SHARED_PTR 1
#endif

This is the correct approach — it uses the C++20 std::atomic<std::shared_ptr<T>> specialization when available (which provides .load(), .exchange(), .compare_exchange_strong() as member functions), and falls back to the deprecated free-standing overloads for older toolchains. The #ifdef branches are consistent and cover all call sites.

As @Anndrey24 correctly explained in the review comments, this wraps the shared_ptr handle in an atomic (for thread-safe concurrent modification of the pointer itself), not the pointed-to object.


5. SVE BFloat16 transpose — Strict aliasing fix

File: aten/src/ATen/cpu/vec/sve/vec_bfloat16.h:600-676

Verdict: LGTM

The previous implementation used BFloat16[] scratch tiles but accessed them through uint16_t* SVE loads/stores, violating strict aliasing. The new implementation uses bfloat16_t[] tiles with svld1_bf16/svst1_bf16, which is type-correct. Casts to/from BFloat16* happen only at the boundary (source/destination pointers), where reinterpret_cast is appropriate since BFloat16 and bfloat16_t have the same underlying representation.

The tiling approach (8x8 blocks with scalar transpose) is straightforward and correct for the small tile sizes involved.


6. GCC 14/15 SVE PCH ICE Workaround

Files: vec_float.h:461-469, vec_bfloat16.h:245-253, vec_int.h:20-31

Verdict: LGTM with one minor suggestion

The workaround applies __attribute__((optimize("O0"))) to specific functions that trigger GCC ICEs when compiled with precompiled headers (TORCH_INDUCTOR_PRECOMPILE_HEADERS). The guard conditions are:

#if defined(TORCH_INDUCTOR_PRECOMPILE_HEADERS) && defined(__GNUC__) && \
    !defined(__clang__) &&                                             \
    ((__GNUC__ == 14 && __GNUC_MINOR__ < 4) ||                         \
     (__GNUC__ == 15 && __GNUC_MINOR__ < 3))

This is narrowly scoped (only PCH builds, only affected GCC versions) and references the upstream bug (GCC PR target/123457). The version bounds are conservative and document when the fix is expected upstream.

Minor suggestion: The vec_bfloat16.h convert_bfloat16_float function at line 229 still has the older GCC 14-only workaround:

#if defined(__GNUC__) && __GNUC__ == 14
// Workaround for gcc-14.2.0 ICE during RTL pass: vregs when compiling for SVE
__attribute__((optimize("no-tree-vectorize")))
#endif

This is a different bug (RTL pass ICE, not PCH ICE), so it's correct to leave it as-is. But it may be worth checking if it's still needed with GCC 15.


Summary

All fixes are well-motivated by specific compiler diagnostics, correctly scoped, and use appropriate mechanisms (feature-test macros, version guards, type-correct intrinsics). The PR is ready to merge from a code quality perspective once CI is green.

The one remaining open item mentioned in the PR description is the test_upsampling_bfloat16 stale xfail, which has a separate fix at #182034.

@aditew01

Copy link
Copy Markdown
Collaborator

@pytorchbot merge -i

@pytorchmergebot

Copy link
Copy Markdown
Collaborator

@aditew01

Copy link
Copy Markdown
Collaborator

@pytorchbot revert -m "Breaks nightly whl builds"

@aditew01

Copy link
Copy Markdown
Collaborator

@pytorchbot revert -m "Breaks nightly whl builds" -c "nosignal"

@pytorchmergebot

Copy link
Copy Markdown
Collaborator

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

@pytorchmergebot

Copy link
Copy Markdown
Collaborator

Reverting PR 182016 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit d8407653cb83447ecee64ce609a1b1bc18c5fb60 returned non-zero exit code 1

Auto-merging .ci/docker/build.sh
Auto-merging .github/workflows/docker-builds.yml
Auto-merging .github/workflows/inductor-perf-test-nightly-aarch64.yml
CONFLICT (content): Merge conflict in .github/workflows/inductor-perf-test-nightly-aarch64.yml
Auto-merging .github/workflows/operator_benchmark.yml
Auto-merging .github/workflows/pull.yml
Auto-merging .github/workflows/trunk.yml
CONFLICT (content): Merge conflict in .github/workflows/trunk.yml
error: could not revert d8407653cb8... Update AArch64 toolchains to GCC15 (#182016)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Details for Dev Infra team Raised by workflow job

@pytorch pytorch deleted a comment from pytorch-bot Bot Jun 29, 2026
aditew01 added a commit that referenced this pull request Jun 29, 2026
gplutop7 pushed a commit to gplutop7/pytorch that referenced this pull request Jul 15, 2026
`std::atomic_load`, `std::atomic_exchange`, and `std::atomic_compare_exchange_*` overloads for `std::shared_ptr<T>` are deprecated since C++20 ([ref](https://en.cppreference.com/cpp/memory/shared_ptr/atomic)). Newer GCC/libstdc++ warns on these deprecated overloads, and warning-as-error builds can turn this into a build failure, as encountered in pytorch#182016.

This PR updates `RpcAgent::currentRpcAgent_` from a plain `std::shared_ptr<RpcAgent>` accessed through the deprecated free-function atomic API to `std::atomic<std::shared_ptr<RpcAgent>>` accessed through the modern member API when `__cpp_lib_atomic_shared_ptr` is available.

The implementation keeps the legacy free-function path on older standard libraries that do not provide the C++20 `std::atomic<std::shared_ptr<T>>` specialization.

cc. @aditew01 @malfet
Pull Request resolved: pytorch#185633
Approved by: https://github.com/Skylion007, https://github.com/cyyever
gplutop7 pushed a commit to gplutop7/pytorch that referenced this pull request Jul 15, 2026
Ahead of updating the AArch64 CI toolchains to GCC15 (pytorch#182016), this PR fixes issues that arise on the newer compiler version.

## Fixes included

### 1. HeaderOnlyArrayRef empty vector construction

GCC 15 reports `-Werror=free-nonheap-object` from inlined `HeaderOnlyArrayRef<T>::vec()` while constructing an `std::vector<T>` from `Data` to `Data + Length`. For empty refs, `Data` may be null, and forming `Data + Length` still performs pointer arithmetic on that null pointer. The fix returns `{}` immediately for empty refs to avoid that.

Failure message:

```text
    inlined from 'std::vector<T> c10::HeaderOnlyArrayRef<T>::vec() const [with T = long int]' at /pytorch/torch/headeronly/util/HeaderOnlyArrayRef.h:237:64,
    inlined from 'at::TensorIterator at::native::{anonymous}::_make_unfold_backward_iter_over_grad_out(at::Tensor&, const at::Tensor&, int64_t, int64_t, int64_t)' at /pytorch/aten/src/ATen/native/UnfoldBackward.h:52:65,
    inlined from 'void at::native::{anonymous}::unfold_backward_cpu_kernel(at::Tensor&, const at::Tensor&, int64_t, int64_t, int64_t)' at /pytorch/aten/src/ATen/native/cpu/UnfoldBackwardKernel.cpp:129:41:
/usr/include/c++/15/bits/new_allocator.h:172:66: error: 'void operator delete(void*, std::size_t)' called on pointer '<unknown>' with nonzero offset [8, 9223372036854775800] [-Werror=free-nonheap-object]
  172 |         _GLIBCXX_OPERATOR_DELETE(_GLIBCXX_SIZED_DEALLOC(__p, __n));
      |                                                                  ^
cc1plus: some warnings being treated as errors
```

Files changed:

- `torch/headeronly/util/HeaderOnlyArrayRef.h`

### 2. OperatorName namespace string insertion overlap

`OperatorName::setNamespaceIfNotSet()` used `std::string::replace()` to copy from `name` back into an overlapping region of the same `name` buffer. GCC 15 inlines that path down to `memcpy` and reports `-Werror=restrict`. The fix uses `std::char_traits<char>::move()` for the overlapping move and `std::char_traits<char>::copy()` for the non-overlapping namespace prefix.

Failure message:

```text
    inlined from 'constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::replace(size_type, size_type, const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, size_type, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]' at /usr/include/c++/15/bits/basic_string.h:2434:29,
    inlined from 'bool c10::OperatorName::setNamespaceIfNotSet(const char*)' at /pytorch/aten/src/ATen/core/operator_name.h:44:19,
    inlined from 'bool c10::OperatorName::setNamespaceIfNotSet(const char*)' at /pytorch/aten/src/ATen/core/operator_name.h:38:8,
    inlined from 'virtual void OperatorNameTest_SetNamespaceIfNotSetWithExistingNamespace_Test::TestBody()' at /pytorch/aten/src/ATen/test/operator_name_test.cpp:17:58:
/usr/include/c++/15/bits/char_traits.h:429:56: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' accessing 28 bytes at offsets 11 and 0 overlaps 17 bytes at offset 11 [-Werror=restrict]
  429 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
      |                                        ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
cc1plus: some warnings being treated as errors
```

Files changed:

- `aten/src/ATen/core/operator_name.h`

### 3. SVE PCH internal compiler errors

GCC reports an internal compiler error while compiling AArch64 SVE vector helpers through the Inductor precompiled-header path, matching [GCC PR target/123457](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=123457).

The affected code is valid, but GCC 14/15 can ICE when those SVE intrinsics are compiled with `TORCH_INDUCTOR_PRECOMPILE_HEADERS` enabled. The workaround adds narrow `__attribute__((optimize("O0")))` guards around the specific helper functions that trigger the compiler bug for the affected GCC versions.

Representative failure:

```text
torch._inductor.exc.InductorError: CppCompileError: C++ compile error

Output:
during GIMPLE pass: ccp
... In member function 'at::vec::CPU_CAPABILITY::Vectorized<float> at::vec::CPU_CAPABILITY::Vectorized<float>::tanh() const':
... internal compiler error: Segmentation fault
```

Files changed:

- `aten/src/ATen/cpu/vec/sve/vec_bfloat16.h`
- `aten/src/ATen/cpu/vec/sve/vec_float.h`
- `aten/src/ATen/cpu/vec/sve/vec_int.h`

### 4. SVE BF16 transpose tile aliasing

The SVE BF16 transpose helper used `BFloat16` scratch tiles while also loading and storing those tiles through raw `uint16_t` SVE accesses. This keeps the existing tile representation and `uint16_t` helper path, but makes the scalar tile zero/copy operations access the raw `BFloat16::x` member. That avoids the mixed whole-object `BFloat16` / raw `uint16_t` access pattern that caused stale tile values with GCC15.

Failure message:

```text
======================================================================
FAIL: test_transpose_copy (__main__.CPUReproTests)
----------------------------------------------------------------------
...
AssertionError: Tensor-likes are not close!

Mismatched elements: 8 / 289 (2.8%)
Greatest absolute difference: 2.0625 at index (16, 15) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (16, 8) (up to 0.016 allowed)
```

Files changed:

- `aten/src/ATen/cpu/vec/sve/vec_bfloat16.h`

Pull Request resolved: pytorch#187167
Approved by: https://github.com/aditew01, https://github.com/jansel
gplutop7 pushed a commit to gplutop7/pytorch that referenced this pull request Jul 15, 2026
Based on pytorch#180850, this PR updates AArch64 CI toolchains to GCC15.

The necessary GCC15 enablement fixes are part of pytorch#187167 and pytorch#185633.

Pull Request resolved: pytorch#182016
Approved by: https://github.com/aditew01, https://github.com/Skylion007

Co-authored-by: Aditya Tewari <aditya.tewari@arm.com>
aws-kingrj pushed a commit to amazon-contributing/upstream-to-pytorch that referenced this pull request Jul 29, 2026
`std::atomic_load`, `std::atomic_exchange`, and `std::atomic_compare_exchange_*` overloads for `std::shared_ptr<T>` are deprecated since C++20 ([ref](https://en.cppreference.com/cpp/memory/shared_ptr/atomic)). Newer GCC/libstdc++ warns on these deprecated overloads, and warning-as-error builds can turn this into a build failure, as encountered in pytorch#182016.

This PR updates `RpcAgent::currentRpcAgent_` from a plain `std::shared_ptr<RpcAgent>` accessed through the deprecated free-function atomic API to `std::atomic<std::shared_ptr<RpcAgent>>` accessed through the modern member API when `__cpp_lib_atomic_shared_ptr` is available.

The implementation keeps the legacy free-function path on older standard libraries that do not provide the C++20 `std::atomic<std::shared_ptr<T>>` specialization.

cc. @aditew01 @malfet
Pull Request resolved: pytorch#185633
Approved by: https://github.com/Skylion007, https://github.com/cyyever
aws-kingrj pushed a commit to amazon-contributing/upstream-to-pytorch that referenced this pull request Jul 29, 2026
Ahead of updating the AArch64 CI toolchains to GCC15 (pytorch#182016), this PR fixes issues that arise on the newer compiler version.

## Fixes included

### 1. HeaderOnlyArrayRef empty vector construction

GCC 15 reports `-Werror=free-nonheap-object` from inlined `HeaderOnlyArrayRef<T>::vec()` while constructing an `std::vector<T>` from `Data` to `Data + Length`. For empty refs, `Data` may be null, and forming `Data + Length` still performs pointer arithmetic on that null pointer. The fix returns `{}` immediately for empty refs to avoid that.

Failure message:

```text
    inlined from 'std::vector<T> c10::HeaderOnlyArrayRef<T>::vec() const [with T = long int]' at /pytorch/torch/headeronly/util/HeaderOnlyArrayRef.h:237:64,
    inlined from 'at::TensorIterator at::native::{anonymous}::_make_unfold_backward_iter_over_grad_out(at::Tensor&, const at::Tensor&, int64_t, int64_t, int64_t)' at /pytorch/aten/src/ATen/native/UnfoldBackward.h:52:65,
    inlined from 'void at::native::{anonymous}::unfold_backward_cpu_kernel(at::Tensor&, const at::Tensor&, int64_t, int64_t, int64_t)' at /pytorch/aten/src/ATen/native/cpu/UnfoldBackwardKernel.cpp:129:41:
/usr/include/c++/15/bits/new_allocator.h:172:66: error: 'void operator delete(void*, std::size_t)' called on pointer '<unknown>' with nonzero offset [8, 9223372036854775800] [-Werror=free-nonheap-object]
  172 |         _GLIBCXX_OPERATOR_DELETE(_GLIBCXX_SIZED_DEALLOC(__p, __n));
      |                                                                  ^
cc1plus: some warnings being treated as errors
```

Files changed:

- `torch/headeronly/util/HeaderOnlyArrayRef.h`

### 2. OperatorName namespace string insertion overlap

`OperatorName::setNamespaceIfNotSet()` used `std::string::replace()` to copy from `name` back into an overlapping region of the same `name` buffer. GCC 15 inlines that path down to `memcpy` and reports `-Werror=restrict`. The fix uses `std::char_traits<char>::move()` for the overlapping move and `std::char_traits<char>::copy()` for the non-overlapping namespace prefix.

Failure message:

```text
    inlined from 'constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::replace(size_type, size_type, const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, size_type, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]' at /usr/include/c++/15/bits/basic_string.h:2434:29,
    inlined from 'bool c10::OperatorName::setNamespaceIfNotSet(const char*)' at /pytorch/aten/src/ATen/core/operator_name.h:44:19,
    inlined from 'bool c10::OperatorName::setNamespaceIfNotSet(const char*)' at /pytorch/aten/src/ATen/core/operator_name.h:38:8,
    inlined from 'virtual void OperatorNameTest_SetNamespaceIfNotSetWithExistingNamespace_Test::TestBody()' at /pytorch/aten/src/ATen/test/operator_name_test.cpp:17:58:
/usr/include/c++/15/bits/char_traits.h:429:56: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' accessing 28 bytes at offsets 11 and 0 overlaps 17 bytes at offset 11 [-Werror=restrict]
  429 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
      |                                        ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
cc1plus: some warnings being treated as errors
```

Files changed:

- `aten/src/ATen/core/operator_name.h`

### 3. SVE PCH internal compiler errors

GCC reports an internal compiler error while compiling AArch64 SVE vector helpers through the Inductor precompiled-header path, matching [GCC PR target/123457](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=123457).

The affected code is valid, but GCC 14/15 can ICE when those SVE intrinsics are compiled with `TORCH_INDUCTOR_PRECOMPILE_HEADERS` enabled. The workaround adds narrow `__attribute__((optimize("O0")))` guards around the specific helper functions that trigger the compiler bug for the affected GCC versions.

Representative failure:

```text
torch._inductor.exc.InductorError: CppCompileError: C++ compile error

Output:
during GIMPLE pass: ccp
... In member function 'at::vec::CPU_CAPABILITY::Vectorized<float> at::vec::CPU_CAPABILITY::Vectorized<float>::tanh() const':
... internal compiler error: Segmentation fault
```

Files changed:

- `aten/src/ATen/cpu/vec/sve/vec_bfloat16.h`
- `aten/src/ATen/cpu/vec/sve/vec_float.h`
- `aten/src/ATen/cpu/vec/sve/vec_int.h`

### 4. SVE BF16 transpose tile aliasing

The SVE BF16 transpose helper used `BFloat16` scratch tiles while also loading and storing those tiles through raw `uint16_t` SVE accesses. This keeps the existing tile representation and `uint16_t` helper path, but makes the scalar tile zero/copy operations access the raw `BFloat16::x` member. That avoids the mixed whole-object `BFloat16` / raw `uint16_t` access pattern that caused stale tile values with GCC15.

Failure message:

```text
======================================================================
FAIL: test_transpose_copy (__main__.CPUReproTests)
----------------------------------------------------------------------
...
AssertionError: Tensor-likes are not close!

Mismatched elements: 8 / 289 (2.8%)
Greatest absolute difference: 2.0625 at index (16, 15) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (16, 8) (up to 0.016 allowed)
```

Files changed:

- `aten/src/ATen/cpu/vec/sve/vec_bfloat16.h`

Pull Request resolved: pytorch#187167
Approved by: https://github.com/aditew01, https://github.com/jansel
aws-kingrj pushed a commit to amazon-contributing/upstream-to-pytorch that referenced this pull request Jul 29, 2026
Based on pytorch#180850, this PR updates AArch64 CI toolchains to GCC15.

The necessary GCC15 enablement fixes are part of pytorch#187167 and pytorch#185633.

Pull Request resolved: pytorch#182016
Approved by: https://github.com/aditew01, https://github.com/Skylion007

Co-authored-by: Aditya Tewari <aditya.tewari@arm.com>
@Skylion007

Copy link
Copy Markdown
Collaborator

We hit a GCC13 compiler error unsigned_clamp #189953

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-no-td Do not run TD on this PR ciflow/docker ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: cpu CPU specific problem (e.g., perf, algorithm) module: inductor no-runner-experiments Bypass Meta/LF runner determinator open source release notes: releng release notes category topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants