`*aarch64-msvc*` job logs sometimes become massive

[jieyouxu]

If I download the jobs log for e.g. https://github.com/rust-lang/rust/actions/runs/24973805754/job/73121722739, I notice that the *aarch64-msvc* jobs are massive (approx. 245 MiB give or take). Other jobs are usually 2 orders of magnitude smaller (around a few MiBs).

Key observations:

• Only happens for *aarch64-msvc*, not *x86_64-msvc*
• Seems to be a lot of logs when building LLVM

dust -F -r
1.0G └─┬ .                                                    │██████████████████████████████████████████████████████████████████████ │ 100%
247M   ├── 24_auto - aarch64-msvc-1.txt                       │█████████████████                                                      │  23%
244M   ├── 7_auto - aarch64-msvc-2.txt                        │█████████████████                                                      │  23%
243M   ├── 10_auto - dist-aarch64-msvc.txt                    │████████████████                                                       │  23%
 12M   ├── 51_auto - dist-arm-linux-gnueabi.txt               │█                                                                      │   1%
 12M   ├── 80_auto - dist-armv7-linux.txt                     │█                                                                      │   1%
 12M   ├── 78_auto - dist-armhf-linux.txt                     │█                                                                      │   1%
9.0M   ├── 71_auto - dist-i586-gnu-i586-i686-musl.txt         │█                                                                      │   1%
8.6M   ├── 27_auto - test-various.txt                         │█                                                                      │   1%
8.1M   ├── 60_auto - dist-x86_64-linux.txt                    │█                                                                      │   1%
7.8M   ├── 77_auto - aarch64-gnu.txt                          │█                                                                      │   1%
7.7M   ├── 21_auto - aarch64-apple.txt                        │█                                                                      │   1%
7.7M   ├── 56_auto - x86_64-gnu-llvm-21-3.txt                 │█                                                                      │   1%
7.7M   ├── 49_auto - x86_64-gnu-llvm-22-3.txt                 │█                                                                      │   1%
7.1M   ├── 9_auto - dist-x86_64-msvc.txt                      │█                                                                      │   1%
7.0M   ├── 85_auto - dist-aarch64-linux.txt                   │█                                                                      │   1%
6.9M   ├── 50_auto - x86_64-gnu-llvm-21-2.txt                 │█                                                                      │   1%
6.9M   ├── 37_auto - x86_64-gnu-llvm-22-2.txt                 │█                                                                      │   1%
6.5M   ├── 28_auto - x86_64-gnu-nopt.txt                      │█                                                                      │   1%
6.4M   ├── 35_auto - optional-x86_64-gnu-parallel-frontend.txt│█                                                                      │   1%
6.4M   ├── 23_auto - x86_64-gnu-stable.txt                    │█                                                                      │   1%
6.4M   ├── 33_auto - x86_64-gnu.txt                           │█                                                                      │   1%
5.8M   ├── 25_auto - x86_64-msvc-1.txt                        │█                                                                      │   1%
5.7M   ├── 18_auto - i686-msvc-1.txt                          │█                                                                      │   1%
5.6M   ├── 12_auto - x86_64-mingw-1.txt                       │█                                                                      │   1%
5.6M   ├── 43_auto - i686-gnu-nopt-1.txt                      │█                                                                      │   1%
5.6M   ├── 72_auto - i686-gnu-1.txt                           │█                                                                      │   1%
4.7M   ├── 6_auto - aarch64-gnu-llvm-21-1.txt                 │█                                                                      │   0%
4.5M   ├── 82_auto - armhf-gnu.txt                            │█                                                                      │   0%
4.3M   ├── 63_auto - arm-android.txt                          │█                                                                      │   0%
4.3M   └── 54_auto - x86_64-gnu-debug.txt                     │█                                                                      │   0%

What the huge amounts of LLVM build logs look like:

2026-04-27T02:45:26.1438350Z running: "cmake" "--build" "C:\\a\\rust\\rust\\build\\aarch64-pc-windows-msvc\\llvm\\build" "--target" "install" "--config" "Release" "--" "-j" "4"
2026-04-27T02:45:27.9450288Z [1/3993] Building CXX object lib\Demangle\CMakeFiles\LLVMDemangle.dir\Demangle.cpp.obj
2026-04-27T02:45:27.9451762Z Note: including file: C:\a\rust\rust\src\llvm-project\llvm\include\llvm/Demangle/Demangle.h
2026-04-27T02:45:27.9452384Z Note: including file:  C:\a\rust\rust\src\llvm-project\llvm\include\llvm/Demangle\DemangleConfig.h
2026-04-27T02:45:27.9453070Z Note: including file:   C:\a\rust\rust\build\aarch64-pc-windows-msvc\llvm\build\include\llvm/Config/llvm-config.h
2026-04-27T02:45:27.9454351Z Note: including file:   C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\cassert
2026-04-27T02:45:27.9455233Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\yvals_core.h
2026-04-27T02:45:27.9456341Z Note: including file:     C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h
2026-04-27T02:45:27.9457192Z Note: including file:      C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\vcruntime.h
2026-04-27T02:45:27.9457987Z Note: including file:       C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\sal.h
2026-04-27T02:45:27.9459246Z Note: including file:        C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\concurrencysal.h
2026-04-27T02:45:27.9459977Z Note: including file:       C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\vadefs.h
2026-04-27T02:45:27.9460934Z Note: including file:        C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\vadefs.h
2026-04-27T02:45:27.9461787Z Note: including file:      C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\xkeycheck.h
2026-04-27T02:45:27.9462506Z Note: including file:    C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt\assert.h
2026-04-27T02:45:27.9463115Z Note: including file:     C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt\corecrt.h
2026-04-27T02:45:27.9463844Z Note: including file:  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\cstddef
2026-04-27T02:45:27.9464933Z Note: including file:   C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\yvals_core.h
2026-04-27T02:45:27.9465501Z Note: including file:   C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\stddef.h
2026-04-27T02:45:27.9466617Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_header_macro.h
2026-04-27T02:45:27.9467527Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_ptrdiff_t.h
2026-04-27T02:45:27.9468217Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_size_t.h
2026-04-27T02:45:27.9469217Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_wchar_t.h
2026-04-27T02:45:27.9469828Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_null.h
2026-04-27T02:45:27.9470896Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_nullptr_t.h
2026-04-27T02:45:27.9471526Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_max_align_t.h
2026-04-27T02:45:27.9472522Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\__stddef_offsetof.h
2026-04-27T02:45:27.9473280Z Note: including file:   C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\xtr1common
2026-04-27T02:45:27.9473972Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\yvals_core.h
2026-04-27T02:45:27.9474655Z Note: including file:  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\optional
2026-04-27T02:45:27.9475689Z Note: including file:   C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals.h
2026-04-27T02:45:27.9476603Z Note: including file:    C:\a\rust\rust\citools\clang-rust\lib\clang\20\include\yvals_core.h
2026-04-27T02:45:27.9477532Z Note: including file:    C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt\crtdbg.h
2026-04-27T02:45:27.9478291Z Note: including file:     C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\vcruntime_new_debug.h
2026-04-27T02:45:27.9479155Z Note: including file:      C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\vcruntime_new.h
2026-04-27T02:45:27.9480302Z Note: including file:    C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\crtdefs.h
2026-04-27T02:45:27.9481097Z Note: including file:    C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\use_ansi.h
2026-04-27T02:45:27.9482040Z Note: including file:   C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\exception
2026-04-27T02:45:27.9482835Z Note: including file:    C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44

[jieyouxu]

Backlink to thread: #t-infra > large aarch64-msvc build logs

[jieyouxu]

Have a hard time pinpointing why. Might be related:

• /showIncludes implicit dependency scanning sporadically broken with clang-cl (and Ninja) due to mixed output llvm/llvm-project#55776

[jieyouxu]

Randomly picking some example runs:

• Rollup of 7 pull requests #155766 -> https://github.com/rust-lang/rust/actions/runs/24924912928/job/72993190585 -> huge
• Rollup of 10 pull requests #155542 -> https://github.com/rust-lang/rust/actions/runs/24659032555/job/72099992692 -> not huge

344M └─┬ .                                                    │██████████████████████████████████████████████████████████████████████ │ 100%
 12M   ├── 82_auto - dist-arm-linux-gnueabi.txt               │███                                                                    │   4%
 12M   ├── 85_auto - dist-armv7-linux.txt                     │███                                                                    │   4%
 12M   ├── 17_auto - dist-armhf-linux.txt                     │███                                                                    │   4%
9.0M   ├── 73_auto - dist-i586-gnu-i586-i686-musl.txt         │██                                                                     │   3%
8.5M   ├── 74_auto - test-various.txt                         │██                                                                     │   2%
8.1M   ├── 38_auto - dist-x86_64-linux.txt                    │██                                                                     │   2%
7.8M   ├── 83_auto - aarch64-gnu.txt                          │██                                                                     │   2%
7.7M   ├── 36_auto - x86_64-gnu-llvm-22-3.txt                 │██                                                                     │   2%
7.7M   ├── 76_auto - x86_64-gnu-llvm-21-3.txt                 │██                                                                     │   2%
7.7M   ├── 49_auto - aarch64-apple.txt                        │██                                                                     │   2%
7.1M   ├── 24_auto - dist-x86_64-msvc.txt                     │██                                                                     │   2%
7.0M   ├── 80_auto - dist-aarch64-linux.txt                   │██                                                                     │   2%
6.9M   ├── 68_auto - x86_64-gnu-llvm-21-2.txt                 │██                                                                     │   2%
6.9M   ├── 72_auto - x86_64-gnu-llvm-22-2.txt                 │██                                                                     │   2%
6.5M   ├── 18_auto - x86_64-gnu-nopt.txt                      │██                                                                     │   2%
6.4M   ├── 33_auto - optional-x86_64-gnu-parallel-frontend.txt│██                                                                     │   2%
6.4M   ├── 35_auto - x86_64-gnu.txt                           │██                                                                     │   2%
6.4M   ├── 32_auto - x86_64-gnu-stable.txt                    │██                                                                     │   2%
5.8M   ├── 70_auto - x86_64-msvc-1.txt                        │██                                                                     │   2%
5.6M   └── 45_auto - aarch64-msvc-1.txt                       │██                                                                     │   2%

I think this is indeed flaky

[jieyouxu]

Discussed in #t-infra > meeting 2026-04-27 @ 💬. Maybe we can try to divert llvm build logs to a separate file, and only reproduce / upload as a separate artifact in case of failure.