[compiletest] Parallelize test discovery #140177

tmandry · 2025-04-22T23:03:52Z

Certain filesystems are slow to service individual read requests, but can service many in parallel. This change brings down the time to run a single cached test on one of those filesystems from 40s to about 8s.

rustbot · 2025-04-22T23:03:57Z

r? @wesleywiser

rustbot has assigned @wesleywiser.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-04-22T23:03:59Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Certain filesystems for large monorepos are slow to service individual read requests, but can service many in parallel. This change brings down the time to run a single cached test on one of those filesystems from 40s to about 8s.

jieyouxu

Thanks! This is quite a significant improvement even in my local testing. It turns out

Certain filesystems

can even include ReFS on a local Windows dev drive.

Test collection time diff

Measured against 1a5bf12 + naive time recording around top-level collect_tests_from_dir.

`x86_64-unknown-linux-gnu` (WSL)

State	`collect_tests_from_dir` duration on `tests/ui`
Without this PR	362.296834ms
With this PR	248.394981ms

`x86_64-pc-windows-msvc` (PowerShell, native)

State	`collect_tests_from_dir` duration on `tests/ui`
Without this PR	2.5187741s
With this PR	507.9907ms

Additionally, I ran ./x test ui and ./x test run-make to double-check if the number of runned tests and test outcomes look "expected", which they did.

jieyouxu · 2025-04-23T07:43:28Z

@bors r+ rollup

bors · 2025-04-23T07:43:30Z

📌 Commit 09e36ce has been approved by jieyouxu

It is now in the queue for this repository.

[compiletest] Parallelize test discovery Certain filesystems are slow to service individual read requests, but can service many in parallel. This change brings down the time to run a single cached test on one of those filesystems from 40s to about 8s.

Rollup of 23 pull requests Successful merges: - rust-lang#139261 (mitigate MSVC alignment issue on x86-32) - rust-lang#139307 (std: Add performance warnings to HashMap::get_disjoint_mut) - rust-lang#139700 (Autodiff flags) - rust-lang#139752 (set subsections_via_symbols for ld64 helper sections) - rust-lang#139809 (Don't warn about `v128` in wasm ABI transition) - rust-lang#139852 (StableMIR: Implement `CompilerInterface`) - rust-lang#139945 (Extend HIR to track the source and syntax of a lifetime) - rust-lang#140028 (`deref_patterns`: support string and byte string literals in explicit `deref!("...")` patterns) - rust-lang#140139 (rustc_target: Adjust RISC-V feature implication) - rust-lang#140143 (Move `sys::pal::os::Env` into `sys::env`) - rust-lang#140148 (CI: use aws codebuild for job dist-arm-linux) - rust-lang#140150 (fix MAX_EXP and MIN_EXP docs) - rust-lang#140172 (Make algebraic functions into `const fn` items.) - rust-lang#140177 ([compiletest] Parallelize test discovery) - rust-lang#140181 (Remove `synstructure::Structure::underscore_const` calls.) - rust-lang#140184 (Update doc of cygwin target) - rust-lang#140186 (Rename `compute_x` methods) - rust-lang#140187 ([AIX] Handle AIX dynamic library extensions within c-link-to-rust-dylib run-make test) - rust-lang#140191 (Remove git repository from git config) - rust-lang#140194 (minicore: Have `//@ add-core-stubs` also imply `-Cforce-unwind-tables=yes`) - rust-lang#140195 (triagebot: label minicore changes w/ `A-test-infra-minicore` and ping jieyouxu on changes) - rust-lang#140202 (Make #![feature(let_chains)] bootstrap conditional in compiler/) - rust-lang#140214 (Remove comment about handling non-global where bounds with corresponding projection) r? `@ghost` `@rustbot` modify labels: rollup

[compiletest] Parallelize test discovery Certain filesystems are slow to service individual read requests, but can service many in parallel. This change brings down the time to run a single cached test on one of those filesystems from 40s to about 8s.

jieyouxu · 2025-04-24T09:27:16Z

Wait sorry this shouldn't have been rollup=always.
@bors r-

jieyouxu · 2025-04-24T09:31:11Z

@tmandry could you introduce a synthetic compiler tree change, like maybe a temporary comment in rustc/Cargo.toml, to inhibit download-rustc from kicking in? I'd like to double-check through post-merge test metrics that we don't have unexpected test changes. I'll follow-up to revert that comment addition. Like include a

# FIXME(jieyouxu): remove this comment after #140177 lands

(Maybe tool changes should inhibit download-rustc in CI, but that's a separate topic)

jieyouxu · 2025-04-24T09:31:21Z

For post-merge test metrics,
@bors rollup=never

jieyouxu · 2025-04-24T09:37:06Z

Please feel free to r=me after adding a synthetic compiler/ tree change, I'll revert that change after this PR merges if there are no unexpected test changes.

rustbot · 2025-04-24T09:48:05Z

Some changes occurred in coverage instrumentation.

cc @Zalathar

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Zalathar · 2025-04-24T09:48:47Z

I've pushed a trivial compiler style fix to inhibit download-rustc. No need to revert it afterwards.

@bors r=jieyouxu

bors · 2025-04-24T09:48:50Z

📌 Commit f673c9b has been approved by jieyouxu

It is now in the queue for this repository.

bors · 2025-04-26T02:03:57Z

⌛ Testing commit f673c9b with merge d3508a8...

bors · 2025-04-26T05:17:15Z

☀️ Test successful - checks-actions
Approved by: jieyouxu
Pushing d3508a8 to master...

github-actions · 2025-04-26T05:19:37Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 555e1d0 (parent) -> d3508a8 (this PR)

Test differences

No test diffs found

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard d3508a8ad0163fab0c9b2188b3adf43c87200788 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

dist-aarch64-linux: 7658.2s -> 5255.6s (-31.4%)
dist-apple-various: 6529.6s -> 8329.2s (27.6%)
x86_64-apple-1: 8007.5s -> 9612.9s (20.0%)
dist-x86_64-apple: 9499.8s -> 10367.2s (9.1%)
dist-aarch64-apple: 5547.4s -> 5104.4s (-8.0%)
dist-i686-mingw: 8078.5s -> 8662.1s (7.2%)
dist-i686-msvc: 6779.1s -> 7144.3s (5.4%)
aarch64-gnu: 6685.8s -> 6362.0s (-4.8%)
x86_64-msvc-2: 6838.2s -> 7130.6s (4.3%)
x86_64-rust-for-linux: 2740.5s -> 2624.0s (-4.3%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2025-04-26T06:47:37Z

Finished benchmarking commit (d3508a8): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 0.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.0%	[3.0%, 3.0%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.7%	[-0.4%, 3.0%]	3

Cycles

Results (primary -0.6%, secondary 3.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.0%	[3.0%, 3.0%]	1
Improvements ✅ (primary)	-0.6%	[-1.0%, -0.5%]	5
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.6%	[-1.0%, -0.5%]	5

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 776.189s -> 775.41s (-0.10%)
Artifact size: 365.11 MiB -> 365.09 MiB (-0.00%)

rustbot assigned wesleywiser Apr 22, 2025

This comment has been minimized.

Sign in to view

[compiletest] Parallelize test discovery

09e36ce

Certain filesystems for large monorepos are slow to service individual read requests, but can service many in parallel. This change brings down the time to run a single cached test on one of those filesystems from 40s to about 8s.

tmandry force-pushed the compiletest-par branch from 6e8e722 to 09e36ce Compare April 22, 2025 23:04

jieyouxu assigned jieyouxu and unassigned wesleywiser Apr 23, 2025

jieyouxu approved these changes Apr 23, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 23, 2025

Zalathar mentioned this pull request Apr 24, 2025

Rollup of 23 pull requests #140231

Closed

matthiaskrgr mentioned this pull request Apr 24, 2025

Rollup of 9 pull requests #140242

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 24, 2025

Trivial compiler change to inhibit download-rustc in CI

f673c9b

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 24, 2025

Zalathar mentioned this pull request Apr 26, 2025

bootstrap: Inhibit download-rustc in CI when tools are changed #140322

Open

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 26, 2025

bors merged commit d3508a8 into rust-lang:master Apr 26, 2025
7 checks passed

rustbot added this to the 1.88.0 milestone Apr 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[compiletest] Parallelize test discovery #140177

[compiletest] Parallelize test discovery #140177

tmandry commented Apr 22, 2025 •

edited

Loading

rustbot commented Apr 22, 2025

rustbot commented Apr 22, 2025

This comment has been minimized.

jieyouxu left a comment •

edited

Loading

jieyouxu commented Apr 23, 2025

bors commented Apr 23, 2025

jieyouxu commented Apr 24, 2025

jieyouxu commented Apr 24, 2025

jieyouxu commented Apr 24, 2025

jieyouxu commented Apr 24, 2025

rustbot commented Apr 24, 2025

Zalathar commented Apr 24, 2025

bors commented Apr 24, 2025

bors commented Apr 26, 2025

bors commented Apr 26, 2025

github-actions bot commented Apr 26, 2025

rust-timer commented Apr 26, 2025

[compiletest] Parallelize test discovery #140177

[compiletest] Parallelize test discovery #140177

Conversation

tmandry commented Apr 22, 2025 • edited Loading

rustbot commented Apr 22, 2025

rustbot commented Apr 22, 2025

This comment has been minimized.

jieyouxu left a comment • edited Loading

Choose a reason for hiding this comment

Test collection time diff

x86_64-unknown-linux-gnu (WSL)

x86_64-pc-windows-msvc (PowerShell, native)

jieyouxu commented Apr 23, 2025

bors commented Apr 23, 2025

jieyouxu commented Apr 24, 2025

jieyouxu commented Apr 24, 2025

jieyouxu commented Apr 24, 2025

jieyouxu commented Apr 24, 2025

rustbot commented Apr 24, 2025

Zalathar commented Apr 24, 2025

bors commented Apr 24, 2025

bors commented Apr 26, 2025

bors commented Apr 26, 2025

github-actions bot commented Apr 26, 2025

Test differences

Job duration changes

rust-timer commented Apr 26, 2025

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

tmandry commented Apr 22, 2025 •

edited

Loading

jieyouxu left a comment •

edited

Loading

`x86_64-unknown-linux-gnu` (WSL)

`x86_64-pc-windows-msvc` (PowerShell, native)