regression: crate compiles much slower with 1.82 #132064

Nutomic · 2024-10-23T13:06:28Z

Our project takes much longer to compile with Rust 1.82 compared to previous versions. To reproduce, git clone https://github.com/LemmyNet/lemmy.git and run cargo check (also happens with build or lint). The problem happens specifically with the crate lemmy_db_views.

cargo check with 1.82: 6m 19s

cargo check with 1.81: 1m 22s

Version it worked on

It most recently worked on:1.81

Version with regression

rustc --version --verbose:

cargo 1.82.0 (8f40fc59f 2024-08-21)
release: 1.82.0
commit-hash: 8f40fc59fb0c8df91c97405785197f3c630304ea
commit-date: 2024-08-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Manjaro 24.1.1 (Xahea) [64-bit]

@rustbot modify labels: +regression-from-stable-to-1.82 -regression-untriaged

The text was updated successfully, but these errors were encountered:

lqd · 2024-10-23T16:55:37Z

This is another regression from #126024

Nothing4You · 2024-10-23T17:10:22Z

some more information:

1.81

$ rustup override set 1.81
info: override toolchain for '/path/to/lemmy' set to '1.81-aarch64-apple-darwin'
$ rustc --version --verbose
rustc 1.81.0 (eeb90cda1 2024-09-04)
binary: rustc
commit-hash: eeb90cda1969383f56a2637cbd3037bdf598841c
commit-date: 2024-09-04
host: aarch64-apple-darwin
release: 1.81.0
LLVM version: 18.1.7
$ rm -rf target && cargo build --timings
[…]
    Finished `dev` profile [unoptimized] target(s) in 2m 54s

1.82

$ rustup override set 1.82
info: override toolchain for '/path/to/lemmy' set to '1.82-aarch64-apple-darwin'
$ rustc --version --verbose
rustc 1.82.0 (f6e511eec 2024-10-15)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: aarch64-apple-darwin
release: 1.82.0
LLVM version: 19.1.1
$ rm -rf target && cargo build --timings
[…]
    Finished `dev` profile [unoptimized] target(s) in 9m 58s

cargo-timings.zip

jieyouxu · 2024-11-07T10:00:14Z

Triage notes (AFAIUI): #132625 is merged, but the compile time is not fully clawed back as #132625 is a compromise between (full) soundness and performance in favor of a full revert (full revert would bring back more soundness problems AFAICT)

This refines the check implemented in #126024 to only disable the global cache if the predicate being considered has opaques in it. This is still theoretically unsound, since goals can indirectly rely on opaques in the defining scope, but we're much less likely to hit it.

It doesn't totally fix #132064: for example, lemmy goes from 1:29 (on rust 1.81) to 9:53 (on nightly) to 4:07 (after this PR). But I think it's at least more sound than a total revert :/

Future improvements might be possible to further claw back the compile times, but should at least be better for now with #132625.

(Please feel free to edit this message to better reflect the status)

sunshowers · 2024-11-08T21:53:19Z

Chiming in -- at Oxide, in our largest repository, this appears to have made many operations much slower. For example, cargo check is twice as slow:

1.81: 3m 37s
1.82: 6m 51s

The slowdown is isolated to Diesel-generated code.

We've pinned to 1.81 for now.

lqd · 2024-11-08T21:54:49Z

Can you test on nightly? Things should be better with #132625.

sunshowers · 2024-11-08T22:04:06Z

On my machine, cargo check --all-targets against Omicron 3f6cd90d6a0fe523349e52e6df0554f67167130f:

Rust 1.81.0:
853.39s user 143.94s system 966% cpu 1:43.15 total

Rust 1.84.0-nightly (b91a3a056 2024-11-07):
847.84s user 142.64s system 955% cpu 1:43.70 total

That looks much better. Thanks!

sunshowers · 2024-11-08T22:13:29Z

(Not clear to me why it looks like the regression has been fully addressed for us, but not for Lemmy.)

sunshowers · 2024-11-08T22:26:33Z

Ah, okay, some commands are still slower. With the same revision, cargo build -p omicron-nexus:

Rust 1.81:
664.65s user 95.12s system 600% cpu 2:06.44 total

Rust 1.84.0-nightly (b91a3a056 2024-11-07):
641.80s user 90.41s system 542% cpu 2:15.03 total

I don't have the time at the moment to dig into exactly what's causing the slowdown. Our Diesel-generated code is already the slowest to compile, so any regression to it is pretty rough for iteration cycles.

Nothing4You · 2024-11-08T22:43:32Z

just did another cargo build test with lemmy on macos arm64:

rustc 1.81.0 (eeb90cd 2024-09-04):

Finished dev profile [unoptimized] target(s) in 2m 49s

rustc 1.84.0-nightly (b91a3a0 2024-11-07):

Finished dev profile [unoptimized] target(s) in 3m 04s

build times for lemmy seem to be fine again for me as well.

@jmpesp

Rust 1.82 has a rather serious compile time regression in Diesel-generated code: rust-lang/rust#132064. This makes Omicron builds up to twice as slow. We used to be on 1.80 before moving to 1.82, but 1.81 has some nice things like `#[expect]`. So move to 1.81. We expect the regression to be fixed in Rust 1.84. Thanks to @jmpesp for tracking this down!

compiler-errors · 2024-11-09T00:44:14Z

Ah, okay, some commands are still slower. With the same revision, cargo build -p omicron-nexus:
Rust 1.81:
664.65s user 95.12s system 600% cpu 2:06.44 total

Rust 1.84.0-nightly (b91a3a056 2024-11-07):
641.80s user 90.41s system 542% cpu 2:15.03 total
I don't have the time at the moment to dig into exactly what's causing the slowdown. Our Diesel-generated code is already the slowest to compile, so any regression to it is pretty rough for iteration cycles.

Hey @sunshowers, I just wanted to let you know that while I'm glad your build times are back down from their 2x build time regressions, we're probably not going to be able to recover that final ~9 seconds that you noted in your comment for omicron-nexus (2:06 -> 2:15) anytime soon, at least not without reintroducing unsoundness into the trait system or spending a disproportionate amount of time to investigate caching improvements to the old trait solver (which is already on its way out, and pretty hard to improve today).

Sorry, and I hope that it doesn't hinder your builds too much! I'll nominate my fix PR for 1.83 backport so it may get to y'all sooner.

sunshowers · 2024-11-09T01:17:13Z

Thanks @compiler-errors. Looking forward to the new trait solver!

jieyouxu · 2024-12-10T01:54:43Z

Looks accidental due to magic comment

We're moving three versions ahead, so this is something we should do with some caution. However there's at least one important illumos-specific fix that's in 1.84 (rust-lang/rust#132984) so we should definitely move over. I've verified that rust-lang/rust#132064 no longer affects us. To be more precise, its impact has been mostly taken care of -- there's still a small regression, see [this comment](rust-lang/rust#132064 (comment)), but the Rust team has said it's one that should get better in the next few months.

dessalines · 2025-04-04T18:15:21Z

This is still an issue with 1.88.0-nightly, and probably a dupe of #131231

lqd · 2025-04-04T18:51:57Z

That seems unlikely: this is an issue for stable compilation, whereas #131231 is about the unstable new trait solver performance.

Nothing4You · 2025-04-10T13:40:58Z

I believe the original issue that was reported here should be resolved.
From the 0.19.11 tag on https://github.com/LemmyNet/lemmy I get the slow builds on Rust 1.82.

for v in 1.8{1..6} ; do rustup override set "$v" && rustc --version --verbose && rm -rf target && cargo build --timings && mv target/cargo-timings "cargo-timings-$v" ; done

1.81

+-zsh:171> rustc --version --verbose
rustc 1.81.0 (eeb90cda1 2024-09-04)
binary: rustc
commit-hash: eeb90cda1969383f56a2637cbd3037bdf598841c
commit-date: 2024-09-04
host: aarch64-apple-darwin
release: 1.81.0
LLVM version: 18.1.7
+-zsh:171> rm -rf target
+-zsh:171> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 2m 35s

1.82

+-zsh:171> rustc --version --verbose
rustc 1.82.0 (f6e511eec 2024-10-15)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: aarch64-apple-darwin
release: 1.82.0
LLVM version: 19.1.1
+-zsh:171> rm -rf target
+-zsh:171> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 9m 50s

1.83

+-zsh:171> rustc --version --verbose
rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: aarch64-apple-darwin
release: 1.83.0
LLVM version: 19.1.1
+-zsh:171> rm -rf target
+-zsh:171> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 2m 47s

1.84

+-zsh:171> rustc --version --verbose
rustc 1.84.1 (e71f9a9a9 2025-01-27)
binary: rustc
commit-hash: e71f9a9a98b0faf423844bf0ba7438f29dc27d58
commit-date: 2025-01-27
host: aarch64-apple-darwin
release: 1.84.1
LLVM version: 19.1.5
+-zsh:171> rm -rf target
+-zsh:171> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 2m 58s

1.85

+-zsh:171> rustc --version --verbose
rustc 1.85.1 (4eb161250 2025-03-15)
binary: rustc
commit-hash: 4eb161250e340c8f48f66e2b929ef4a5bed7c181
commit-date: 2025-03-15
host: aarch64-apple-darwin
release: 1.85.1
LLVM version: 19.1.7
+-zsh:171> rm -rf target
+-zsh:171> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 2m 53s

1.86

+-zsh:171> rustc --version --verbose
rustc 1.86.0 (05f9846f8 2025-03-31)
binary: rustc
commit-hash: 05f9846f893b09a1be1fc8560e33fc3c815cfecb
commit-date: 2025-03-31
host: aarch64-apple-darwin
release: 1.86.0
LLVM version: 19.1.7
+-zsh:171> rm -rf target
+-zsh:171> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 2m 54s

The main branch of the Lemmy repo is currently seeing significantly slower builds, but that doesn't seem to be caused by the Rust version used and is likely due to other changes in the code since the switch from 0.19.

1.81: 7m 08s
1.82: 68m 45s
1.83: 11m 09s
1.84: 13m 50s
1.85: 11m 05s
1.86: 10m 56s

dessalines · 2025-04-10T13:51:57Z

The ~4m difference is enough to make me develop using 1.81 . I'm not sure if the clippy rebuild times are different also, but 1.88 seems much slower while developing.

Nothing4You · 2025-04-10T17:33:11Z

perhaps a different underlying issue than the one reported initially, as that was primarily for 1.82?

these builds are on LemmyNet/lemmy@80a5406, full output from the previous runs and added beta and nightly:

1.81

+-zsh:177> rustc --version --verbose
rustc 1.81.0 (eeb90cda1 2024-09-04)
binary: rustc
commit-hash: eeb90cda1969383f56a2637cbd3037bdf598841c
commit-date: 2024-09-04
host: aarch64-apple-darwin
release: 1.81.0
LLVM version: 18.1.7
+-zsh:177> rm -rf target
+-zsh:177> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 7m 08s

1.82

+-zsh:177> rustc --version --verbose
rustc 1.82.0 (f6e511eec 2024-10-15)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: aarch64-apple-darwin
release: 1.82.0
LLVM version: 19.1.1
+-zsh:177> rm -rf target
+-zsh:177> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 68m 45s

1.83

+-zsh:177> rustc --version --verbose
rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: aarch64-apple-darwin
release: 1.83.0
LLVM version: 19.1.1
+-zsh:177> rm -rf target
+-zsh:177> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 11m 09s

1.84

+-zsh:177> rustc --version --verbose
rustc 1.84.1 (e71f9a9a9 2025-01-27)
binary: rustc
commit-hash: e71f9a9a98b0faf423844bf0ba7438f29dc27d58
commit-date: 2025-01-27
host: aarch64-apple-darwin
release: 1.84.1
LLVM version: 19.1.5
+-zsh:177> rm -rf target
+-zsh:177> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 13m 50s

1.85

+-zsh:177> rustc --version --verbose
rustc 1.85.1 (4eb161250 2025-03-15)
binary: rustc
commit-hash: 4eb161250e340c8f48f66e2b929ef4a5bed7c181
commit-date: 2025-03-15
host: aarch64-apple-darwin
release: 1.85.1
LLVM version: 19.1.7
+-zsh:177> rm -rf target
+-zsh:177> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 11m 05s

1.86

+-zsh:177> rustc --version --verbose
rustc 1.86.0 (05f9846f8 2025-03-31)
binary: rustc
commit-hash: 05f9846f893b09a1be1fc8560e33fc3c815cfecb
commit-date: 2025-03-31
host: aarch64-apple-darwin
release: 1.86.0
LLVM version: 19.1.7
+-zsh:177> rm -rf target
+-zsh:177> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 10m 56s

1.87 (beta)

+-zsh:184> rustc --version --verbose
rustc 1.87.0-beta.3 (a22ecb51f 2025-04-07)
binary: rustc
commit-hash: a22ecb51f85284abc5b18a32b27696b0e3e9cf85
commit-date: 2025-04-07
host: aarch64-apple-darwin
release: 1.87.0-beta.3
LLVM version: 20.1.1
+-zsh:184> rm -rf target
+-zsh:184> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 11m 54s

1.88 (nightly)

+-zsh:184> rustc --version --verbose
rustc 1.88.0-nightly (934880f58 2025-04-09)
binary: rustc
commit-hash: 934880f586f6ac1f952c7090e2a943fcd7775e7b
commit-date: 2025-04-09
host: aarch64-apple-darwin
release: 1.88.0-nightly
LLVM version: 20.1.2
+-zsh:184> rm -rf target
+-zsh:184> cargo build --timings
[...]
    Finished `dev` profile [unoptimized] target(s) in 11m 47s

Comparing 1.81, which currently still has the best compilation times, with current stable, beta and nightly, we can see that compile times for certain crates is still substantially worse.
While some third party crates are compiling quite a bit faster now, first party Lemmy crates still have a substantial degradation.

1.81:

1.86:

1.87:

1.88:

full html reports: lemmy-cargo-timings.zip

dessalines · 2025-04-10T19:36:17Z

It very much could be a different issue, but its still likely diesel related, as evidenced by the lemmy_db_views compile-time difference:

Version	time
1.81	123s
1.88	392s

Nutomic added C-bug Category: This is a bug. regression-untriaged Untriaged performance or correctness regression. labels Oct 23, 2024

rustbot added I-prioritize Issue: Indicates that prioritization has been requested for this issue. needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Oct 23, 2024

lqd mentioned this issue Oct 23, 2024

Revert #126024 "Do not use global caches if opaque types can be defined" #132075

Closed

Noratrieb removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Oct 23, 2024

workingjubilee added regression-from-stable-to-stable Performance or correctness regression from one stable version to another. and removed regression-untriaged Untriaged performance or correctness regression. labels Oct 24, 2024

Nutomic mentioned this issue Oct 28, 2024

Build times have deteriorated significantly LemmyNet/lemmy#3610

Closed

jeff-hiner mentioned this issue Oct 31, 2024

allow_tables_to_appear_in_same_query! does not scale with larger number of tables diesel-rs/diesel#4333

Open

3 tasks

compiler-errors mentioned this issue Nov 5, 2024

Only disable cache if predicate has opaques within it #132625

Merged

bors closed this as completed in c07aa1e Nov 7, 2024

lqd reopened this Nov 7, 2024

apiraino removed the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Nov 7, 2024

jieyouxu added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. I-compiletime Issue: Problems and improvements with respect to compile times. labels Nov 7, 2024

sunshowers mentioned this issue Nov 8, 2024

move back to Rust 1.81 oxidecomputer/omicron#7020

Merged

antiguru mentioned this issue Nov 14, 2024

protobuf-src compile time increased massively from v1.1.0 -> v2.1.0 MaterializeInc/rust-protobuf-native#24

Closed

shepmaster closed this as completed in shepmaster/rust-analyzer@45ca9bb Dec 9, 2024

jieyouxu reopened this Dec 10, 2024

sunshowers mentioned this issue Jan 10, 2025

[meta] update rust to 1.84 oxidecomputer/omicron#7334

Merged

dessalines mentioned this issue Feb 1, 2025

Converting joins, filters, and selects to common functions for DB views. LemmyNet/lemmy#5359

Merged

dessalines mentioned this issue Mar 7, 2025

Create actions structs LemmyNet/lemmy#5482

Merged

sunshowers mentioned this issue Mar 10, 2025

base64ct: un-yank v1.7.0 or publish v2.0.0? RustCrypto/formats#1695

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

regression: crate compiles much slower with 1.82 #132064

regression: crate compiles much slower with 1.82 #132064

Nutomic commented Oct 23, 2024

lqd commented Oct 23, 2024

Nothing4You commented Oct 23, 2024

jieyouxu commented Nov 7, 2024 •

edited

Loading

sunshowers commented Nov 8, 2024 •

edited

Loading

lqd commented Nov 8, 2024

sunshowers commented Nov 8, 2024 •

edited

Loading

sunshowers commented Nov 8, 2024

sunshowers commented Nov 8, 2024

Nothing4You commented Nov 8, 2024

compiler-errors commented Nov 9, 2024

sunshowers commented Nov 9, 2024

jieyouxu commented Dec 10, 2024 •

edited

Loading

dessalines commented Apr 4, 2025

lqd commented Apr 4, 2025

Nothing4You commented Apr 10, 2025

dessalines commented Apr 10, 2025

Nothing4You commented Apr 10, 2025 •

edited

Loading

dessalines commented Apr 10, 2025 •

edited

Loading

regression: crate compiles much slower with 1.82 #132064

regression: crate compiles much slower with 1.82 #132064

Comments

Nutomic commented Oct 23, 2024

Version it worked on

Version with regression

lqd commented Oct 23, 2024

Nothing4You commented Oct 23, 2024

jieyouxu commented Nov 7, 2024 • edited Loading

sunshowers commented Nov 8, 2024 • edited Loading

lqd commented Nov 8, 2024

sunshowers commented Nov 8, 2024 • edited Loading

sunshowers commented Nov 8, 2024

sunshowers commented Nov 8, 2024

Nothing4You commented Nov 8, 2024

compiler-errors commented Nov 9, 2024

sunshowers commented Nov 9, 2024

jieyouxu commented Dec 10, 2024 • edited Loading

dessalines commented Apr 4, 2025

lqd commented Apr 4, 2025

Nothing4You commented Apr 10, 2025

dessalines commented Apr 10, 2025

Nothing4You commented Apr 10, 2025 • edited Loading

dessalines commented Apr 10, 2025 • edited Loading

jieyouxu commented Nov 7, 2024 •

edited

Loading

sunshowers commented Nov 8, 2024 •

edited

Loading

sunshowers commented Nov 8, 2024 •

edited

Loading

jieyouxu commented Dec 10, 2024 •

edited

Loading

Nothing4You commented Apr 10, 2025 •

edited

Loading

dessalines commented Apr 10, 2025 •

edited

Loading