Rollup merge of #155355 - erickt:trait-solver-hang, r=lcnr

Fix pathological performance in trait solver cycles with errors

Fuchsia's Starnix system has had a multi-year long bug where occasionally a typo could cause the rust compiler to take 10+ hours to report an error (see #136516 and #150907). This was particularly hard to trace down since Starnix's codebase is massive, over 384 thousand lines as of writing.

With the help of treereduce, cargo-minimize, and rustmerge, after about a month of running we reduced it down to a couple [lines of code], which only takes about 35 seconds to report an error on my machine. The bug also appears to happen with `-Z next-solver=no` and `-Z next-solver=coherence`, but does not occur with `-Z next-solver` or `-Z next-solver=globally`.

I used Gemini to help diagnose the problem and proposed solution (which is the one proposed in this patch):

1. The trait solver gets stuck in an exponential loop evaluating auto-trait bounds (like Send and Sync) on cyclic types that contain compilation errors (TyKind::Error).

2. Normally, if the solver detects a cycle, it prevents the result from being stored in the Global Cache because the result depends on the current evaluation stack. However, when an error is involved, the depth tracking gets pinned to a low value, forcing the solver to rely on the short-lived Provisional Cache. Since the provisional cache is cleared between high-level iterations of the fulfillment loop, the solver ends up re-discovering and re-evaluating the same large cycle thousands of times.

3. Allow global caching of results even if they appear stack-dependent, provided that the inference context is already "tainted by errors" (`self.infcx.tainted_by_errors().is_some()`). This violates the strict invariant that global cache entries shouldn't depend on the stack, but it is safe because the compilation is already guaranteed to fail due to the presence of errors. Prioritizing compiler responsiveness and termination over perfect correctness in error states is the correct trade-off here.

I added the reduction as the test case for this. However, I don't see an easy way to catch if this bug comes back. Should we add some way to timeout the test if it takes longer than 10 seconds to compile? That could be a source of flakes though.

I don't have any experience with the trait solver code, but I did try to review the code to the best of my ability. This approach seems a bit of a bandaid to the solution, but I don't see a better solution. We could try to teach the solver to not clear the provisional cache in this circumstance, but I suspect that'd be a pretty invasive change.

I'm guessing if this does cause problems, it might report an incorrect error, but I (and Gemini) were unable to come up with an example that reported a different error with and without this fix.

Resolves #150907

[lines of code]: https://gist.github.com/erickt/255bc4006292cac88de906bd6bd9220a

Author: JonathanBrouwer

compiler/rustc_trait_selection/src/traits/select/mod.rs

@@ -1115,6 +1115,14 @@ impl<'cx, 'tcx> SelectionContext<'cx, 'tcx> {
             debug!("CACHE MISS");
             self.insert_evaluation_cache(param_env, fresh_trait_pred, dep_node, result);
             stack.cache().on_completion(stack.dfn);
+        } else if let Some(_guar) = self.infcx.tainted_by_errors() {
+            // When an error has occurred, we allow global caching of results even if they
+            // appear stack-dependent. This prevents exponential re-evaluation of cycles
+            // in the presence of errors, avoiding compiler hangs like #150907.
+            // This is safe because compilation will fail anyway.
+            debug!("CACHE MISS (tainted by errors)");
+            self.insert_evaluation_cache(param_env, fresh_trait_pred, dep_node, result);
+            stack.cache().on_completion(stack.dfn);
         } else {
             debug!("PROVISIONAL");
             debug!(

tests/ui/traits/cycle-cache-err-150907.rs

@@ -0,0 +1,125 @@
+// Test that we don't hang or take exponential time when evaluating
+// auto-traits for cyclic types containing errors, based on the reproducer
+// provided in issue #150907.
+
+use std::sync::Arc;
+use std::marker::PhantomData;
+
+struct Weak<T>(PhantomData<T>);
+unsafe impl<T: Sync + Send> Send for Weak<T> {}
+unsafe impl<T: Sync + Send> Sync for Weak<T> {}
+
+struct BTreeMap<K, V>(K, V);
+
+trait DeviceOps: Send {}
+
+struct SerialDevice {
+    terminal: Weak<Terminal>,
+}
+
+impl DeviceOps for SerialDevice {}
+
+struct TtyState {
+    terminals: Weak<Terminal>,
+}
+
+struct TerminalMutableState {
+    controller: TerminalController,
+}
+
+struct Terminal {
+    weak_self: Weak<Self>,
+    state: Arc<TtyState>,
+    mutable_state: Weak<TerminalMutableState>,
+}
+
+struct TerminalController {
+    session: Weak<Session>,
+}
+
+struct Kernel {
+    weak_self: Weak<Kernel>,
+    kthreads: KernelThreads,
+    pids: Weak<PidTable>,
+}
+
+struct KernelThreads {
+    system_task: SystemTask,
+    kernel: Weak<Kernel>,
+}
+
+struct SystemTask {
+    system_thread_group: Weak<ThreadGroup>,
+}
+
+enum ProcessEntry {
+    ThreadGroup(Weak<ThreadGroup>),
+}
+
+struct PidEntry {
+    task: Arc<Task>,
+    process: ProcessEntry,
+}
+
+struct PidTable {
+    table: PidEntry,
+    process_groups: Arc<ProcessGroup>,
+}
+
+struct ProcessGroupMutableState {
+    thread_groups: Weak<ThreadGroup>,
+}
+
+struct ProcessGroup {
+    session: Arc<Session>,
+    mutable_state: Arc<ProcessGroupMutableState>,
+}
+
+struct SessionMutableState {
+    process_groups: BTreeMap<(), Weak<ProcessGroup>>,
+    controlling_terminal: ControllingTerminal,
+}
+
+struct Session {
+    mutable_state: Weak<SessionMutableState>,
+}
+
+struct ControllingTerminal {
+    terminal: Arc<Terminal>,
+}
+
+struct TaskPersistentInfoState {
+    thread_group_key: ThreadGroupKey,
+}
+
+struct Task {
+    thread_group_key: ThreadGroupKey,
+    kernel: Arc<Kernel>,
+    thread_group: Arc<ThreadGroup>,
+    persistent_info: Arc<TaskPersistentInfoState>,
+    vfork_event: Arc, //~ ERROR missing generics for struct `Arc`
+}
+
+struct ThreadGroupKey {
+    thread_group: Arc<ThreadGroup>,
+}
+
+struct ThreadGroupMutableState {
+    tasks: BTreeMap<(), TaskContainer>,
+    children: BTreeMap<(), Weak<ThreadGroup>>,
+    process_group: Arc<ProcessGroup>,
+}
+
+struct ThreadGroup {
+    kernel: Arc<Kernel>,
+    mutable_state: Weak<ThreadGroupMutableState>,
+}
+
+struct TaskContainer(Arc<Task>, Arc<TaskPersistentInfoState>);
+
+fn main() {
+    // Trigger auto-trait check for one of the cyclic types
+    is_send::<Kernel>();
+}
+
+fn is_send<T: Send>() {}

tests/ui/traits/cycle-cache-err-150907.stderr

@@ -0,0 +1,14 @@
+error[E0107]: missing generics for struct `Arc`
+  --> $DIR/cycle-cache-err-150907.rs:100:18
+   |
+LL |     vfork_event: Arc,
+   |                  ^^^ expected at least 1 generic argument
+   |
+help: add missing generic argument
+   |
+LL |     vfork_event: Arc<T>,
+   |                     +++
+
+error: aborting due to 1 previous error
+
+For more information about this error, try `rustc --explain E0107`.