Fix pathological performance in trait solver cycles with errors

Fuchsia's Starnix system has had a multi-year long bug where
occasionally a typo could cause the rust compiler to take 10+ hours to
report an error. This was particularly hard to trace down since
Starnix's codebase is massive, over 384 thousand lines as of writing.

With the help of treereduce, cargo-minimize, and rustmerge, after about
a month of running we reduced it down to a couple [lines of code], which
only takes about 35 seconds to report an error on my machine. The bug
also appears to happen with `-Z next-solver=no` and `-Z
next-solver=coherence`, but does not occur with `-Z next-solver` or `-Z
next-solver=globally`.

I used Gemini to help diagnose the problem and proposed solution (which
is the one proposed in this patch):

1. The trait solver gets stuck in an exponential loop evaluating
   auto-trait bounds (like Send and Sync) on cyclic types that contain
   compilation errors (TyKind::Error).

2. Normally, if the solver detects a cycle, it prevents the result from
   being stored in the Global Cache because the result depends on the
   current evaluation stack. However, when an error is involved, the
   depth tracking gets pinned to a low value, forcing the solver to rely
   on the short-lived Provisional Cache. Since the provisional cache is
   cleared between high-level iterations of the fulfillment loop, the
   solver ends up re-discovering and re-evaluating the same large cycle
   thousands of times.

3. Allow global caching of results even if they appear stack-dependent,
   provided that the inference context is already "tainted by errors"
   (`self.infcx.tainted_by_errors().is_some()`). This violates the
   strict invariant that global cache entries shouldn't depend on the
   stack, but it is safe because the compilation is already guaranteed
   to fail due to the presence of errors. Prioritizing compiler
   responsiveness and termination over perfect correctness in error
   states is the correct trade-off here.

I added the reduction as the test case for this. However, I don't see an
easy way to catch if this bug comes back. Should we add some way to
timeout the test if it takes longer than 10 seconds to compile? That
could be a source of flakes though.

I don't have any experience with the trait solver code, but I did try to
review the code to the best of my ability. This approach seems a bit of
a bandaid to the solution, but I don't see a better solution. We could
try to teach the solver to not clear the provisional cache in this
circumstance, but I suspect that'd be a pretty invasive change.

I'm guessing if this does cause problems, it might report an incorrect
error, but I (and Gemini) were unable to come up with an example that
reported a different error with and without this fix.

[lines of code]: https://gist.github.com/erickt/255bc4006292cac88de906bd6bd9220a

Author: erickt

compiler/rustc_trait_selection/src/traits/select/mod.rs

@@ -1113,6 +1113,14 @@ impl<'cx, 'tcx> SelectionContext<'cx, 'tcx> {
             debug!("CACHE MISS");
             self.insert_evaluation_cache(param_env, fresh_trait_pred, dep_node, result);
             stack.cache().on_completion(stack.dfn);
+        } else if let Some(_guar) = self.infcx.tainted_by_errors() {
+            // When an error has occurred, we allow global caching of results even if they
+            // appear stack-dependent. This prevents exponential re-evaluation of cycles
+            // in the presence of errors, avoiding compiler hangs like #150907.
+            // This is safe because compilation will fail anyway.
+            debug!("CACHE MISS (tainted by errors)");
+            self.insert_evaluation_cache(param_env, fresh_trait_pred, dep_node, result);
+            stack.cache().on_completion(stack.dfn);
         } else {
             debug!("PROVISIONAL");
             debug!(

tests/ui/traits/cycle-cache-err-150907.rs

@@ -0,0 +1,125 @@
+// Test that we don't hang or take exponential time when evaluating
+// auto-traits for cyclic types containing errors, based on the reproducer
+// provided in issue #150907.
+
+use std::sync::Arc;
+use std::marker::PhantomData;
+
+struct Weak<T>(PhantomData<T>);
+unsafe impl<T: Sync + Send> Send for Weak<T> {}
+unsafe impl<T: Sync + Send> Sync for Weak<T> {}
+
+struct BTreeMap<K, V>(K, V);
+
+trait DeviceOps: Send {}
+
+struct SerialDevice {
+    terminal: Weak<Terminal>,
+}
+
+impl DeviceOps for SerialDevice {}
+
+struct TtyState {
+    terminals: Weak<Terminal>,
+}
+
+struct TerminalMutableState {
+    controller: TerminalController,
+}
+
+struct Terminal {
+    weak_self: Weak<Self>,
+    state: Arc<TtyState>,
+    mutable_state: Weak<TerminalMutableState>,
+}
+
+struct TerminalController {
+    session: Weak<Session>,
+}
+
+struct Kernel {
+    weak_self: Weak<Kernel>,
+    kthreads: KernelThreads,
+    pids: Weak<PidTable>,
+}
+
+struct KernelThreads {
+    system_task: SystemTask,
+    kernel: Weak<Kernel>,
+}
+
+struct SystemTask {
+    system_thread_group: Weak<ThreadGroup>,
+}
+
+enum ProcessEntry {
+    ThreadGroup(Weak<ThreadGroup>),
+}
+
+struct PidEntry {
+    task: Arc<Task>,
+    process: ProcessEntry,
+}
+
+struct PidTable {
+    table: PidEntry,
+    process_groups: Arc<ProcessGroup>,
+}
+
+struct ProcessGroupMutableState {
+    thread_groups: Weak<ThreadGroup>,
+}
+
+struct ProcessGroup {
+    session: Arc<Session>,
+    mutable_state: Arc<ProcessGroupMutableState>,
+}
+
+struct SessionMutableState {
+    process_groups: BTreeMap<(), Weak<ProcessGroup>>,
+    controlling_terminal: ControllingTerminal,
+}
+
+struct Session {
+    mutable_state: Weak<SessionMutableState>,
+}
+
+struct ControllingTerminal {
+    terminal: Arc<Terminal>,
+}
+
+struct TaskPersistentInfoState {
+    thread_group_key: ThreadGroupKey,
+}
+
+struct Task {
+    thread_group_key: ThreadGroupKey,
+    kernel: Arc<Kernel>,
+    thread_group: Arc<ThreadGroup>,
+    persistent_info: Arc<TaskPersistentInfoState>,
+    vfork_event: Arc, //~ ERROR missing generics for struct `Arc`
+}
+
+struct ThreadGroupKey {
+    thread_group: Arc<ThreadGroup>,
+}
+
+struct ThreadGroupMutableState {
+    tasks: BTreeMap<(), TaskContainer>,
+    children: BTreeMap<(), Weak<ThreadGroup>>,
+    process_group: Arc<ProcessGroup>,
+}
+
+struct ThreadGroup {
+    kernel: Arc<Kernel>,
+    mutable_state: Weak<ThreadGroupMutableState>,
+}
+
+struct TaskContainer(Arc<Task>, Arc<TaskPersistentInfoState>);
+
+fn main() {
+    // Trigger auto-trait check for one of the cyclic types
+    is_send::<Kernel>();
+}
+
+fn is_send<T: Send>() {}

tests/ui/traits/cycle-cache-err-150907.stderr

@@ -0,0 +1,14 @@
+error[E0107]: missing generics for struct `Arc`
+  --> $DIR/cycle-cache-err-150907.rs:100:18
+   |
+LL |     vfork_event: Arc,
+   |                  ^^^ expected at least 1 generic argument
+   |
+help: add missing generic argument
+   |
+LL |     vfork_event: Arc<T>,
+   |                     +++
+
+error: aborting due to 1 previous error
+
+For more information about this error, try `rustc --explain E0107`.