[LV] Invalidate disposition of SCEV values after loop vectorization

[dtcxzyw]

This PR fixes the assertion failure of SE.verify() after loop vectorization. I found this problem when investigating #69097.

[llvmbot]

@llvm/pr-subscribers-llvm-transforms

Author: Yingwei Zheng (dtcxzyw)

Changes

This PR fixes the assertion failure of SE.Verify() after loop vectorization. I found this problem when investigating #69097.

---

Full diff: https://github.com/llvm/llvm-project/pull/69230.diff

2 Files Affected:

• (modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+4-2)
• (added) llvm/test/Transforms/LoopVectorize/scev-invalidation.ll (+22)

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 2ca7e75f97f0f02..685c561fc4d2e1b 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -3543,6 +3543,7 @@ void InnerLoopVectorizer::fixVectorizedLoop(VPTransformState &State,
 
   // Forget the original basic block.
   PSE.getSE()->forgetLoop(OrigLoop);
+  PSE.getSE()->forgetBlockAndLoopDispositions();
 
   // After vectorization, the exit blocks of the original loop will have
   // additional predecessors. Invalidate SCEVs for the exit phis in case SE
@@ -10389,8 +10390,9 @@ PreservedAnalyses LoopVectorizePass::run(Function &F,
       PA.preserve<DominatorTreeAnalysis>();
       PA.preserve<ScalarEvolutionAnalysis>();
 
-#ifdef EXPENSIVE_CHECKS
-      SE.verify();
+#ifndef NDEBUG
+      if (VerifySCEV)
+        SE.verify();
 #endif
     }
 
diff --git a/llvm/test/Transforms/LoopVectorize/scev-invalidation.ll b/llvm/test/Transforms/LoopVectorize/scev-invalidation.ll
new file mode 100644
index 000000000000000..08163293c14e85e
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/scev-invalidation.ll
@@ -0,0 +1,22 @@
+; RUN: opt < %s -passes="require<scalar-evolution>,print<scalar-evolution>,loop-vectorize" --verify-scev -force-vector-interleave=2 -force-vector-width=8 -S | FileCheck %s
+
+; CHECK-LABEL: @main(
+; CHECK: vector.body
+define i32 @main(i32 %.pre) {
+entry:
+  br label %for.body
+
+for.body:
+  %g.019 = phi i16 [ 0, %entry ], [ %dec7, %for.body ]
+  %and = and i32 %.pre, 40
+  %0 = sub i32 0, %and
+  %dec7 = add i16 %g.019, 1
+  %cmp.not = icmp eq i16 %dec7, 0
+  br i1 %cmp.not, label %for.inc16, label %for.body
+
+for.inc16:
+  %1 = phi i32 [ %inc, %for.inc16 ], [ 0, %for.body ]
+  %inc = add i32 %1, 1
+  %add12 = add i32 %0, %1
+  br label %for.inc16
+}

[fhahn]

LGTM, thanks!