Skip to content

[LV] Check if the VF is scalar by VFRange in handleUncountableEarlyExit. #135294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 17, 2025
Merged

[LV] Check if the VF is scalar by VFRange in handleUncountableEarlyExit. #135294

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1b83ae5
[LV] Check if the VF is scalar by VFRange instead of VPlan in `handle…
ElvisWang123 Apr 11, 2025
b7eca73
!fixup, formatting.
ElvisWang123 Apr 11, 2025
9e96e1d
!fixup, add const.
ElvisWang123 Apr 11, 2025
01625c0
!fixup, clamp the range to make sure the plan has scalar/vector VF only.
ElvisWang123 Apr 11, 2025
65bf335
!fixup, address comment. Only clamp the range when extract is needed.
ElvisWang123 Apr 14, 2025
53c168b
!fixup, address comment.
ElvisWang123 Apr 14, 2025
ce30391
!fixup, address comments.
ElvisWang123 Apr 15, 2025
dcaed17
!fixup, update comments.
ElvisWang123 Apr 17, 2025
aa4395a
Merge branch 'main' into lv-NFC-ee-scalar-extract
ElvisWang123 Apr 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9744,7 +9744,7 @@ LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(VFRange &Range) {
Legal->getUncountableEarlyExitingBlock()) {
VPlanTransforms::runPass(VPlanTransforms::handleUncountableEarlyExit, *Plan,
*PSE.getSE(), OrigLoop, UncountableExitingBlock,
RecipeBuilder);
RecipeBuilder, Range);
}
DenseMap<VPValue *, VPValue *> IVEndValues;
addScalarResumePhis(RecipeBuilder, *Plan, IVEndValues);
Expand Down
10 changes: 8 additions & 2 deletions llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
#include "VPlanAnalysis.h"
#include "VPlanCFG.h"
#include "VPlanDominatorTree.h"
#include "VPlanHelpers.h"
#include "VPlanPatternMatch.h"
#include "VPlanUtils.h"
#include "VPlanVerifier.h"
Expand Down Expand Up @@ -2380,7 +2381,8 @@ void VPlanTransforms::convertToConcreteRecipes(VPlan &Plan) {

void VPlanTransforms::handleUncountableEarlyExit(
VPlan &Plan, ScalarEvolution &SE, Loop *OrigLoop,
BasicBlock *UncountableExitingBlock, VPRecipeBuilder &RecipeBuilder) {
BasicBlock *UncountableExitingBlock, VPRecipeBuilder &RecipeBuilder,
VFRange &Range) {
VPRegionBlock *LoopRegion = Plan.getVectorLoopRegion();
auto *LatchVPBB = cast<VPBasicBlock>(LoopRegion->getExiting());
VPBuilder Builder(LatchVPBB->getTerminator());
Expand Down Expand Up @@ -2435,8 +2437,12 @@ void VPlanTransforms::handleUncountableEarlyExit(
ExitIRI->addOperand(IncomingFromLatch);
ExitIRI->extractLastLaneOfOperand(MiddleBuilder);
}

auto IsVector = [](ElementCount VF) { return VF.isVector(); };
// Add the incoming value from the early exit.
if (!IncomingFromEarlyExit->isLiveIn() && !Plan.hasScalarVFOnly()) {
if (!IncomingFromEarlyExit->isLiveIn() &&
// Limit range to scalar VF only, if the range contains the scalar VF.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this say // Limit range to vector VF only ...? The code below only works if we're dealing with vectors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if using Limit range to vector VF only ... will be more confusing or not. Since it won't change the range when it only contains vector VFs and will limit the range to scalar VF when it contains scalar VF.

Perhaps using Limit range to scalar VF only and not insert extract, if the range contains the scalar VF is better?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code below if (.. & getDecisionAndClampRange(IsVector, Range)) only applies to vectors and so only applies if Range does not contain any scalar VF. Perhaps you can say that the entire transform is different for vector VFs, as it requires an additional extract element, and so the range needs clamping.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, thanks for advice!

LoopVectorizationPlanner::getDecisionAndClampRange(IsVector, Range)) {
VPValue *FirstActiveLane = EarlyExitB.createNaryOp(
VPInstruction::FirstActiveLane, {EarlyExitTakenCond}, nullptr,
"first.active.lane");
Expand Down
4 changes: 3 additions & 1 deletion llvm/lib/Transforms/Vectorize/VPlanTransforms.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ class PredicatedScalarEvolution;
class TargetLibraryInfo;
class VPBuilder;
class VPRecipeBuilder;
class VFRange;

extern cl::opt<bool> VerifyEachVPlan;

Expand Down Expand Up @@ -174,7 +175,8 @@ struct VPlanTransforms {
static void handleUncountableEarlyExit(VPlan &Plan, ScalarEvolution &SE,
Loop *OrigLoop,
BasicBlock *UncountableExitingBlock,
VPRecipeBuilder &RecipeBuilder);
VPRecipeBuilder &RecipeBuilder,
VFRange &Range);

/// Lower abstract recipes to concrete ones, that can be codegen'd.
static void convertToConcreteRecipes(VPlan &Plan);
Expand Down
Loading