schedule: add affinity checker #10040

lhy1024 · 2025-12-10T10:46:05Z

What problem does this PR solve?

Issue Number: Ref #9764

What is changed and how does it work?

Check List

Tests

Unit test

Release note

None.

ti-chi-bot · 2025-12-10T10:46:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

codecov · 2025-12-10T10:55:27Z

Codecov Report

❌ Patch coverage is 82.44898% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.58%. Comparing base (09f1b5d) to head (9e19804).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10040      +/-   ##
==========================================
+ Coverage   78.51%   78.58%   +0.07%     
==========================================
  Files         513      514       +1     
  Lines       68893    69108     +215     
==========================================
+ Hits        54091    54309     +218     
- Misses      10887    10889       +2     
+ Partials     3915     3910       -5

Flag	Coverage Δ
unittests	`78.58% <82.44%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lhy1024 · 2025-12-11T10:31:06Z

/retest

lhy1024 · 2025-12-11T14:33:22Z

/retest

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2025-12-12T03:26:01Z

/retest

rleungx · 2025-12-12T11:59:48Z

pkg/schedule/checker/affinity_checker.go

+		affinityCheckerAbnormalReplicaCounter.Inc()
+		return nil
+	}
+	if filter.HasWitnessPeers(region) {


We can remove it?

If we confirm that no clusters are using witness, I believe it can be removed.
BTW, should we create an issue to mark when the witness is to be completely removed?

yes, we can

pkg/schedule/checker/affinity_checker.go

rleungx · 2025-12-12T12:07:15Z

pkg/schedule/checker/affinity_checker.go

+
+	options := make([]core.RegionCreateOption, 0, len(sourceOnly)+1)
+	for i, sourceStoreID := range sourceOnly {
+		options = append(options, core.WithReplacePeerStore(sourceStoreID, targetOnly[i]))


Does the targetOnly always have the same length with sourceOnly? BTW, targetOnly and sourceOnly are confused.

How about storesToRemove and storesToAdd?

We have check len(sourceVoters) and len(voterStoreIDs), they are removed the same stores.

rleungx · 2025-12-12T12:08:06Z

pkg/schedule/checker/affinity_checker.go

+		// so expire the group first, then provide the available Region information and fetch the Group state again.
+		if !isAffinity {
+			targetRegion := cloneRegionWithReplacePeerStores(region, group.LeaderStoreID, group.VoterStoreIDs...)
+			if targetRegion == nil || !filter.IsRegionReplicated(c.cluster, targetRegion) {


When will the targetRegion not RegionReplicated?

placement rules changed

pkg/schedule/checker/affinity_checker.go

Signed-off-by: lhy1024 <admin@liudos.us>

bufferflies · 2025-12-12T02:35:11Z

pkg/core/region_option.go

+// WithLeaderStore sets the voter peer on leaderStoreID as the leader.
+func WithLeaderStore(leaderStoreID uint64) RegionCreateOption {
+	return func(region *RegionInfo) {
+		for _, p := range region.GetPeers() {


It should iterate over voters, not all peers.

No, RegionInfo.Clone will create &RegionInfo firstly, then apply opts(contains WithLeaderStore), then set voters(call classifyVoterAndLearner(region)). So voters is nil in opts.

So we iterate peers and check whether is learner.

bufferflies · 2025-12-12T12:35:04Z

pkg/schedule/checker/affinity_checker.go

+		affinityCheckerAbnormalReplicaCounter.Inc()
+		return nil
+	}
+	if filter.HasWitnessPeers(region) {


yes, we can

bufferflies · 2025-12-12T12:49:53Z

pkg/schedule/checker/affinity_checker.go

+	// Preserve all existing learner peers
+	// Since regions that violate placement rules won't reach this function,
+	// we can safely assume all existing learners should be preserved
+	for _, learner := range region.GetLearners() {


Why do we need to change learner

We don't change learners. This code preserves all existing learner peers (e.g., TiFlash replicas) by including them in the target roles map.

If a learner's storeID conflicts with an affinity voter, we abort the operator creation (return nil) instead of modifying the learner

TestAffinityCheckerPreserveLearners keeps all learners
TestAffinityCheckerLearnerVoterConflict tests the case that learner and leader are conflict

bufferflies · 2025-12-12T12:50:28Z

pkg/schedule/checker/affinity_checker.go

+	).SetPeers(peers).SetExpectedRoles(roles)
+
+	// Skip building if target leader store currently disallows leader in (e.g., evict-leader / reject-leader).
+	if targetLeader := c.cluster.GetStore(group.LeaderStoreID); targetLeader != nil {


this check should place before the build

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2025-12-12T13:42:14Z

/retest

rleungx · 2025-12-15T05:29:45Z

pkg/schedule/checker/checker_controller.go

+			ops := c.affinityChecker.Check(region)
+			if len(ops) > 0 {
+				opKind := ops[0].Kind()
+				if (opKind & operator.OpMerge) == 0 {


So there is no way to control the speed independently?

It is not neccessary to introduce a new limit for affinity scheduling

What if it affects the performance?

Generally speaking, once it enters the stable state, no further scheduling occurs, and we can restrict it using region-schedule-limit and merge-schedule-limit.

Simultaneously, we can disable affinity scheduling to prevent it from affecting the entire cluster.

I prefer to decouple these parameters.

after discussion, we use a affinity-schedule-limit

pkg/schedule/checker/affinity_checker.go

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2025-12-15T10:49:40Z

/retest

lhy1024 · 2025-12-15T11:21:58Z

/retest

lhy1024 · 2025-12-15T11:44:10Z

/retest

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot · 2025-12-16T02:26:23Z

[LGTM Timeline notifier]

Timeline:

2025-12-15 03:01:52.342699513 +0000 UTC m=+1442057.156477086: ☑️ agreed by bufferflies.
2025-12-16 02:26:22.790132081 +0000 UTC m=+1526327.603909653: ☑️ agreed by rleungx.

lhy1024 · 2025-12-16T02:56:01Z

/retest

lhy1024 · 2025-12-16T05:13:39Z

@niubell PTAL

ti-chi-bot · 2025-12-16T08:42:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, niubell, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [bufferflies,rleungx]
~~pkg/mcs/scheduling/server/config/OWNERS~~ [niubell]
~~pkg/schedule/config/OWNERS~~ [niubell]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-12-16T08:46:39Z

In response to a cherrypick label: new pull request created to branch release-8.5: #10074.
But this PR has conflicts, please resolve them!

ref tikv#9764 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ref tikv#9764 Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 mentioned this pull request Dec 10, 2025

Support data affinity schedule for performance optimization #9764

Open

28 tasks

lhy1024 force-pushed the affinity-pr4 branch 2 times, most recently from 731212b to a699d6e Compare December 11, 2025 10:04

lhy1024 marked this pull request as ready for review December 11, 2025 10:04

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2025

lhy1024 force-pushed the affinity-pr4 branch from a699d6e to 47dff6e Compare December 11, 2025 14:24

schedule: add affinity checker

868b8ac

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the affinity-pr4 branch 2 times, most recently from dae7ba8 to 868b8ac Compare December 12, 2025 02:43

fix WithLeaderStore

578f3d9

Signed-off-by: lhy1024 <admin@liudos.us>

rleungx reviewed Dec 12, 2025

View reviewed changes

lhy1024 added 2 commits December 12, 2025 20:42

address comments

e873a77

Signed-off-by: lhy1024 <admin@liudos.us>

fix lint

6d3c53d

Signed-off-by: lhy1024 <admin@liudos.us>

bufferflies reviewed Dec 12, 2025

View reviewed changes

lhy1024 added 2 commits December 12, 2025 20:59

reuse CreateMoveRegionOperator

5ef7c53

Signed-off-by: lhy1024 <admin@liudos.us>

add learner check

9987eb7

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 requested review from bufferflies and rleungx December 12, 2025 13:26

fix lint

a0ad750

Signed-off-by: lhy1024 <admin@liudos.us>

rleungx reviewed Dec 15, 2025

View reviewed changes

pkg/schedule/checker/affinity_checker.go Show resolved Hide resolved

rleungx reviewed Dec 15, 2025

View reviewed changes

pkg/schedule/checker/affinity_checker.go Show resolved Hide resolved

address comments: add limit and rename

17d0de8

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot bot removed the approved label Dec 15, 2025

lhy1024 added 2 commits December 15, 2025 18:19

Merge branch 'master' of github.com:tikv/pd into affinity-pr4

a8b262a

Signed-off-by: lhy1024 <admin@liudos.us>

fix lint

ce85874

Signed-off-by: lhy1024 <admin@liudos.us>

remove limit check

9e19804

Signed-off-by: lhy1024 <admin@liudos.us>

rleungx approved these changes Dec 16, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 16, 2025

niubell approved these changes Dec 16, 2025

View reviewed changes

ti-chi-bot bot added the approved label Dec 16, 2025

ti-chi-bot bot merged commit fdc1cdf into tikv:master Dec 16, 2025
40 of 42 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Dec 16, 2025

This is an automated cherry-pick of tikv#10040

3175488

ref tikv#9764 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 16, 2025

schedule: add affinity checker (#10040) #10074

Closed

lhy1024 added a commit to lhy1024/pd that referenced this pull request Dec 16, 2025

schedule: add affinity checker (tikv#10040)

f091462

ref tikv#9764 Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 added a commit to lhy1024/pd that referenced this pull request Dec 16, 2025

schedule: add affinity checker (tikv#10040)

8b81448

ref tikv#9764 Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 added a commit to lhy1024/pd that referenced this pull request Dec 16, 2025

schedule: add affinity checker (tikv#10040)

b9da740

ref tikv#9764 Signed-off-by: lhy1024 <admin@liudos.us>

HunDunDM pushed a commit to HunDunDM/pd that referenced this pull request Dec 16, 2025

schedule: add affinity checker (tikv#10040)

db2c8b1

ref tikv#9764 Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 added a commit to lhy1024/pd that referenced this pull request Dec 16, 2025

schedule: add affinity checker (tikv#10040)

0c594a3

ref tikv#9764 Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 mentioned this pull request Dec 17, 2025

cherrypick data affinity feature to release-8.5 #10061

Merged

schedule: add affinity checker #10040

schedule: add affinity checker #10040

Uh oh!

Conversation

lhy1024 commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot bot commented Dec 10, 2025

Uh oh!

codecov bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lhy1024 commented Dec 11, 2025

Uh oh!

lhy1024 commented Dec 11, 2025

Uh oh!

lhy1024 commented Dec 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhy1024 Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhy1024 commented Dec 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lhy1024 commented Dec 15, 2025

Uh oh!

lhy1024 commented Dec 15, 2025

lhy1024 commented Dec 10, 2025 •

edited

Loading

codecov bot commented Dec 10, 2025 •

edited

Loading

lhy1024 Dec 12, 2025 •

edited

Loading