region-cache: leader-read keeps retrying same store on `ServerIsBusy` without `EstimatedWaitMs` (stale leader cache)

[AndreMouche]

Description

For leader reads, when TiKV returns ServerIsBusy without EstimatedWaitMs (e.g. unified read pool full / MaxPendingTasksExceeded), replicaSelectorV2 does not switch to other replicas and does not invalidate the region cache. It only marks the store slow, backs off (BoTiKVServerBusy), and retries the same cached leader.
This becomes a severe availability gap when the cached leader store is half-dead:

• Raft / PD has already transferred leaders away from that store
• The old requests on this store were stuck at async_snapshot forever(Running tasks in unified read pool continue to grow while CPU usage is very low tikv#18491) which will lead to increasing running tasks(never done and will not timeout)
• When the new request comes, the store can still answer gRPC and return ServerIsBusy before Raft-level NotLeader checks
• TiDB keeps hammering the stale leader until the store becomes unreachable
  

  client-go/internal/locate/replica_selector.go

  Lines 483 to 526 in 4d50d67

  	func (s *replicaSelectorV2) onServerIsBusy(
  	bo *retry.Backoffer, ctx *RPCContext, req *tikvrpc.Request, serverIsBusy *errorpb.ServerIsBusy,
  	) (shouldRetry bool, err error) {
  	var store *Store
  	if ctx != nil && ctx.Store != nil {
  	store = ctx.Store
  	if serverIsBusy.EstimatedWaitMs != 0 {
  	ctx.Store.updateServerLoadStats(serverIsBusy.EstimatedWaitMs)
  	if s.busyThreshold != 0 && isReadReq(req.Type) {
  	// do not retry with batched coprocessor requests.
  	// it'll be region misses if we send the tasks to replica.
  	if req.Type == tikvrpc.CmdCop && len(req.Cop().Tasks) > 0 {
  	return false, nil
  	}
  	if s.target != nil {
  	s.target.serverIsBusy = true
  	}
  	}
  	} else {
  	// Mark the server is busy (the next incoming READs could be redirected to expected followers.)
  	ctx.Store.healthStatus.markAlreadySlow()
  	}
  	}
  	backoffErr := errors.Errorf("server is busy, ctx: %v", ctx)
  	if s.canFastRetry() {
  	s.addPendingBackoff(store, retry.BoTiKVServerBusy, backoffErr)
  	return true, nil
  	}
  	err = bo.Backoff(retry.BoTiKVServerBusy, backoffErr)
  	if err != nil {
  	return false, err
  	}
  	return true, nil
  	}
  	func (s *replicaSelectorV2) canFastRetry() bool {
  	if s.replicaReadType == kv.ReplicaReadLeader {
  	leaderIdx := s.region.getStore().workTiKVIdx
  	leader := s.replicas[leaderIdx]
  	if isLeaderCandidate(leader) && !leader.serverIsBusy {
  	return false
  	}
  	}
  	return true

metrics also showed: after leader count on the store dropped to 0, unified read-pool running tasks stayed high with ~0 CPU, while gRPC was still alive — consistent with returning ServerIsBusy instead of NotLeader.

Expected behavior

When a leader-read hits ServerIsBusy (especially with EstimatedWaitMs == 0) repeatedly on the cached leader, client-go should do one of:

• Invalidate / refresh region cache (so a later attempt can discover the new leader), or
• Fail over to other replicas (follower read / reload leader from PD), instead of backoff+retrying the same peer only.

Impact

Leader eviction / slow-store protection on the PD/Raft side cannot fully protect TiDB latency if the half-dead store keeps answering ServerIsBusy and the client never updates its leader cache.

workaround:

stop the half-dead tikv instance

[LykxSassinator]

Could you confirm the user's version? It seems that the replicaSelectorV2 has already been deprecated and removed in #1265.