Skip to content

fix CRR queue progression after deletion#2485

Open
Jayant-kernel wants to merge 2 commits into
openkruise:masterfrom
Jayant-kernel:codex/fix-crr-delete-requeue
Open

fix CRR queue progression after deletion#2485
Jayant-kernel wants to merge 2 commits into
openkruise:masterfrom
Jayant-kernel:codex/fix-crr-delete-requeue

Conversation

@Jayant-kernel

Copy link
Copy Markdown

Ⅰ. Describe what this PR does

This PR prevents a ContainerRecreateRequest (CRR) from remaining Pending when it is created immediately after a completed CRR for the same Pod is deleted.

The kruise-daemon intentionally serializes CRRs by using <namespace>/<podName> as its workqueue key. A race was possible when a second CRR was created while the daemon informer still contained the first CRR:

  1. The second CRR's add event queued the Pod.
  2. The daemon's stale informer view still selected the first CRR, leaving the second CRR Pending.
  3. When the first CRR was deleted or lost its active label after completion, the filtered informer's delete handler only cleared its resource-version expectation.
  4. Because the informer has no periodic resync, no event queued the Pod again and the second CRR remained Pending indefinitely.

The delete handler now:

  • clears the deleted CRR's resource-version expectation;
  • requeues its Pod so the next pending CRR can progress; and
  • handles DeletedFinalStateUnknown tombstones when the original delete notification was missed.

Unit coverage verifies that both normal delete events and tombstone delete events enqueue the expected Pod key. The existing one-at-a-time execution semantics for CRRs targeting the same Pod are preserved.

Ⅱ. Does this pull request fix one issue?

Fixes #2474

Ⅲ. Describe how to verify it

The following checks pass locally with Go 1.23.8:

go test -count=1 ./pkg/daemon/containerrecreate
go test -run TestEnqueueAfterDelete -count=100 ./pkg/daemon/containerrecreate
go vet ./pkg/daemon/containerrecreate
git diff --check

To verify the original scenario in a cluster:

  1. Create a Pod containing two containers.
  2. Create a CRR for the first container and wait for it to complete.
  3. Delete that CRR and immediately create another CRR for the second container without sleeping.
  4. Confirm that the second CRR advances from Pending to Recreating and then Completed.

Ⅳ. Special notes for reviews

This change deliberately keeps the Pod-scoped queue key instead of allowing CRRs for one Pod to execute concurrently. It only restores the missing wake-up when the previous active CRR leaves the filtered informer.

go test ./pkg/daemon/... was also attempted. The modified containerrecreate package and the other daemon packages passed, except the unrelated existing pkg/daemon/criruntime/imageruntime test, which requires a credential-provider plugin binary that is not present in the local environment. A live cluster e2e run was not available locally.

Signed-off-by: Jayant <212013719+Jayant-kernel@users.noreply.github.com>
@kruise-bot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign fillzpp for approval by writing /assign @fillzpp in a comment. For more information see:The Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kruise-bot kruise-bot added the size/L size/L: 100-499 label Jun 20, 2026
@kruise-bot

Copy link
Copy Markdown

Welcome @Jayant-kernel! It looks like this is your first PR to openkruise/kruise 🎉

@Jayant-kernel
Jayant-kernel marked this pull request as ready for review June 20, 2026 17:28
Copilot AI review requested due to automatic review settings June 20, 2026 17:28
@kruise-bot
kruise-bot requested review from furykerry and zmberg June 20, 2026 17:28
@Jayant-kernel Jayant-kernel changed the title [codex] fix CRR queue progression after deletion fix CRR queue progression after deletion Jun 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a race in the kruise-daemon ContainerRecreateRequest (CRR) controller where a subsequent CRR for the same Pod could remain stuck in Pending after the previous active CRR leaves the filtered informer (delete / active-label removal), by ensuring the Pod-scoped queue key is re-enqueued on delete events (including tombstones).

Changes:

  • Route CRR delete events through a new enqueueAfterDelete helper that clears resource-version expectations and requeues the Pod key.
  • Add tombstone (cache.DeletedFinalStateUnknown) handling to cover missed delete notifications.
  • Add a unit test validating that both normal delete and tombstone delete enqueue the expected Pod-scoped key.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pkg/daemon/containerrecreate/crr_daemon_controller.go Re-enqueues the Pod workqueue key on CRR delete/tombstone to allow the next pending CRR to progress.
pkg/daemon/containerrecreate/crr_daemon_controller_test.go Adds unit coverage asserting delete and tombstone events enqueue the correct Pod-scoped queue key.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov

codecov Bot commented Jun 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 49.95%. Comparing base (7ac0414) to head (c128cf9).
⚠️ Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
.../daemon/containerrecreate/crr_daemon_controller.go 92.85% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2485      +/-   ##
==========================================
+ Coverage   49.93%   49.95%   +0.01%     
==========================================
  Files         325      325              
  Lines       28435    28449      +14     
==========================================
+ Hits        14200    14212      +12     
- Misses      12581    12582       +1     
- Partials     1654     1655       +1     
Flag Coverage Δ
unittests 49.95% <92.85%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Jayant <212013719+Jayant-kernel@users.noreply.github.com>
@Jayant-kernel

Copy link
Copy Markdown
Author

@furykerry review my pr

@a932846905

Copy link
Copy Markdown
Contributor

/assign @FillZpp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L size/L: 100-499

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ContainerRecreateRequest stuck in Pending when recreating different containers in the same Pod continuously

5 participants