Skip to content

fix(kv-router): route KvIndexer worker removals through the FIFO mutation queue - #16990

Merged
jthomson04 merged 10 commits into
ai-dynamo:mainfrom
TanNgocDo:fix/kv-indexer-remove-worker-fifo
Nov 12, 2026
Merged

jthomson04 merged 10 commits into
ai-dynamo:mainfrom
TanNgocDo:fix/kv-indexer-remove-worker-fifo

Conversation

@TanNgocDo

@TanNgocDo TanNgocDo commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #16988

  • KvIndexer::remove_worker and remove_worker_dp_rank used separate channels that the actor polled first (biased select!), and flush/dump drained them before the mutation queue. A removal could overtake KV events enqueued earlier, and the remaining Stored events then resurrected the removed worker with stale blocks (overcounted overlap in find_matches).
  • RemoveWorker and RemoveWorkerDpRank now go through the existing mutation_tx FIFO. The separate channels are gone, and ResetWorkerDpRank is merged into RemoveWorkerDpRank (acknowledged when resp is Some), so there is a single apply path for rank removal.
  • remove_worker_sender() is removed (no remaining callers). LocalKvIndexer::remove_worker now delegates directly to KvIndexer::remove_worker.
  • Impact: real impact is the standalone indexer service (Single backend unregister/deregister, selection ingress). LocalKvIndexer has no production caller of remove_worker today, so the worker -> router recovery path is not affected; the change is preventive there. Frontend Indexer::Single and CRTC already use the FIFO reset_worker_dp_rank_and_wait.
  • Trade-off: removals no longer have their own queue or priority, so remove_worker().await can wait behind already-queued events when mutation_tx is full.
  • Known gaps, out of scope here: routing decisions on routing_rx ([BUG]: KvIndexer routing decisions can be applied after remove_worker (approximate TTL mode) #18519) and the rest of an in-flight listener batch ([BUG]: Standalone indexer deregister removes a worker while its listener can still apply the rest of an in-flight batch #18518) can still be applied after a removal.

Files (after the crate split in #18055): lib/router/kv-indexer/src/kv_indexer.rs, lib/router/kv-indexer/src/local.rs, lib/router/kv-router/src/services/indexer/registry.rs.

Validation

  • New regression tests remove_worker_does_not_overtake_queued_stores and remove_worker_dp_rank_does_not_overtake_queued_stores (the latter also checks a sibling dp_rank is kept). They park the indexer thread on a KvIndexerDelegate gate, so they are deterministic: both fail 20/20 on main and pass 20/20 with this change.
  • cargo test -p dynamo-kv-indexer --lib --features testing: 606 passed, 2 ignored.
  • cargo test -p dynamo-kv-router --lib --features testing,http,ingest-zmq,peer-recovery services::indexer: 27 passed (includes registration_progresses_while_removal_is_backpressured, which now fills the queue through the testing-gated KvIndexer::saturate_mutation_queue_for_test).
  • cargo clippy --locked --no-deps -p dynamo-kv-indexer -p dynamo-kv-router --all-targets (with the features above) -- -D warnings and cargo fmt --all --check: clean.

Summary by CodeRabbit

  • Bug Fixes

    • Worker and rank removals now remain in the correct order with queued updates, preventing removed entries from reappearing.
    • Removing a worker or rank now consistently clears related index and cleanup state.
    • Preserved sibling rank data when removing an individual rank.
  • Refactor

    • Unified worker and rank removal processing for more reliable index behavior.
    • Improved worker-removal request handling, including capacity visibility and optional completion acknowledgement.

…tion queue

KvIndexer applied remove_worker and remove_worker_dp_rank through separate
channels that the actor polled first (biased select!), and flush/dump drained
them before the mutation queue. A removal could therefore overtake KV events
enqueued earlier, and the remaining Stored events would resurrect the removed
worker with stale blocks.

Send RemoveWorker and RemoveWorkerDpRank through the existing mutation_tx
FIFO, drop the separate channels, and merge ResetWorkerDpRank into
RemoveWorkerDpRank (acknowledged when resp is Some). RemoveWorkerSender now
enqueues on the same queue, so removals no longer have their own queue or
priority.

Add regression tests for remove_worker and remove_worker_dp_rank that fail on
the previous implementation.

Closes ai-dynamo#16988

Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@TanNgocDo
TanNgocDo requested a review from a team as a code owner October 21, 2026 07:30
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator October 21, 2026 07:30 — with GitHub Actions Inactive
@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator October 21, 2026 07:30 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi TanNgocDo! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor labels Oct 21, 2026
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

Worker and rank removals now share the FIFO mutation queue with KV events. The indexer applies removals through unified mutation handling, updating approximate-LRU, trie, and prune-manager state. Sender and interface APIs changed, and tests cover removal ordering and rank preservation.

Changes

Ordered indexer removals

Layer / File(s) Summary
Removal requests and sender API
lib/kv-router/src/indexer/kv_indexer.rs
MutationRequest now represents whole-worker removal and optionally acknowledged rank removal. RemoveWorkerSender queues worker removals on the shared mutation queue.
Unified mutation processing
lib/kv-router/src/indexer/kv_indexer.rs
The event loop and drain paths process removals through the unified mutation queue. Applying a removal updates approximate-LRU, trie, and prune-manager state.
Interface integration and ordering tests
lib/kv-router/src/indexer/kv_indexer.rs, lib/kv-router/src/indexer/local.rs
Interface removal methods enqueue mutations, and LocalKvIndexer awaits primary-indexer removal. Tests cover queued stores, worker and rank removal, and sibling-rank preservation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to d79ac

No concrete issue remains that would prevent merging after normal checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: routing worker removals through the FIFO mutation queue.
Linked Issues check Passed For [#16988], RemoveWorker and RemoveWorkerDpRank now use mutation_tx, the same FIFO queue as KV events. The actor and drain paths apply these mutations through one path. remove_worker_dp_rank…
Out of Scope Changes check Passed The changes stay within [#16988]. They remove the obsolete removal channels, add the queue-backed RemoveWorkerSender, update LocalKvIndexer delegation, and adjust the affected mutation paths. The …
Description check Passed The description follows the repository template closely enough to pass. It has a Summary that explains the problem (removals on separate channels could overtake earlier KV events, and stale Stored eve…
Full details: Linked Issues check

Explanation

For [#16988], RemoveWorker and RemoveWorkerDpRank now use mutation_tx, the same FIFO queue as KV events. The actor and drain paths apply these mutations through one path. remove_worker_dp_rank removes only the requested rank, while the added regression test verifies that sibling rank 1 remains. The added whole-worker regression test verifies that queued stores do not resurrect the removed worker. The reported test suites also pass.

Full details: Out of Scope Changes check

Explanation

The changes stay within [#16988]. They remove the obsolete removal channels, add the queue-backed RemoveWorkerSender, update LocalKvIndexer delegation, and adjust the affected mutation paths. The added tests directly validate the required ordering and rank behavior. No unrelated production behavior or unrelated files are included in the supplied change summary.

Full details: Description check

Explanation

The description follows the repository template closely enough to pass. It has a Summary that explains the problem (removals on separate channels could overtake earlier KV events, and stale Stored events then resurrected removed workers), the fix (removals routed through the mutation_tx FIFO, ResetWorkerDpRank merged into RemoveWorkerDpRank, remove_worker_sender removed), the production impact and the trade-off. It has a Validation section with named regression tests, pass counts for cargo test, and clippy and fmt results. It links the issue with 'Closes #16988'. The template's Overview and Issues and Ongoing Work headings are not used by name, and the PR checklist item for the /pr-checklist skill is not visible in the description. These are minor gaps. The description also says remove_worker_sender() is removed, but the raw summary says it now returns a RemoveWorkerSender wrapper. That mismatch should be checked, but it does not make the description incomplete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
ℹ️ Recent review info

⚙️ Run configuration

  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: d96d22cf-9e8d-4e36-bf74-cb4c0ce830f2
📥 Commits

Reviewing files that changed from the base of the PR and between bc2b624 and d79ac2f.

📒 Files selected for processing: 2

  • lib/kv-router/src/indexer/kv_indexer.rs
  • lib/kv-router/src/indexer/local.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator October 21, 2026 07:38 — with GitHub Actions Inactive
@TanNgocDo

TanNgocDo commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

@PeaBrane PTAL , this fixes KvIndexer removals overtaking queued KV events (#16988) . Can you please help to check ? Thanks

@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator October 21, 2026 08:26 — with GitHub Actions Inactive
Addresses the CodeRabbit docstring coverage warning on ai-dynamo#16990.

Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator October 21, 2026 08:28 — with GitHub Actions Inactive
@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator October 27, 2026 09:21 — with GitHub Actions Inactive
@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator November 5, 2026 03:36 — with GitHub Actions Inactive

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right to me. Putting removals on the same FIFO as KV events fixes the overtaking at its source, and I didn't find deadlock or hot-path problems with the shared queue: the actor never waits on its producers, and MutationRequest doesn't get bigger.

Two small, non-blocking notes inline. The new doc comment claims a bit more ordering than the queue provides, and the regression tests could be kept from passing without exercising the bug.

Comment thread lib/kv-router/src/indexer/kv_indexer.rs Resolved
Comment thread lib/kv-router/src/indexer/kv_indexer.rs Resolved
Comment thread lib/kv-router/src/indexer/kv_indexer.rs Resolved
- Scope the apply_mutation doc to KV events and removals, and note the
  routing_rx and in-flight listener batch gaps (TODO).
- Make the removal-ordering regression tests deterministic by parking the
  indexer thread on a delegate gate; they now fail on every run against main.
- Remove RemoveWorkerSender and remove_worker_sender(); the backpressure test
  fills the mutation queue through a cfg(test) accessor.

Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com>
@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator November 10, 2026 15:16 — with GitHub Actions Inactive
Resolve conflicts from the dynamo-kv-router crate split (ai-dynamo#18055): keep the
track_resident_blocks_rx arm, update test imports to crate::testing, and replace
the cross-crate mutation_tx_for_test accessor with a testing-gated
saturate_mutation_queue_for_test hook.

Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com>
@TanNgocDo
TanNgocDo temporarily deployed to external_collaborator November 12, 2026 09:21 — with GitHub Actions Inactive
- MutationRequest is private again; pub(crate) was only needed by the
  removed RemoveWorkerSender::new.
- The ordering NOTE/TODO on apply_mutation cites ai-dynamo#18518 and ai-dynamo#18519 and
  names WorkerRegistry::deregister.
- Drop Gate.armed from the test helper; entered.take() already fires
  exactly once.

No behaviour change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 temporarily deployed to external_collaborator November 12, 2026 17:32 — with GitHub Actions Inactive
@jthomson04

Copy link
Copy Markdown
Contributor

Pushed a small follow-up commit (7abcc6c) to this branch so it doesn't need another round. MutationRequest is private again (the pub(crate) was for RemoveWorkerSender::new), the ordering TODO now cites #18518 and #18519, and Gate.armed is gone from the test helper since entered.take() already fires once. No behaviour change.

@jthomson04

Copy link
Copy Markdown
Contributor

/ok to test 7abcc6c

@jthomson04
jthomson04 enabled auto-merge (squash) November 12, 2026 17:33
@jthomson04
jthomson04 merged commit 164358a into ai-dynamo:main Nov 12, 2026
121 checks passed

This branch was previously deployed

1 inactive deployment
external_collaborator — 7abcc6c7 Deployed Nov 12, 2026 by jthomson04 via ok-to-test #50551
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: KvIndexer remove_worker / remove_worker_dp_rank can overtake queued KV events

3 participants