Repository navigation
fix(kv-router): route KvIndexer worker removals through the FIFO mutation queue - #16990
Conversation
…tion queue KvIndexer applied remove_worker and remove_worker_dp_rank through separate channels that the actor polled first (biased select!), and flush/dump drained them before the mutation queue. A removal could therefore overtake KV events enqueued earlier, and the remaining Stored events would resurrect the removed worker with stale blocks. Send RemoveWorker and RemoveWorkerDpRank through the existing mutation_tx FIFO, drop the separate channels, and merge ResetWorkerDpRank into RemoveWorkerDpRank (acknowledged when resp is Some). RemoveWorkerSender now enqueues on the same queue, so removals no longer have their own queue or priority. Add regression tests for remove_worker and remove_worker_dp_rank that fail on the previous implementation. Closes ai-dynamo#16988 Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
👋 Hi TanNgocDo! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
Addresses the CodeRabbit docstring coverage warning on ai-dynamo#16990. Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
PeaBrane
left a comment
There was a problem hiding this comment.
Looks right to me. Putting removals on the same FIFO as KV events fixes the overtaking at its source, and I didn't find deadlock or hot-path problems with the shared queue: the actor never waits on its producers, and MutationRequest doesn't get bigger.
Two small, non-blocking notes inline. The new doc comment claims a bit more ordering than the queue provides, and the regression tests could be kept from passing without exercising the bug.
- Scope the apply_mutation doc to KV events and removals, and note the routing_rx and in-flight listener batch gaps (TODO). - Make the removal-ordering regression tests deterministic by parking the indexer thread on a delegate gate; they now fail on every run against main. - Remove RemoveWorkerSender and remove_worker_sender(); the backpressure test fills the mutation queue through a cfg(test) accessor. Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com>
Resolve conflicts from the dynamo-kv-router crate split (ai-dynamo#18055): keep the track_resident_blocks_rx arm, update test imports to crate::testing, and replace the cross-crate mutation_tx_for_test accessor with a testing-gated saturate_mutation_queue_for_test hook. Signed-off-by: Tan Do <darkknightkhtn2008@gmail.com>
- MutationRequest is private again; pub(crate) was only needed by the removed RemoveWorkerSender::new. - The ordering NOTE/TODO on apply_mutation cites ai-dynamo#18518 and ai-dynamo#18519 and names WorkerRegistry::deregister. - Drop Gate.armed from the test helper; entered.take() already fires exactly once. No behaviour change. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
Pushed a small follow-up commit (7abcc6c) to this branch so it doesn't need another round. |
|
/ok to test 7abcc6c |
Summary
Closes #16988
KvIndexer::remove_workerandremove_worker_dp_rankused separate channels that the actor polled first (biasedselect!), andflush/dumpdrained them before the mutation queue. A removal could overtake KV events enqueued earlier, and the remainingStoredevents then resurrected the removed worker with stale blocks (overcounted overlap infind_matches).RemoveWorkerandRemoveWorkerDpRanknow go through the existingmutation_txFIFO. The separate channels are gone, andResetWorkerDpRankis merged intoRemoveWorkerDpRank(acknowledged whenrespisSome), so there is a single apply path for rank removal.remove_worker_sender()is removed (no remaining callers).LocalKvIndexer::remove_workernow delegates directly toKvIndexer::remove_worker.Singlebackend unregister/deregister, selection ingress).LocalKvIndexerhas no production caller ofremove_workertoday, so the worker -> router recovery path is not affected; the change is preventive there. FrontendIndexer::Singleand CRTC already use the FIFOreset_worker_dp_rank_and_wait.remove_worker().awaitcan wait behind already-queued events whenmutation_txis full.routing_rx([BUG]: KvIndexer routing decisions can be applied after remove_worker (approximate TTL mode) #18519) and the rest of an in-flight listener batch ([BUG]: Standalone indexer deregister removes a worker while its listener can still apply the rest of an in-flight batch #18518) can still be applied after a removal.Files (after the crate split in #18055):
lib/router/kv-indexer/src/kv_indexer.rs,lib/router/kv-indexer/src/local.rs,lib/router/kv-router/src/services/indexer/registry.rs.Validation
remove_worker_does_not_overtake_queued_storesandremove_worker_dp_rank_does_not_overtake_queued_stores(the latter also checks a sibling dp_rank is kept). They park the indexer thread on aKvIndexerDelegategate, so they are deterministic: both fail 20/20 onmainand pass 20/20 with this change.cargo test -p dynamo-kv-indexer --lib --features testing: 606 passed, 2 ignored.cargo test -p dynamo-kv-router --lib --features testing,http,ingest-zmq,peer-recovery services::indexer: 27 passed (includesregistration_progresses_while_removal_is_backpressured, which now fills the queue through thetesting-gatedKvIndexer::saturate_mutation_queue_for_test).cargo clippy --locked --no-deps -p dynamo-kv-indexer -p dynamo-kv-router --all-targets(with the features above)-- -D warningsandcargo fmt --all --check: clean.Summary by CodeRabbit
Bug Fixes
Refactor