Repository navigation
test(router): assert routed worker for cache-salt isolation repeats - #18007
Conversation
The unpinned tenant-a and tenant-b repeats in _test_router_cache_salt_isolation only checked cache scores, which the seeded requests had already satisfied. Read the routed worker from the response, as _test_router_lora_isolation does, and assert each repeat reaches its seeded worker before the score checks. Refs DYN-5300, follows ai-dynamo#15746. Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
|
👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
factory: Initial validation snapshot. Validation status: complete
Main gap: the only committed caller, Detailed evidence — validation completeLeft out of the table because CI repeats them: Check 1 (builds the changed Dynamo source and imports its compiled extension) and Check 2 (fast lint and formatting). Evidence tally: [3/3 validated] Advisory AI assessment: sound. This is an automated, advisory judgment, not an approval. Validation result: complete — pass — the changed helper ran on a real two-worker vLLM pair and passed, it failed on the destination line when the expected worker was swapped or a repeat was routed wrongly, and lint, collection, and the build check passed. Evidence audit: complete [3/3 validated] — the command report comes from recorded runs. Commands and results [3/3 validated]Generated from the commands recorded during this run. Check 1Builds the changed Dynamo source and confirms that Python can import its compiled extension. Result: Passed ( Command: Not shown because the exact command contained private run data. Check 2Checks the changed files with the repository's fast lint and formatting commands. Result: Passed ( Command: Not shown because the exact command contained private run data. Check 3Runs the relevant GPU-dependent Python tests against the changed source. Result: Passed ( Command: Not shown because the exact command contained private run data. |
|
No description provided. |
|
factory: @dynamo-ops please run full CI for 44b0e51 |
|
/ok to test 44b0e51 |
|
No actionable comments were generated in the recent review. WalkthroughThe router cache salt isolation test now captures decode worker IDs from streamed routing data and checks that repeated unpinned requests from each tenant reach the expected worker. ChangesRouter cache salt isolation
Estimated code review effort: 2 (Simple) | ~8 minutes Merge Risk:
|
✅ Dynamo PR CI passed — run 41934815499 (attempt 1) on
|
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for 84e1af2 |
|
/ok to test 84e1af2 |
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for 71216db |
|
/ok to test 71216db |
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for 9d5bb62 |
|
/ok to test 9d5bb62 |
Absorbs 19 upstream commits (c5680e5..3a8226a). One conflict, the root Cargo.toml: kept upstream's dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 and dynamo-protocols 8.5.0 bumps together with the stack's workspace members and path crates; the four lockfiles were regenerated with cargo metadata (no drift from the auto-merge). - ai-dynamo#17923 perf(kv-router) no block-lookup growth for existing-key updates: lands in lib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs (rename-detected; the stack copy differs from upstream only by the crate::prelude import). - ai-dynamo#17741 build(deps) dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0, tokenizers 0.23.2 in lib/llm and lib/bench: root Cargo.toml conflict resolved as above; lib/llm/tests/tokenizer_cache.rs. - ai-dynamo#17520 feat(vllm) realtime text tools, held-request prefill, chat completions and sampling options: Python vllm, dynamo-protocols 8.5.0 bump. - ai-dynamo#18380 perf(bindings) direct token_ids/extra_args reads in RoutedEngine.generate and ai-dynamo#18448 refactor(bindings) py_convert: new lib/bindings/python/rust/llm/py_convert.rs, routed_engine.rs, engine.rs, _convert_preprocessed_request registered in lib.rs, _core.pyi, tests/test_preprocessed_request_conversion.py; llm/kv.rs extract_token_ids now delegates to py_convert::read_exact_u32_list with the stack's owner-crate imports kept. - ai-dynamo#18454 fix(frontend) named passthrough fields to Python chat processors: lib/llm protocols/openai/chat_completions.rs, Python sglang and vllm processors. - ai-dynamo#18253 fix(frontend) 16-bit image narrowing and EXIF opt-in: lib/llm preprocessor/media decoders, image_reader, loader. - ai-dynamo#18185 fix(frontend) rendering context in speculative prefill: lib/llm preprocessor/speculative_prefill.rs. - ai-dynamo#18428 fix(frontend) image step count without accounting: lib/llm protocols/openai/images aggregator.rs and nvext.rs. - ai-dynamo#17656 fix(frontend) recipe-based DeepSeek encoding with SGLang: Python frontend sglang_deepseek.py, sglang_prompt_renderer.py, sglang_prepost.py, sglang_processor.py. - ai-dynamo#18378 fix(runtime) TCP_NODELAY on accepted request-plane connections: lib/runtime pipeline/network/ingress/shared_tcp_endpoint.rs. - ai-dynamo#18007 test(router) routed-worker assertion for cache-salt isolation repeats: tests/router/common.py and helper.py. - ai-dynamo#18452 feat(backend) embedded engines skip Dynamo logging setup, ai-dynamo#18170 feat(trtllm) per-request metrics opt-in, ai-dynamo#18135 and ai-dynamo#16164 fix(vllm), ai-dynamo#16879 and ai-dynamo#18437 fix(recipes), c5680e5 docs community events: Python, recipes, docs; all auto-merged. No change to the router kernel, admission, worker-selection, slot-tracker, session-affinity, kv-events or kv-hints crates; no [features] table or Dynamo path-crate dependency edge changed. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Overview
The cache-salt isolation router test seeds one tenant on each of two workers. It then sends one unpinned repeat per tenant through the KV router. After the repeats, it only checked cache scores. The seeded requests had already met those scores. So a repeat routed to the wrong worker could still pass if its KV events arrived after the check. The LoRA isolation test in the same file already asserts the routed worker, and #15746 named this test as having the same gap.
Key Changes
The cache-salt and LoRA scenarios share
read_routed_worker_from_streamintests/router/helper.py. Each scenario still builds its own request. Onlytests/router/common.pyandtests/router/helper.pychange. No router or engine code changes.Validation
Full CI ran on
44b0e5163e. The only committed caller,tests/router/test_router_e2e_with_sglang.py::test_router_cache_salt_isolation_sglang[tcp], passed insglang-runtime / Test cuda13.0, amd64(73 s).Before that, a temporary, uncommitted test ran the helper with two vLLM workers serving TinyLlama on one A100. It passed. It failed on the new destination assertion when the expected worker was swapped, and when a tenant-a repeat was sent to the wrong worker.
Initial CI at
44b0e5163efailed in two planner tests that still passed--router-mode power-of-two. The same failures occurred onmain(job). Later base merges included the upstream fix, and the planner tests passed at84e1af2eff. That head failed the final gate afterdynamo-runtime / mypyexceeded its job time limit (job).Full CI passed on the previous head,
71216dbda91198a54b801a11d0abcabcb04ef746(run), after mergingmainto rerun CI. All seven required checks passed.Full CI passed on the shared response reader change at
9d5bb628510c2a1a8ff5f5bfcd18778e723a2b34(run). All seven required checks passed. The SGLang amd64 test job passed; the cache-salt scenario runs before merge, while the LoRA scenario runs after merge.Issues and Ongoing Work
Relates to #15746.
Closes DYN-5300.
Summary by CodeRabbit