Skip to content

test(router): assert routed worker for cache-salt isolation repeats - #18007

Merged
MatejKosec merged 5 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5300-assert-routed-worker-cache-salt-isolation-96244af7268b
Nov 9, 2026
Merged

MatejKosec merged 5 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5300-assert-routed-worker-cache-salt-isolation-96244af7268b

Conversation

@glamr-agent

@glamr-agent glamr-agent commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

The cache-salt isolation router test seeds one tenant on each of two workers. It then sends one unpinned repeat per tenant through the KV router. After the repeats, it only checked cache scores. The seeded requests had already met those scores. So a repeat routed to the wrong worker could still pass if its KV events arrived after the check. The LoRA isolation test in the same file already asserts the routed worker, and #15746 named this test as having the same gap.

Key Changes

  • The test's request helper now reads the worker the router chose from the response stream and returns it. It fails when no worker is reported.
  • Each unpinned repeat now asserts that it reached its tenant's seeded worker. The score checks after the repeats stay.

The cache-salt and LoRA scenarios share read_routed_worker_from_stream in tests/router/helper.py. Each scenario still builds its own request. Only tests/router/common.py and tests/router/helper.py change. No router or engine code changes.

Validation

  • Full CI ran on 44b0e5163e. The only committed caller, tests/router/test_router_e2e_with_sglang.py::test_router_cache_salt_isolation_sglang[tcp], passed in sglang-runtime / Test cuda13.0, amd64 (73 s).

  • Before that, a temporary, uncommitted test ran the helper with two vLLM workers serving TinyLlama on one A100. It passed. It failed on the new destination assertion when the expected worker was swapped, and when a tenant-a repeat was sent to the wrong worker.

  • Initial CI at 44b0e5163e failed in two planner tests that still passed --router-mode power-of-two. The same failures occurred on main (job). Later base merges included the upstream fix, and the planner tests passed at 84e1af2eff. That head failed the final gate after dynamo-runtime / mypy exceeded its job time limit (job).

  • Full CI passed on the previous head, 71216dbda91198a54b801a11d0abcabcb04ef746 (run), after merging main to rerun CI. All seven required checks passed.

  • Full CI passed on the shared response reader change at 9d5bb628510c2a1a8ff5f5bfcd18778e723a2b34 (run). All seven required checks passed. The SGLang amd64 test job passed; the cache-salt scenario runs before merge, while the LoRA scenario runs after merge.

Issues and Ongoing Work

Relates to #15746.

Closes DYN-5300.

Summary by CodeRabbit

  • Tests
    • Improved router cache-salt isolation checks to verify that repeated requests for each tenant route to the expected worker and complete with a reported worker ID.

The unpinned tenant-a and tenant-b repeats in
_test_router_cache_salt_isolation only checked cache scores, which the
seeded requests had already satisfied. Read the routed worker from the
response, as _test_router_lora_isolation does, and assert each repeat
reaches its seeded worker before the score checks.

Refs DYN-5300, follows ai-dynamo#15746.

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent requested a review from a team as a code owner November 4, 2026 04:18
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 4, 2026 04:18 — with GitHub Actions Inactive
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 4, 2026 04:18 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added test external-contribution Pull request is from an external contributor labels Nov 4, 2026
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: Initial validation snapshot.

Validation status: complete

Check Result Scope or reason
Runs the relevant GPU-dependent Python tests against the changed source. Passed (exit 0) Ran the changed helper through a temporary, uncommitted vLLM test (two dynamo.vllm workers, TinyLlama/TinyLlama-1.1B-Chat-v1.0, one A100-SXM4-80GB, vLLM 0.32.0) at commit 34453b2490. That commit differs from the PR head only in a code comment. Baseline on main passed; the change passed; swapping the expected worker and routing a tenant-a repeat to the wrong worker both failed on the new destination assertion.

Main gap: the only committed caller, test_router_cache_salt_isolation_sglang, did not run because SGLang is not installed in the test environment (ModuleNotFoundError: No module named 'sglang'). The pre-merge SGLang router GPU job runs it.

Detailed evidence — validation complete

Left out of the table because CI repeats them: Check 1 (builds the changed Dynamo source and imports its compiled extension) and Check 2 (fast lint and formatting).

Evidence tally: [3/3 validated]

Advisory AI assessment: sound. This is an automated, advisory judgment, not an approval.

Validation result: complete — pass — the changed helper ran on a real two-worker vLLM pair and passed, it failed on the destination line when the expected worker was swapped or a repeat was routed wrongly, and lint, collection, and the build check passed.

Evidence audit: complete [3/3 validated] — the command report comes from recorded runs.

Commands and results [3/3 validated]

Generated from the commands recorded during this run.

Check 1

Builds the changed Dynamo source and confirms that Python can import its compiled extension.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 2

Checks the changed files with the repository's fast lint and formatting commands.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 3

Runs the relevant GPU-dependent Python tests against the changed source.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

@glamr-agent

glamr-agent commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

No description provided.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: @dynamo-ops please run full CI for 44b0e51

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 44b0e51

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The router cache salt isolation test now captures decode worker IDs from streamed routing data and checks that repeated unpinned requests from each tenant reach the expected worker.

Changes

Router cache salt isolation

Layer / File(s) Summary
Capture and assert routed worker IDs
tests/router/common.py
The generate helper returns a decode worker ID from stream responses and fails if the response lacks one. Unpinned repeat checks assert that tenant A reaches worker A and tenant B reaches worker B.

Estimated code review effort: 2 (Simple) | ~8 minutes

Merge Risk: Minimal · up to 44b0e

No merge-blocking risk is evident; the unrun SGLang test remains a normal validation follow-up.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: adding routed-worker assertions for cache-salt isolation repeats.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections. The Overview explains the test gap and the fix. Key Changes lists the h…
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files.

Full details: Description check

Explanation

The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections. The Overview explains the test gap and the fix. Key Changes lists the helper change and the new destination assertions. Validation reports a temporary vLLM run, CI results on several heads, and a committed SGLang test that passed in CI. Issues lists the related issue and Linear ticket. The Pull Request Checklist item (using the /pr-checklist skill) is not checked and no checkbox state is shown for it. The description also contains an unresolved CI failure history and a mix of CI heads. The core required sections are present and substantive, so the description is mostly complete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between de49ef3 and 44b0e51.

Files selected for processing: 1

  • tests/router/common.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@devin-ai-integration

devin-ai-integration Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 41934815499 (attempt 1) on 9d5bb62851

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 41934815499. Updated on every full-CI run of this PR.

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 4, 2026 18:24 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 84e1af2

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 84e1af2

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 4, 2026 19:37 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 71216db

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 71216db

@MatejKosec
MatejKosec requested a review from krishung5 November 4, 2026 22:27
Comment thread tests/router/common.py Resolved
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 9d5bb62

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 9d5bb62

@MatejKosec
MatejKosec merged commit 54a93ca into ai-dynamo:main Nov 9, 2026
121 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 10, 2026
Absorbs 19 upstream commits (c5680e5..3a8226a). One conflict, the root Cargo.toml: kept upstream's dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 and dynamo-protocols 8.5.0 bumps together with the stack's workspace members and path crates; the four lockfiles were regenerated with cargo metadata (no drift from the auto-merge).

- ai-dynamo#17923 perf(kv-router) no block-lookup growth for existing-key updates: lands in lib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs (rename-detected; the stack copy differs from upstream only by the crate::prelude import).
- ai-dynamo#17741 build(deps) dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0, tokenizers 0.23.2 in lib/llm and lib/bench: root Cargo.toml conflict resolved as above; lib/llm/tests/tokenizer_cache.rs.
- ai-dynamo#17520 feat(vllm) realtime text tools, held-request prefill, chat completions and sampling options: Python vllm, dynamo-protocols 8.5.0 bump.
- ai-dynamo#18380 perf(bindings) direct token_ids/extra_args reads in RoutedEngine.generate and ai-dynamo#18448 refactor(bindings) py_convert: new lib/bindings/python/rust/llm/py_convert.rs, routed_engine.rs, engine.rs, _convert_preprocessed_request registered in lib.rs, _core.pyi, tests/test_preprocessed_request_conversion.py; llm/kv.rs extract_token_ids now delegates to py_convert::read_exact_u32_list with the stack's owner-crate imports kept.
- ai-dynamo#18454 fix(frontend) named passthrough fields to Python chat processors: lib/llm protocols/openai/chat_completions.rs, Python sglang and vllm processors.
- ai-dynamo#18253 fix(frontend) 16-bit image narrowing and EXIF opt-in: lib/llm preprocessor/media decoders, image_reader, loader.
- ai-dynamo#18185 fix(frontend) rendering context in speculative prefill: lib/llm preprocessor/speculative_prefill.rs.
- ai-dynamo#18428 fix(frontend) image step count without accounting: lib/llm protocols/openai/images aggregator.rs and nvext.rs.
- ai-dynamo#17656 fix(frontend) recipe-based DeepSeek encoding with SGLang: Python frontend sglang_deepseek.py, sglang_prompt_renderer.py, sglang_prepost.py, sglang_processor.py.
- ai-dynamo#18378 fix(runtime) TCP_NODELAY on accepted request-plane connections: lib/runtime pipeline/network/ingress/shared_tcp_endpoint.rs.
- ai-dynamo#18007 test(router) routed-worker assertion for cache-salt isolation repeats: tests/router/common.py and helper.py.
- ai-dynamo#18452 feat(backend) embedded engines skip Dynamo logging setup, ai-dynamo#18170 feat(trtllm) per-request metrics opt-in, ai-dynamo#18135 and ai-dynamo#16164 fix(vllm), ai-dynamo#16879 and ai-dynamo#18437 fix(recipes), c5680e5 docs community events: Python, recipes, docs; all auto-merged.

No change to the router kernel, admission, worker-selection, slot-tracker, session-affinity, kv-events or kv-hints crates; no [features] table or Dynamo path-crate dependency edge changed.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 9d5bb628 Deployed Nov 4, 2026 by glamr-agent via ok-to-test #46071
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor size/M test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants