Skip to content

feat(router): make session affinity capacity configurable - #17356

Open
Bafoleco wants to merge 5 commits into
ai-dynamo:mainfrom
Bafoleco:bay/upstream-affinity-capacity
Open

Bafoleco wants to merge 5 commits into
ai-dynamo:mainfrom
Bafoleco:bay/upstream-affinity-capacity

Conversation

@Bafoleco

@Bafoleco Bafoleco commented Oct 26, 2026 •

Copy link
Copy Markdown

Summary

Adds DYN_ROUTER_SESSION_AFFINITY_MAX_ENTRIES to configure capacity per affinity table. We've found this change was necessary as very long large deployments with long L3 backed TTLs can exceed 65k sessions.

Validation

  • cargo test -p dynamo-llm --no-default-features --lib session_affinity::tests --locked passed (35 tests).
  • Rust formatting and docs lint passed.

Related Issues

Closes #17363

Summary by CodeRabbit

  • New Features
    • Added a configurable limit for session-affinity entries per frontend table. Set DYN_ROUTER_SESSION_AFFINITY_MAX_ENTRIES to a positive integer; the default is 65,536.
    • When the limit is reached, new distinct sessions are not admitted to the table. Invalid values prevent the configuration from being applied.

@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Bafoleco
Bafoleco temporarily deployed to external_collaborator October 26, 2026 22:53 — with GitHub Actions Inactive
@Bafoleco
Bafoleco temporarily deployed to external_collaborator October 26, 2026 22:53 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi Bafoleco! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation router Relates to routing, KV-aware routing, etc. feat labels Oct 26, 2026
@Bafoleco
Bafoleco temporarily deployed to external_collaborator October 26, 2026 23:09 — with GitHub Actions Inactive
@Bafoleco
Bafoleco temporarily deployed to external_collaborator October 26, 2026 23:10 — with GitHub Actions Inactive
@Bafoleco
Bafoleco temporarily deployed to external_collaborator October 26, 2026 23:15 — with GitHub Actions Inactive
@Bafoleco
Bafoleco marked this pull request as ready for review October 26, 2026 23:24
@Bafoleco
Bafoleco requested review from a team as code owners October 26, 2026 23:24
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between b5313e7 and acd227a.

Files selected for processing: 6
  • docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md
  • lib/llm/src/kv_router/embedded.rs
  • lib/llm/src/session_affinity/coordinator.rs
  • lib/llm/src/session_affinity/mod.rs
  • lib/llm/src/session_affinity/tests.rs
  • lib/runtime/src/config/environment_names.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The router now reads a configurable maximum entry count for each session-affinity table. Both coordinator paths use the shared environment-based configuration, which validates the setting and preserves the existing default when it is unset.

Changes

Session-affinity capacity

Layer / File(s) Summary
Configure the session-affinity capacity
lib/runtime/src/config/environment_names.rs, lib/llm/src/session_affinity/mod.rs, lib/llm/src/session_affinity/tests.rs, docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md
Adds and documents DYN_ROUTER_SESSION_AFFINITY_MAX_ENTRIES. The configuration helper accepts positive integer values, preserves the default when unset, and returns errors for invalid values or lookup failures. Tests cover the default, invalid values, and admission at capacity.
Use environment-based configuration
lib/llm/src/session_affinity/coordinator.rs, lib/llm/src/kv_router/embedded.rs
Both coordinator paths obtain session-affinity configuration from config_from_env and propagate errors.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to acd22

The configurable session-affinity limit appears ready to merge after normal checks; no specific blocking issue is established.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 5 files. (1 skipped: 1 … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title "feat(router): make session affinity capacity configurable" matches the main change. The PR adds the DYN_ROUTER_SESSION_AFFINITY_MAX_ENTRIES environment variable, which sets the per-table en…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The pull request description includes the change summary, validation results, and the required Related Issues section with a valid issue reference. It does not include the template's "Where should the…
Full details: Title check

Explanation

The title "feat(router): make session affinity capacity configurable" matches the main change. The PR adds the DYN_ROUTER_SESSION_AFFINITY_MAX_ENTRIES environment variable, which sets the per-table entry limit (default 65536) for session affinity. The title is short, specific, and follows a conventional-commit style.

Full details: Description check

Explanation

The pull request description includes the change summary, validation results, and the required Related Issues section with a valid issue reference. It does not include the template's "Where should the reviewer start?" section, but the description is otherwise complete and on topic.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 5 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Comment thread lib/llm/src/session_affinity/tests.rs Resolved
@Bafoleco
Bafoleco temporarily deployed to external_collaborator October 26, 2026 23:57 — with GitHub Actions Inactive
Signed-off-by: Bay Foley-Cox <bayfoleycox@gmail.com>
Signed-off-by: Bay Foley-Cox <bayfoleycox@gmail.com>
Signed-off-by: Bay Foley-Cox <bayfoleycox@gmail.com>
…sses

Signed-off-by: Bay Foley-Cox <bayfoleycox@gmail.com>
Signed-off-by: Bay Foley-Cox <bayfoleycox@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — c23ae7a7 Deployed Oct 30, 2026 by Bafoleco via ok-to-test #42613
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat router Relates to routing, KV-aware routing, etc. size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CONTRIBUTION]: Allow customization of session affinity table size to account for high-scale / long-ttl workloads

1 participant