Skip to content

docs(router): add disaggregated custom policy example - #12992

Merged
ishandhanani merged 8 commits into
mainfrom
idhanani/dyn-custom-policy-disagg-example
Aug 11, 2026
Merged

ishandhanani merged 8 commits into
mainfrom
idhanani/dyn-custom-policy-disagg-example

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Adds runnable aggregated and disaggregated worker-selection examples plus a complete custom-policy walkthrough. Teams can now create, register, configure, and link a policy without copying an example algorithm.

Related Issues

  • None.

Review Start

  • Start with examples/router/custom-policy-example/disaggregated/src/lib.rs for policy behavior.
  • Then review docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx for the end-to-end workflow.

How This Was Implemented

  • Added a disaggregated-load policy that scores prefill workers by active prefill tokens and decode workers by projected decode blocks.
  • Registered the aggregated and disaggregated examples through one catalog with clear dependency aliases.
  • Added example-local build, configuration, and contributor guidance.
  • Added a shared setup flow for creating the crates, adding dependencies, implementing traits, registering the provider, and configuring an instance.
  • Added Python Frontend and EPP tabs with the exact link and run commands for each host.
Walkthrough

Mental model

The policy crate owns scoring and picking. The catalog registers the policy type, and the host binary links that catalog at build time.

flowchart LR
    P["Policy crate"] --> C["Team catalog"]
    C --> F["Python frontend"]
    C --> E["Custom standalone EPP"]
    F -->|prefill or decode| W["Policy factory"]
    E -->|select| W
    W --> S["Scorers"]
    S --> K["Picker"]
    K --> R["Dynamo reservation and routing"]
Loading

Scoring and selection

The frontend supplies prefill or decode as worker_type. The disaggregated example uses active prefill tokens for prefill workers and projected decode blocks for decode workers.

Standalone EPP supplies select because it selects from one worker pool. The EPP example selects the aggregated least-busy instance, and custom factories must handle select explicitly.

Each scorer declares its cache, load, or routing inputs. Dynamo adds all scorer contributions, then the picker returns one valid candidate row.

Boundaries and limitations

Dynamo still owns eligibility, queueing, reservation, accounting, and metrics. The example algorithms demonstrate the extension contract and do not replace Dynamo's built-in cache-aware selector.

Validation

  • cargo test -p dynamo-custom-policy-example-basic -p dynamo-custom-policy-example-disaggregated -p dynamo-custom-policy-example-catalog -p dynamo-custom-policy-example-epp
  • cargo clippy -p dynamo-custom-policy-example-basic -p dynamo-custom-policy-example-disaggregated -p dynamo-custom-policy-example-catalog -p dynamo-custom-policy-example-epp --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • pre-commit run --files <changed files>
  • fern check
  • fern docs broken-links
  • Fresh external policy and EPP crates created with the documented cargo init and cargo add commands.

Summary by CodeRabbit

  • New Features

    • Added a disaggregated routing policy example for selecting prefill and decode workers based on load.
    • Expanded custom routing examples to cover multiple policy types and configuration options.
    • Added guidance for building, testing, configuring, and integrating custom routing strategies.
  • Documentation

    • Renamed and substantially updated the advanced customization guide to explain custom routing strategy design and integration.
    • Updated related documentation links and navigation references.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@github-actions github-actions Bot added docs documentation Improvements or additions to documentation labels Aug 11, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@github-actions

github-actions Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@datadog-official

datadog-official Bot commented Aug 11, 2026 •

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

PR | backend-status-check   View in Datadog   GitHub Actions

PR | vllm-runtime / Test cuda13.0, arm64   View in Datadog   GitHub Actions

See error Couldn't use data file '/workspace/.coverage': UNIQUE constraint failed: context.context

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5f68cbd | Docs | Datadog PR Page | Give us feedback!

@ishandhanani
ishandhanani marked this pull request as ready for review August 11, 2026 02:36
@ishandhanani
ishandhanani requested review from a team as code owners August 11, 2026 02:36
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The change adds a disaggregated custom worker-selection policy, registers it in the example catalog, adds example and contributor documentation, renames related documentation links, and enables link validation for custom-policy example URLs.

Changes

Custom routing policy

Layer / File(s) Summary
Disaggregated policy implementation
Cargo.toml, examples/router/custom-policy-example/disaggregated/*
Adds the disaggregated policy package. It scores prefill and decode workers from load data, selects the lowest-cost candidate, validates parameters, and registers the policy.
Catalog registration and validation
examples/router/custom-policy-example/catalog/*
Registers the aggregation and disaggregation policies. Tests verify duplicate-registration errors for both policy names.
Routing strategy documentation
docs/fern/*, examples/router/custom-policy-example/*, .lycheeignore
Replaces the custom worker-selection tutorial with a conceptual guide, updates related links, documents the example crates, and removes the link-validation exclusion.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the primary change: adding a disaggregated custom policy example.
Description check ✅ Passed The description covers the scope, implementation, review starting points, and validation, with only minor deviation from the required issue-link template.

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@harryskim harryskim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ishandhanani
ishandhanani enabled auto-merge (squash) August 11, 2026 04:37
@ishandhanani
ishandhanani merged commit 4dc3aa4 into main Aug 11, 2026
197 of 199 checks passed
@ishandhanani
ishandhanani deleted the idhanani/dyn-custom-policy-disagg-example branch August 11, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs documentation Improvements or additions to documentation size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants