Skip to content

feat(epp): run custom selection services through standard bootstrap - #12720

Merged
ishandhanani merged 3 commits into
mainfrom
idhanani/dyn-3775-python-frontend-policy
Aug 8, 2026
Merged

ishandhanani merged 3 commits into
mainfrom
idhanani/dyn-3775-python-frontend-policy

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Moves the existing Rust EPP process bootstrap into the dynamo-ext-proc library and exposes it as a reusable runner. This lets custom linked policies pass a prebuilt SelectionService without copying Dynamo's TLS, health, metrics, discovery, readiness, or server setup.

CLOSES: DYN-3775

How This Was Implemented

  • Move the stock EPP bootstrap from the binary entrypoint into library-owned run() and run_with_selection_service(...) functions.
  • Keep dynamo-ext-proc as a seven-line binary that calls the same default runner.
  • Route a supplied service through the existing EppRouter::from_selection_service constructor in standalone EPP mode.
  • Document the custom EPP invocation.
Walkthrough

Mental model

The stock and custom binaries now differ only in how the selection service is supplied. Both enter the same process bootstrap and the same ext-proc request path.

flowchart LR
    D["Stock binary"] --> R["run()"]
    P["Custom policy crate"] --> S["Prebuilt SelectionService"]
    S --> C["run_with_selection_service()"]
    R --> B["Standard EPP bootstrap"]
    C --> B
    B --> E["EppRouter"]
    E --> X["ExtProcServer"]
Loading

State and ordering

run() preserves the existing environment-driven mode selection: standalone mode builds the default embedded selector, while Dynamo runtime mode uses discovery. run_with_selection_service(...) explicitly selects standalone mode and moves the caller-owned service into EppRouter; callers do not need to set DYN_EPP_MODE for the custom path.

Request lifecycle

Startup still initializes tracing, health reporting, metrics, standalone pod discovery or Dynamo discovery, readiness mirroring, TLS, and the ext-proc server in the existing order. After construction, requests follow the existing EppRouter and SelectionService paths for tokenization, eligibility, scheduling, validation, accounting, reservations, and completion callbacks.

Boundaries and limitations

This PR does not change the Python frontend, HttpFrontend, worker-selection traits, default selection behavior, or request hot path. It adds no macro, runtime policy loading, C ABI, policy registry, or frontend configuration surface; custom policies remain statically linked into a Rust EPP image.

Validation

  • cargo test -p dynamo-ext-proc --lib (117 passed)
  • cargo check -p dynamo-ext-proc --all-targets
  • cargo clippy -p dynamo-ext-proc --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • pre-commit run --files deploy/inference-gateway/ext-proc/Cargo.toml deploy/inference-gateway/ext-proc/src/bin/dynamo-ext-proc.rs deploy/inference-gateway/ext-proc/src/lib.rs deploy/inference-gateway/ext-proc/src/runner.rs docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.md
  • fern check
  • fern docs broken-links

Summary by CodeRabbit

  • New Features

    • Added support for running the inference gateway with a caller-provided selection service.
    • Standardized standalone gateway startup and health initialization.
  • Documentation

    • Updated custom worker-selection guidance to explain service ownership and standalone mode.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation labels Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

@datadog-official

datadog-official Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f0ac559 | Docs | Datadog PR Page | Give us feedback!

@ishandhanani
ishandhanani marked this pull request as ready for review August 6, 2026 15:43
@ishandhanani
ishandhanani requested review from a team as code owners August 6, 2026 15:43
devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3775-python-frontend-policy branch from 78872d0 to 7102393 Compare August 6, 2026 16:09
@ishandhanani
ishandhanani requested review from a team as code owners August 6, 2026 16:09
@copy-pr-bot

copy-pr-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pull-request-size pull-request-size Bot added size/M and removed size/L labels Aug 6, 2026
@ishandhanani ishandhanani changed the title feat(frontend): support linked policies in rebuilt Python frontend feat(epp): run custom selection services through standard bootstrap Aug 6, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

An error occurred while trying to automatically change base from idhanani/dyn-3712-native-worker-selection-api to main August 7, 2026 01:39
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3775-python-frontend-policy branch from 2f09560 to bd9a4b9 Compare August 7, 2026 20:58
@ishandhanani
ishandhanani requested review from a team as code owners August 7, 2026 20:58
@ishandhanani
ishandhanani requested review from a team as code owners August 7, 2026 20:58
@pull-request-size pull-request-size Bot removed the size/M label Aug 7, 2026
@github-actions github-actions Bot added backend::vllm Relates to the vllm backend deployment::k8s Relates to dynamo deployment in kubernetes backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend planner frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. actions labels Aug 7, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani changed the base branch from idhanani/dyn-3712-native-worker-selection-api to main August 7, 2026 21:11
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3775-python-frontend-policy branch from bd9a4b9 to f9229f5 Compare August 7, 2026 21:22
@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok-to-test f9229f5

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The EPP runner moved into reusable library code. The binary now invokes the crate-level runner, and custom worker selection passes a prebuilt SelectionService through the new API.

Changes

EPP runner integration

Layer / File(s) Summary
Runner APIs and startup flow
deploy/inference-gateway/ext-proc/src/lib.rs, deploy/inference-gateway/ext-proc/src/runner.rs
The crate exposes run and run_with_selection_service. Shared startup logic accepts an optional SelectionService and builds the router from the service or selector.
Binary and custom integration wiring
deploy/inference-gateway/ext-proc/Cargo.toml, deploy/inference-gateway/ext-proc/src/bin/dynamo-ext-proc.rs, docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.md
The binary target uses the new Tokio entry point. Custom worker-selection integration invokes run_with_selection_service with its prebuilt service.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: routing custom selection services through the standard EPP bootstrap.
Description check ✅ Passed The description explains the implementation, validation, and linked issue, but it does not identify specific files for reviewer focus.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok-to-test f0ac559

@ishandhanani
ishandhanani enabled auto-merge (squash) August 7, 2026 23:26
@ishandhanani
ishandhanani merged commit f95f86b into main Aug 8, 2026
127 of 129 checks passed
@ishandhanani
ishandhanani deleted the idhanani/dyn-3775-python-frontend-policy branch August 8, 2026 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` planner router Relates to routing, KV-aware routing, etc. size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants