Skip to content

feat(router): route the prefill hop by the prefill worker's advertised config - #13197

Merged
GuanLuo merged 7 commits into
mainfrom
gluo/prefill-card-router-mode
Aug 22, 2026
Merged

GuanLuo merged 7 commits into
mainfrom
gluo/prefill-card-router-mode

Conversation

@GuanLuo

@GuanLuo GuanLuo commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #13192. Base branch is gluo/per-role-router-mode, not main. Review that one first; this PR's diff is the three commits on top.

Summary

A disaggregated deployment routes twice — frontend → prefill worker, then frontend → decode worker — but both hops were driven by a single router_mode. --router-mode kv meant KV on both, round-robin meant round-robin on both, and there was no way to ask for one on prefill and the other on decode.

#13192 lets a worker set declare its own routing in its model deployment card. This makes that declaration take effect on the prefill hop, which the frontend previously routed using the decode set's mode.

Before / After

Before — the prefill hop inherits the decode set's mode, whatever the prefill workers advertise:

python -m dynamo.frontend --router-mode round-robin
python -m dynamo.vllm --model Qwen/Qwen3-0.6B --disaggregation-mode prefill --router-mode kv   # ignored
python -m dynamo.vllm --model Qwen/Qwen3-0.6B --disaggregation-mode decode
# both hops: round-robin

After — the prefill tier's own configuration governs its hop:

python -m dynamo.frontend --router-mode round-robin
python -m dynamo.vllm --model Qwen/Qwen3-0.6B --disaggregation-mode prefill --router-mode kv
python -m dynamo.vllm --model Qwen/Qwen3-0.6B --disaggregation-mode decode
# prefill: KV (prefix reuse)   decode: round-robin

The reverse pairing works equally — nothing here is KV-specific.

Before After
Prefill hop mode decode set's router_mode prefill card's, falling back to the decode set's
Prefill KV tuning decode set's prefill card's when it advertises one
Prefill block size decode set's prefill card's own (decode set's if the card reports 0)
Prefill session-affinity TTL decode set's prefill card's when it advertises a router_config
Prefill card advertises nothing n/a inherits the decode set — unchanged behavior
Unreadable prefill card fell back to the configured EAGLE mode activation fails and retries with backoff

Block size is the one of these that could be wrong rather than merely inflexible: the prefill pool's KV events are keyed on its own block size, so the decode set's value would index that pool at the wrong granularity. Disaggregation requires the two to match for KV transfer, so this is latent rather than live — but the frontend should not be the reason it holds.

A deployment where no prefill worker advertises anything behaves exactly as before: resolve_advertisement_from_cards returns the decode set's mode, and the decode set's tuning is used unchanged.

Details

The prefill router is owned by the decode worker set, so its mode came from there. PrefillRouter::router_mode was doing two jobs — gating conditional disagg (a decode-side decision, since it peeks the decode router) and gating Direct routing on the prefill dispatch. Those are split: decode_router_mode stays on the router, and the resolved prefill mode lives on PrefillBinding, because it is unknowable until a prefill target is discovered and changes when the binding is rebuilt.

build_binding already fetched the prefill cards to read enable_eagle, so resolution reuses that read rather than adding one. A card that advertises kv also carries its KV tuning; honoring only the mode would silently reset that hop to defaults.

An unreadable card now fails activation instead of silently inheriting. drive_target already retries with backoff, and a delayed activation costs less than routing a whole tier with the wrong strategy until the binding is next rebuilt. Requests during that window fall through to decode aggregated, the same as any pre-activation window.

A prefill fleet mid-rolling-update can disagree; the first card wins and the split is logged rather than passing silently.

The third commit drops the now-redundant is_eagle parameter from PrefillRouter::new. Every caller derived it from card.runtime_config.enable_eagle and handed it in, which is the field build_binding now reads itself; leaving the parameter would have meant a public argument that is silently ignored. epp.rs and the C bindings keep their copy for the decode chooser, which genuinely needs it.

Where should the reviewer start?

  • lib/llm/src/kv_router/prefill_router/activation.rs — resolve_advertisement_from_cards and its use in build_binding
  • lib/llm/src/kv_router/prefill_router/mod.rs — the decode_router_mode / prefill_router_mode split and the relocated Direct-routing check

Validation

  • cargo test -p dynamo-llm --lib — 2086 passed; cargo clippy --workspace -D warnings and cargo fmt --check clean
  • 6 unit tests on the resolver: inherit, override, fleet disagreement, unreadable card, block size, session TTL
  • tests/router/test_router_e2e_with_mockers.py -k "disagg or per_worker_config" — 17 passed, 1 skipped, unchanged from before this change
  • New e2e test_disagg_per_role_router_modes asserts prefix-extending requests converge on one prefill worker (KV) while spreading across decode workers (round-robin). Observed: prefill [335, 335, 335, 335], decode [380, 377, 380, 377]
  • Negative control: forcing the resolver to ignore the card makes prefill collapse to round-robin ([706, 703, 706, 703]) and the test fails on its intended assertion, so it is sensitive to the change rather than passing incidentally

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

Open in Devin Review

Summary by CodeRabbit

  • New Features

    • Prefill and decode workers can now use different routing modes.
    • Prefill routing automatically applies configuration advertised by active workers, with sensible fallback behavior.
    • Improved validation and diagnostics for conflicting or unavailable worker routing information.
  • Bug Fixes

    • Conditional disaggregation now correctly evaluates decode routing configuration.
    • Added end-to-end coverage to verify routing behavior across prefill and decode workers.

@GuanLuo
GuanLuo requested review from a team as code owners August 13, 2026 20:05
@github-actions github-actions Bot added feat router Relates to routing, KV-aware routing, etc. labels Aug 13, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 5 potential issues.

Open in Devin Review

@datadog-official

datadog-official Bot commented Aug 13, 2026 •

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 3 Pipeline jobs failed

PR | dynamo-runtime / test / parallel cuda13.0, amd64   View in Datadog   GitHub Actions

See error Failed to install build dependencies for 'aiconfigurator-core'.

PR | dynamo-runtime / test / parallel cuda13.0, arm64   View in Datadog   GitHub Actions

See error OSError: Could not install packages due to an HTTPS connection error: Max retries exceeded with url, caused by too many 502 error responses.

PR | dynamo-status-check   View in Datadog   GitHub Actions

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 6.06%
• Overall Coverage: 45.67% (-5.48%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6f4c03f | Docs | Datadog PR Page | Give us feedback!

@GuanLuo
GuanLuo requested review from a team as code owners August 14, 2026 16:11
@GuanLuo
GuanLuo force-pushed the gluo/prefill-card-router-mode branch from 881df79 to a431a44 Compare August 14, 2026 16:30
Base automatically changed from gluo/per-role-router-mode to main August 14, 2026 21:51
@GuanLuo
GuanLuo force-pushed the gluo/prefill-card-router-mode branch from a431a44 to 6ce5ddc Compare August 14, 2026 22:37
@coderabbitai

coderabbitai Bot commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The prefill router now resolves its mode, KV settings, and EAGLE state from prefill worker model cards. Decode routing remains separate. Call sites and end-to-end tests now use the updated constructor and per-role routing behavior.

Changes

Per-role prefill routing

Layer / File(s) Summary
Resolve prefill worker advertisements
lib/llm/src/kv_router/prefill_router/activation.rs
Activation discovers readable worker model cards, resolves prefill routing settings, handles conflicts, and reports missing-card errors.
Apply resolved modes during activation
lib/llm/src/kv_router/prefill_router/activation.rs, lib/llm/src/kv_router/prefill_router/mod.rs, lib/llm/src/kv_router/prefill_router/conditional_bypass.rs, lib/llm/src/kv_router/prefill_router/query.rs
Router state separates decode and prefill modes. Activation applies advertised KV settings and validates direct routing with the resolved prefill mode.
Update construction call sites and end-to-end coverage
deploy/inference-gateway/ext-proc/src/epp.rs, lib/bindings/c/src/lib.rs, lib/llm/src/discovery/model_manager.rs, lib/llm/src/discovery/watcher.rs, tests/router/common.py, tests/router/mocker_process.py, tests/router/test_router_e2e_with_mockers.py
Constructor callers use the revised arguments. Tests verify KV-based prefill convergence, round-robin decode distribution, and worker-role separation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🔵 Low · up to 6ce5d

This change makes prefill routing honor each prefill worker's advertised mode and tuning while preserving decode fallback behavior. The PR is mergeable with explicit owner follow-up on a bounded fallback-configuration correctness risk, CI timeout sensitivity, and limited visibility into inconsistent worker configuration.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 85.71% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The description includes the required sections, explains the behavior change, identifies review files, documents validation, and confirms that no issue is linked.
Title check ✅ Passed The title clearly summarizes the primary change: routing the prefill hop using the prefill worker's advertised configuration.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (4)
tests/router/common.py (2)

3763-3766: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use lazy logging arguments.

These logger.info calls use f-strings. The repository guidelines require format strings with separate arguments.

As per coding guidelines: "Use lazy formatting for log messages; pass format strings and arguments separately to logging calls instead of using f-strings or eager formatting".

♻️ Example for the collected-ids log
-        logger.info(f"Collected prefill_worker_ids: {prefill_ids}")
-        logger.info(f"Collected decode_worker_ids: {decode_ids}")
+        logger.info("Collected prefill_worker_ids: %s", prefill_ids)
+        logger.info("Collected decode_worker_ids: %s", decode_ids)

Also applies to: 3778-3779, 3814-3818

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@tests/router/common.py` around lines 3763 - 3766, Update the logger.info
calls near the request, collected-IDs, and worker-status logging to use lazy
logging: pass a format string with placeholders and the values as separate
arguments instead of f-strings, preserving the existing messages and values.

Sources: Coding guidelines, Path instructions


3729-3776: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider reusing the existing progressive-request loop.

This request loop duplicates the body of send_progressive_requests in _test_router_decisions_disagg (lines 2325-2402). The payload construction, SSE parsing, worker-id extraction, and the 1-second inter-request delay are identical. Only the timing assertions differ.

Extract a shared async helper that returns (prefill_ids, decode_ids) and let both scenarios call it with a flag for the timing checks. This keeps one place to update when the nvext.worker_id contract changes.

As per path instructions: "Prefer parameterized wrappers over copied test bodies when scenarios run across multiple backends, router modes, request planes, or storage backends" and "DO NOT copy-paste test infrastructure; reuse and refactor setup logic into fixtures in conftest.py or helper functions in tests/utils/".

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@tests/router/common.py` around lines 3729 - 3776, The progressive request
body in send_progressive_requests duplicates the corresponding loop in
_test_router_decisions_disagg. Extract the shared async request logic—including
payload construction, SSE worker-ID extraction, response handling, and
inter-request delay—into one helper returning prefill and decode ID lists, then
have both scenarios call it while retaining their distinct timing assertions.

Sources: Coding guidelines, Path instructions

lib/llm/src/kv_router/prefill_router/activation.rs (2)

420-432: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Raise the log level for skipped prefill cards.

The comment explains that an unreadable card removes a worker from the routing decision. The code logs that event at debug. In production the level is normally info or higher, so a partial-visibility resolution stays invisible while the routing mode silently depends on the remaining cards. Use warn for this event.

♻️ Proposed change
                     .inspect_err(|error| {
-                        tracing::debug!(%error, %endpoint_id, "Skipping unreadable prefill card")
+                        tracing::warn!(%error, %endpoint_id, "Skipping unreadable prefill card")
                     })
🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@lib/llm/src/kv_router/prefill_router/activation.rs` around lines 420 - 432,
Change the unreadable-card log in the deserialize_model filter within the
prefill routing flow from debug to warn, preserving the existing error and
endpoint_id fields and message.

76-88: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider also detecting divergent KV tuning across cards.

The disagreement check compares only mode_of. Two prefill cards can advertise the same router_mode with different kv_router_config values. In that case the first card's tuning is applied silently and no warning is emitted.

Extend the check to cover the KV config, or state in the doc comment that only the mode is validated.

🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

In `@lib/llm/src/kv_router/prefill_router/activation.rs` around lines 76 - 88,
Extend the disagreement detection around mode_of to also compare each card’s
kv_router_config with the selected first card’s configuration, and include
divergent KV tuning in the existing tracing::warn! path while preserving the
first-card resolution behavior.
🤖 Prompt for all review comments with AI agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

Inline comments:
In `@lib/llm/src/kv_router/prefill_router/activation.rs`:
- Around line 298-308: Update the prefill_kv_config selection to enforce
router_track_active_blocks = false for the existing kv_router_config in the None
branch as well as for advertised configs, preserving the invariant described by
the surrounding comment before the prefill KV chooser uses the result.

In `@tests/router/test_router_e2e_with_mockers.py`:
- Line 1429: Increase the pytest timeout marker for
_test_disagg_per_role_router_modes above the combined worst-case budgets of
frontend startup, both disaggregated worker launches, and the delayed requests;
retain the existing timeout=120 wait_for_frontend_ready behavior and choose a
value with sufficient CI variance headroom.

---

Nitpick comments:
In `@lib/llm/src/kv_router/prefill_router/activation.rs`:
- Around line 420-432: Change the unreadable-card log in the deserialize_model
filter within the prefill routing flow from debug to warn, preserving the
existing error and endpoint_id fields and message.
- Around line 76-88: Extend the disagreement detection around mode_of to also
compare each card’s kv_router_config with the selected first card’s
configuration, and include divergent KV tuning in the existing tracing::warn!
path while preserving the first-card resolution behavior.

In `@tests/router/common.py`:
- Around line 3763-3766: Update the logger.info calls near the request,
collected-IDs, and worker-status logging to use lazy logging: pass a format
string with placeholders and the values as separate arguments instead of
f-strings, preserving the existing messages and values.
- Around line 3729-3776: The progressive request body in
send_progressive_requests duplicates the corresponding loop in
_test_router_decisions_disagg. Extract the shared async request logic—including
payload construction, SSE worker-ID extraction, response handling, and
inter-request delay—into one helper returning prefill and decode ID lists, then
have both scenarios call it while retaining their distinct timing assertions.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 31258494-020a-4190-acdb-b7f5d5c3c445

📥 Commits

Reviewing files that changed from the base of the PR and between 5c950bf and 6ce5ddc.

📒 Files selected for processing (11)
  • deploy/inference-gateway/ext-proc/src/epp.rs
  • lib/bindings/c/src/lib.rs
  • lib/llm/src/discovery/model_manager.rs
  • lib/llm/src/discovery/watcher.rs
  • lib/llm/src/kv_router/prefill_router/activation.rs
  • lib/llm/src/kv_router/prefill_router/conditional_bypass.rs
  • lib/llm/src/kv_router/prefill_router/mod.rs
  • lib/llm/src/kv_router/prefill_router/query.rs
  • tests/router/common.py
  • tests/router/mocker_process.py
  • tests/router/test_router_e2e_with_mockers.py
💤 Files with no reviewable changes (3)
  • lib/llm/src/discovery/model_manager.rs
  • lib/bindings/c/src/lib.rs
  • deploy/inference-gateway/ext-proc/src/epp.rs

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 15, 2026
@github-actions

github-actions Bot commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

@alec-flowers

Copy link
Copy Markdown
Contributor

I feel this is confusing. Why wouldn't you decide this on the router?

GuanLuo and others added 7 commits August 20, 2026 09:35
…card

The frontend held a single `router_mode` and used it for both routers in a
disaggregated pipeline: the decode router built in the watcher and the prefill
router built underneath it. `--router-mode kv` meant KV on both hops and
`--router-mode round-robin` meant round-robin on both, with no way to ask for
one on prefill and the other on decode.

Per-worker-set `RouterConfig` already rides in the model deployment card, and
`register_model` already takes both `worker_type` and `router_config`, so a
prefill worker can already declare how it wants to be routed to. Nothing read
it: the watcher short-circuits prefill cards before applying `router_config`,
and the prefill router inherited the decode set's mode instead.

Resolve the prefill hop's mode from the prefill worker set's own cards at
activation, falling back to the decode set's mode when the card declares
nothing. `build_binding` already fetched those cards to read `enable_eagle`,
so this reuses that read rather than adding one. A KV-routed prefill tier in
front of a round-robin decode tier now works, as does the reverse.

Three things this required getting right:

- `router_mode` on `PrefillRouter` meant two things. Conditional disagg peeks
  the decode router, so it needs the decode set's mode; the Direct-routing
  guard is about the prefill dispatch, so it needs the prefill hop's. Split
  them: the field is now `decode_router_mode`, and the resolved prefill mode
  lives on `PrefillBinding` because it is only knowable once a prefill target
  is discovered. The Direct guard moved after the binding load accordingly.

- A card that declares a mode may declare KV tuning with it. Honoring only
  half of a `RouterConfig` would be a trap, so an advertised
  `kv_router_config` is used too, with `router_track_active_blocks` still
  forced off as the decode set does.

- An unreadable card now fails activation instead of silently inheriting.
  `drive_target` already retries with backoff, and a delayed activation costs
  far less than routing a whole deployment with the wrong strategy because of
  one transient discovery miss -- a downgrade that would persist until the
  binding was rebuilt.

A prefill fleet mid-rolling-update can disagree about its mode; the first card
wins, and the split is logged rather than passing silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Asserts the two hops route on their own terms rather than sharing one mode.
Prefill mockers advertise RouterConfig(RouterMode.KV); decode mockers advertise
nothing and inherit the frontend's --router-mode round-robin.

Prefix-extending requests must converge on one prefill worker, which only KV
routing produces, while spreading across more than one decode worker, which
only round-robin produces. The second assertion is the one that fails if the
per-role override is ignored and the pipeline silently runs in a single mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Self-review follow-up to the prefill card resolution.

`build_binding` now reads `enable_eagle` from the prefill card it already
fetches, so the value threaded through `PrefillRouter::new` was dead. Every
caller was computing the same lookup and handing it in: the frontend hardcoded
`false` (already overridden by the card read), while ext-proc and the C
bindings passed `card.runtime_config.enable_eagle` -- the same field, from a
card. Both keep their copy for the decode chooser, which genuinely needs it.

Removing the parameter is what makes that visible; keeping it would have left a
public argument that is silently ignored.

Also from the review:
- Log cards that fail to deserialize. An unreadable card used to cost only an
  EAGLE hint; it now drops a worker out of the vote that decides how the hop is
  routed, so it should not vanish silently.
- Drop `PrefillAdvertisement::advertised`; it was exactly
  `kv_router_config.is_some()` and existed only to feed one tracing field.
- Stop shadowing the `kv_router_config` parameter in `build_binding`.
- Remove the doubled error context around card resolution.
- Trim comments that restated the code, and the test that asserted the override
  works in the reverse direction -- same assertion with the modes swapped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
…d too

The prefill hop took its mode and KV tuning from the prefill card but its block
size and session-affinity TTL from the decode set, so a prefill tier could not
fully describe how it wants to be routed to. Resolve all four from the same
place.

Block size is the one that can be wrong rather than merely inflexible: the
prefill pool's KV events are keyed on its own block size, so using the decode
set's would index that pool at the wrong granularity. Disaggregation requires
the two to match for KV transfer, which is why this has not bitten anyone, but
the frontend should not be the reason it holds. A card reporting 0 never
declared one, so the decode set's value still applies there.

Session TTL rides in `router_config`, so it follows the same rule as the KV
tuning: an advertisement replaces the decode set's configuration rather than
merging with it. A card that advertises without a TTL therefore means "no
affinity on this hop", not "inherit the frontend's" -- hence `Option<Option<_>>`
to keep "advertised nothing" distinct from "advertised no TTL". Block size is
not part of `router_config` and is intrinsic to the worker, so it is taken
unconditionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Asserts the prefill and decode hops take their session-affinity setting from
their own cards rather than sharing one. Both tiers advertise round-robin and
only prefill advertises a TTL, so affinity is the sole difference: requests
carrying one session id stay on a single prefill worker while rotating across
decode workers.

Observed: prefill `[458, 458, 458, 458]`, decode `[501, 498, 501, 498]`. Giving
decode a TTL as well collapses it to `[861, 861, 861, 861]` and fails the decode
assertion, so the test is sensitive to the setting rather than to round-robin
timing.

This covers configuration, not expiry. Proving a TTL elapses means sleeping past
it and asserting a rebind, which is timing-dependent and can legitimately
re-select the same worker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
The helper landed in a584b62 without the test that calls it: reverting a
throwaway negative-control edit with `git checkout --` also discarded the
wrapper, because that file had not been committed yet. The helper was dead code
and the scenario never ran.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
An advertised card replaces the decode set's configuration rather than merging
with it, so a worker that names a mode without restating tuning silently gets
defaults. Nothing reported which values a hop ended up with, leaving "why did my
temperature reset" answerable only by reading code.

Log the resolved values -- mode, block size, session TTL, KV tuning -- once per
activation, after resolution rather than before, so the line reflects what the
hop actually uses rather than what the card asked for.

The docs said "restate your tuning" without saying why. State the rule directly:
on a worker an unset flag means the default, not "inherit the frontend's". Also
note that both sides build from the same argument groups, so defaults are
identical and values diverge only when the frontend is tuned and the worker is
not -- and that a shared environment changes that answer, since the worker then
picks the tuning up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
@GuanLuo
GuanLuo force-pushed the gluo/prefill-card-router-mode branch from 6f4c03f to b34d6b7 Compare August 20, 2026 16:35

@KrishnanPrash KrishnanPrash left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamped to unblock

@alec-flowers alec-flowers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still unsettled if we want this. Seems to me we are leaning towards 1 FE <-> 1 MDC

@GuanLuo
GuanLuo merged commit 9a4b1ff into main Aug 22, 2026
119 checks passed
@GuanLuo
GuanLuo deleted the gluo/prefill-card-router-mode branch August 22, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat router Relates to routing, KV-aware routing, etc. size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants