Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion deploy/inference-gateway/ext-proc/src/epp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,6 @@ impl Router {
None,
model_name.clone(),
actual_namespace.to_string(),
enable_eagle,
// ext-proc constructs no KvWorkerMonitor; overload publishing is
// unused on this path (matches the prior namespace-lookup miss).
None,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -189,9 +189,15 @@ This is not specific to router flags — any card difference splits a set the sa
- **Applying the flag to only some replicas of a set splits it.** Whether by flag or by an exported `DYN_ROUTER_MODE` that reaches some processes and not others, every member of a set must end up with the same value.

> [!IMPORTANT]
> An advertised configuration **replaces** the frontend's for that worker set rather than merging with it. If you set `--router-mode kv` on a worker and the frontend was tuned with KV flags such as `--router-temperature`, restate those flags on the worker too, or that set falls back to KV defaults.
> An advertised configuration **replaces** the frontend's for that worker set rather than merging with it. On a worker, a flag you do not pass means **the default**, not "inherit the frontend's value". If the frontend was tuned with `--router-kv-overlap-score-credit 2.5` and a worker advertises only `--router-mode kv`, that worker set routes with the default `1.0` — the tuning is not inherited, it is replaced. Restate on the worker every flag that matters for it.

Because the flags are shared with the frontend, they read the same environment variables — `--router-mode` reads `DYN_ROUTER_MODE`. Kubernetes scopes environment per service, so a frontend's value does not reach workers, and the launch scripts pass the flag rather than exporting it. The one case to watch is a shell that exports `DYN_ROUTER_MODE` and then starts both the frontend and workers: there the workers advertise it instead of inheriting.
Merging is not offered because it cannot be done correctly: the KV tuning is flat concrete values with no record of which ones were set explicitly, so a deliberate `1.0` is indistinguishable from a default `1.0`.

Workers and the frontend share defaults exactly — both build their configuration from the same argument groups — so the values only diverge when the frontend is tuned and the worker is not.

The flags also read the same environment variables, and this changes the answer depending on how you deploy. If the frontend's tuning comes from the environment and the worker shares that environment — a single shell starting both — the worker picks the tuning up and nothing is lost. Kubernetes scopes environment per service, so there the worker sees only its own flags. The same intent can therefore produce different routing in the two setups. `--router-mode` itself is the exception in the other direction: a shared `DYN_ROUTER_MODE` makes a worker advertise when you meant it to inherit.

The `Activating prefill router` log line reports the configuration each hop actually resolved — mode, block size, session TTL, and KV tuning — which is the quickest way to confirm what a worker set ended up with.

The frontend-only options (`--router-min-initial-workers`, `--enforce-disagg`, `--admission-control`) are not accepted by workers, since a model card does not carry them.

Expand Down
1 change: 0 additions & 1 deletion lib/bindings/c/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -861,7 +861,6 @@ pub unsafe extern "C" fn create_routers(
None,
model_name.clone(),
actual_namespace.clone(),
enable_eagle,
// C bindings construct no KvWorkerMonitor; overload publishing is
// unused on this path (matches the prior namespace-lookup miss).
None,
Expand Down
1 change: 0 additions & 1 deletion lib/llm/src/discovery/model_manager.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3287,7 +3287,6 @@ mod tests {
None,
"topology-model".to_string(),
worker_set.namespace().to_string(),
false,
None,
);
let encoder = crate::kv_router::EncoderRouter::new(
Expand Down
4 changes: 2 additions & 2 deletions lib/llm/src/discovery/watcher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -651,8 +651,9 @@ where
{
let mut prefill_config = router_config.kv_router_config.clone();
prefill_config.router_track_active_blocks = false;
let prefill_enable_eagle = false;

// Fallback only: a prefill worker that declares its own
// `router_config` overrides this at activation time.
Some(PrefillRouter::new_with_selector_factory(
None,
self.manager.clone(),
Expand All @@ -665,7 +666,6 @@ where
router_config.session_affinity_ttl_secs,
model_name.clone(),
namespace.clone(),
prefill_enable_eagle,
worker_monitor.clone(),
Some(allocator_trim.clone()),
))
Expand Down
Loading
Loading