Skip to content

fix: keep 1.7 frontends and workers compatible with 1.5 and 1.6 peers - #18655

Draft
tmonty12 wants to merge 16 commits into
mainfrom
tmonty12/n2-skew-compat-fixes
Draft

tmonty12 wants to merge 16 commits into
mainfrom
tmonty12/n2-skew-compat-fixes

Conversation

@tmonty12

@tmonty12 tmonty12 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Dynamo supports mixed-version frontends and workers across an N-2 window, but several 1.7 changes break or degrade specific pairings with 1.5 and 1.6 peers, and the N-2 test suite never pairs a 1.7 candidate with 1.6. This PR fixes the pairings that can be fixed in 1.7 code, documents the ones that cannot, and points the suite at the right baselines.

Key Changes

Core logic: 295 lines (high), in seven independent fixes.

  • Response plane per worker. A worker advertises its response plane in its discovery instance and answers over tcp as well as that plane. A frontend uses its own plane with workers that advertise the same one and tcp with the rest. The rule is ResponsePlaneMode::for_worker and serves, the frontend applies it in responses_for_worker, and the worker in push_handler.rs.

    • A request to a worker on the frontend's own plane adds one field read and one compare. A velo or quic frontend already runs the tcp server that the fallback uses.
    • This has to ship in the release that introduces velo. A velo frontend treats a worker with no advertisement as tcp-only. release/1.7.0 already has velo without the advertisement, so if 1.7.0 ships without this, a later frontend with it cannot reach 1.7.0 velo workers over velo.
    • 1.6 runs quic without the advertisement, so six pairings with a 1.6 peer on quic still fail. The Compatibility page lists them.
    • The tcp fallback is encrypted only when DYN_TCP_TLS_* is set. A quic deployment that secures its request plane another way, and relied on a mismatched worker being refused, now gets plaintext responses from that worker.
  • 30-second response-stream bound on quic only. push_router.rs: a quic frontend can still pick a 1.5 worker, which accepts the request and never answers. tcp and velo keep no default, and DYN_RESPONSE_STREAM_ESTABLISH_TIMEOUT_SECS=0 disables it. The bound runs from the worker's ACK to its prologue, so a backend admission queue that holds a request longer than 30 seconds on quic times out.

  • Tolerant model-type reader. model_type.rs: ModelType skips flag names it does not know when a model card also carries one it does, so a 1.7 frontend keeps working with model types added in 1.8 and 1.9. A card with no known type is still rejected.

  • Worker-side parser-pair check. A 1.7 frontend refuses a model whose tool-call and reasoning parser pair it cannot serve, and the worker sees nothing: it stays up and the model never becomes ready. A worker now checks its pair when it registers the model, and exits with an error that names both selectors.

    • The check reuses the frontend's validation. validate_parser_pair_for_any_version refuses only a pair that the frontend's check refuses at auto, 1 and 2, because a worker cannot see its frontend's DYN_PARSER_VERSION. Both sides take which cards are checked from has_validated_parser_pair.
    • A frontend that runs a Python chat processor ignores these selectors. A worker whose pair has no Rust parser served behind such a frontend before, and exits now. That includes a sidecar that publishes engine-native parser names.
    • The check runs at registration, which is after the engine has loaded.

Also in this PR:

  • A worker-advertised power-of-two router mode maps to least-loaded (RouterMode::effective), for both the WorkerSet build and the prefill hop.
  • A serialized PoolId carries both cluster_id and dc_id, and the reader accepts either, with cluster_id authoritative. dc_id goes in 1.9. refactor: rename KV DC Relay runtime component and identities to PoolRelay #17215 had renamed the field with no alias.
  • A state-agent host matches an attachment intent to itself by instance identity and transport. It compared the whole instance, which a 1.5 or 1.6 worker echoes without the new response-plane field.
  • tests/deploy/n2: add the 1.6.2 images, move 1.5 to 1.5.1, and set DEFAULT_RELEASE_LINE to 1.7. All four image tags exist on NGC.
  • Docs: a "Feature-Specific Exceptions in 1.7" section on the Compatibility page, a --response-plane entry in the runtime configuration reference, and ledger entries for each behavior change.

Not changed, documented instead:

  • Worker request limit. A 1.7 worker refuses over-limit requests from its admission gate, after the request-plane ACK has gone out, so it cannot send the pre-1.7 Server overloaded reply that older frontends understand.
  • Parser pairs from 1.5 and 1.6 workers. A 1.7 frontend refuses a tool-call and reasoning parser pair from different families when one name is unified-only. The check is deliberate and tested (fix(parsers): [P2] validate V1 fallback and V2 selector pairs #17475), and an older worker does not have the worker-side check above. All 20 pairs used in the 1.6.2 recipes are admitted.

Validation

Runs used mock workers on one host: one etcd, a frontend from one release, workers from another. "1.7.0" is release/1.7.0 at 609df855e2 built from source; 1.6.2 and 1.5.1 are the published wheels.

Response plane. Each row is one frontend and one worker, two chat requests.

Frontend (plane) Worker (plane) 1.7.0 This PR
1.7 (velo) 1.7 (velo) 200 200 over velo
1.7 (velo) 1.7 (tcp) 503 200
1.7 (tcp) 1.7 (velo) 503 200
1.7 (velo) 1.6.2 or 1.5.1 Hangs; the worker stays routable 200
1.6.2 (tcp) 1.7 (velo) 503 200
1.5.1 (tcp) 1.7 (velo) 500 200
  • The same holds with quic in place of velo between two builds of this PR, and for a 1.6.2 frontend or worker on quic with this PR on quic.
  • Still failing, as documented: this PR on quic with a 1.6.2 tcp worker (503) or a 1.5.1 worker (504 after 30.0 s); this PR on tcp or velo with a 1.6.2 quic worker (503); a 1.6.2 quic frontend with this PR on tcp or velo (503).
  • Behind one velo frontend, 40 requests split 20/20 across two workers. The frontend's velo_transport_frames_total rose by 129 with two velo workers, 46 with one velo and one 1.6.2 worker, and 0 with a 1.6.2 and a 1.5.1 worker.

Other fixes.

Case 1.7.0 frontend This PR
Aggregated 1.6.2 or 1.5.1 worker started with --router-mode power-of-two Model materialization failed; scheduling retry; the model never becomes ready Model ready, chat returns 200, one warning naming the fallback
Disaggregated 1.6.2 or 1.5.1 prefill and decode workers with the same flag Same failure Chat returns 200
Model card with a model type the frontend does not know 1.6.2 and 1.5.1 frontends reject a 1.7 transcription card with unrecognized named flag; 1.7 has the same reader Unit test only: unknown names are skipped when a known one remains

Worker parser-pair check. One frontend and three workers, all on this PR, with DYN_PARSER_VERSION unset.

Worker selectors (tool-call + reasoning) Worker Frontend
kimi_k25 + kimi Exits with code 1; the error names both selectors Model not listed
kimi_k25 + kimi_k25 Registers Chat returns 200
beam + unset Registers, because DYN_PARSER_VERSION=2 accepts the pair Rejects the model, as before

Regression. Basic serving (chat and completions, unary and streaming, sampling options, client disconnect), KV-aware routing (24 of 24 repeated prompts return to the first worker), and disaggregated serving pass in all five pairings of this PR with 1.6.2 and 1.5.1, on the default response plane and with velo on the frontend, the workers, or both.

Tests added, and the regression each catches:

  • frontend_picks_the_response_plane_each_worker_serves, worker_serves_tcp_and_its_own_response_plane: the rule for any frontend plane, worker plane, and missing advertisement.
  • endpoint_response_plane_metadata_tolerates_older_and_newer_workers: an instance without the field, with null, or with a plane this version does not know fails to load.
  • quic_router_uses_tcp_for_a_worker_on_another_response_plane: a quic router names the wrong response server for a worker, loses the data of a tcp fallback, or misses its cancellation or tombstone when the worker leaves discovery.
  • tcp_request_reaches_the_engine_on_any_response_plane: a quic or velo worker refuses a tcp request again.
  • establish_timeout_defaults_only_for_quic: tcp or velo gains a default bound, quic loses it, 0 stops disabling it, or an unparsable value changes the default.
  • worker_advertised_power_of_two_routes_least_loaded, advertised_power_of_two_routes_least_loaded: the frontend fails a WorkerSet or a prefill hop whose workers advertise the removed mode.
  • unknown_model_type_names_are_skipped: an unknown model type fails a card that also has a known one, or a card with none is admitted as a worker with no endpoint.
  • any_version_check_refuses_only_what_every_version_refuses: the worker refuses a pair that some DYN_PARSER_VERSION value accepts, such as Beam in one slot, or admits a pair that none accepts.
  • attach_refuses_a_parser_pair_no_frontend_accepts: a worker publishes a model card whose pair no frontend accepts, or refuses a prefill worker, whose pair a frontend does not check.
  • pool_id_writes_both_field_names_and_reads_either: a 1.5 or 1.6 reader cannot read what 1.7 writes, 1.7 cannot read what a 1.5 or 1.6 writer sends, or dc_id wins when the two names disagree.
  • intent_from_an_older_worker_still_names_this_host: a state-agent host rejects the intent of a worker that echoes its instance without the response plane.
  • prefill_cancel_capability_reads_from_older_worker_cards: the frontend stops reading the prefill-cancel capability from a 1.6.2 worker's runtime_config, or starts reading it from a 1.5.1 worker's. The two payloads were captured from the published wheels.

Not validated:

  • No throughput benchmark of the per-request response-plane choice.
  • No streaming-input (bidirectional) request over the tcp fallback, and no run on Kubernetes discovery. Both share code paths that are covered: the tcp registration of a tcp frontend, and DiscoverySpec::into_instance.
  • The PoolId and state-agent host changes have unit tests only, because state agents need a real engine.
  • The worker parser-pair check ran with a mock worker through the Python register_model binding. No sidecar and no frontend with a Python chat processor was run.
  • The N-2 suite runs nightly on a cluster and was not run here; its unit tests pass against the new catalog.
  • No run used a real engine, so SGLang, vLLM and TensorRT-LLM handler paths are untested. That includes the prefill-cancel path itself: only its capability gate has a test.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

…-loaded

A v1.5 or v1.6 worker started with --router-mode power-of-two writes that mode
into its model card, and the frontend uses a worker's advertised router config
for its WorkerSet and for the prefill hop. The v1.7 frontend failed to build
both, so the model never became ready, or returned 503 in disaggregated
deployments. Map the removed mode to least-loaded, which picks on the same
in-flight request counts, and log a warning.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A worker that predates the frontend's response transport accepts the request,
fails to open a stream, and sends nothing back. With the establish timeout off
by default, the request waited forever and the worker stayed routable. Default
the timeout to 30 seconds when the response plane is quic or velo. tcp keeps no
default, DYN_RESPONSE_STREAM_ESTABLISH_TIMEOUT_SECS=0 still disables it, and a
value that does not parse now logs a warning.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The derived bitflags reader fails the whole card on an unknown name, so a
frontend rejects a worker that registers a model type added in a later
release alongside ones it knows. Skip unknown names and keep the rest, so a
v1.7 frontend stays compatible with model types added in v1.8 and v1.9. A card
left with no known type is still rejected, because an empty ModelType means a
worker with no client-facing endpoint.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The Pool Relay rename changed the serialized PoolId field from dc_id to
cluster_id with no alias, so v1.7 and v1.5/v1.6 state agents and routers could
not read each other's advertisements or event state sources. Write dc_id,
which the older releases know, and read both names.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The suite still tested a candidate against the 1.5 and 1.4 lines, so a v1.7
candidate was never paired with v1.6. Add the 1.6.2 images, move 1.5 to its
latest patch, and set the default release line to 1.7.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Add the settings that narrow the N-2 guarantee in 1.7 to the Compatibility
page, record the new response-stream bound and the parser-pair rejection in
the deprecations ledger, and update the power-of-two and Pool Relay entries
to match the code.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
@github-actions github-actions Bot added fix documentation Improvements or additions to documentation router Relates to routing, KV-aware routing, etc. labels Nov 12, 2026
…efusal

Replace the prose under Worker Request Limit with a table of what a 1.7, 1.6
and 1.5 frontend returns with migration off and on, and what each does to the
worker that refused.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
… write it

A v1.7 frontend forwards a client cancel to the prefill worker only when the
worker's model card advertises disagg_prefill_cancel_anytime_v1. v1.6 SGLang
disaggregated workers already advertise it, and v1.5 workers do not. Read the
runtime_config each release writes, captured from the published wheels, so a
change to how the flag is read cannot silently turn the path on or off for
older workers.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Writing only dc_id kept v1.5 and v1.6 peers working but left cluster_id off
the wire until v1.9 and the dc_id reader in place until v1.11. Write both
names instead. An older reader takes dc_id and ignores the key it does not
know, and a v1.7 reader accepts either name, with cluster_id authoritative
when both are present. dc_id and the wire struct can then go together in
v1.9, when v1.6 leaves the N-2 window.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A frontend sent its own response plane to every worker, and a worker refused
a request that named another plane. A velo frontend therefore hung on v1.5
and v1.6 workers, and a v1.7 worker on velo or quic refused every request
from a tcp frontend, including v1.5 and v1.6 frontends.

A worker now advertises its response plane in its discovery instance and
answers over tcp as well as that plane. A frontend uses its own plane with
workers that advertise the same one and tcp with the rest. Requests to a
worker on the frontend's plane take the same path as before, plus one field
read on the instance record.

v1.6 workers can run quic and do not advertise, so a quic frontend still
sends quic to a worker without an advertisement. Velo has not shipped in a
release, so a worker without an advertisement cannot serve it; that only
holds if this ships in the release that introduces velo.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Add --response-plane to the runtime configuration reference, rewrite the
Response Plane section of the Compatibility page around the pairings that
still have to match, and record the behavior change in the ledger.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
@github-actions github-actions Bot added the frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` label Nov 13, 2026
The router held a second handle to the tcp response server for workers on
another plane, which added a reference-count on the clone the guarded
dispatch path makes. Borrow the existing handle per request instead, so a
request to a worker on the frontend's own plane adds one field read and one
compare and nothing else.

Default the response-stream establish bound on quic only. With the
advertisement, a velo frontend can no longer pick a worker that never
answers, and the bound made a velo frontend time out requests that v1.5 and
v1.6 workers queue for longer than 30 seconds, where a tcp frontend waits.
A quic frontend can still pick a v1.5 worker, so it keeps the bound until
v1.5 leaves the window.

Add tests for a tcp request reaching the engine of a quic or velo worker,
for the server a velo router picks per worker, and for removal of a worker
that a quic router serves over tcp.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A worker echoes the instance its state-agent host advertised, and the host
compared that echo with its own instance field by field. A v1.5 or v1.6
worker drops instance fields it does not know, such as the response plane,
so the host rejected its attachment intent. Compare the endpoint instance
identity and transport instead.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
List every pairing that a v1.6 peer on quic still breaks, say which changes
are safe in which order, note that the tcp fallback is encrypted only with
DYN_TCP_TLS_*, and describe the 30-second bound as quic-only.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A frontend refuses a model whose tool-call and reasoning parser pair it
cannot serve. The worker sees nothing: it stays up and the model never
becomes ready.

Check the pair in the worker when it registers the model. A worker cannot
see its frontend's DYN_PARSER_VERSION, so it refuses only a pair that the
frontend's validation refuses at auto, 1 and 2. Both sides now take which
cards are checked from one predicate on the model card.

A frontend that runs a Python chat processor ignores these selectors, so
a worker with such a pair served there before and exits now.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant