Repository navigation
Conversation
…-loaded A v1.5 or v1.6 worker started with --router-mode power-of-two writes that mode into its model card, and the frontend uses a worker's advertised router config for its WorkerSet and for the prefill hop. The v1.7 frontend failed to build both, so the model never became ready, or returned 503 in disaggregated deployments. Map the removed mode to least-loaded, which picks on the same in-flight request counts, and log a warning. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A worker that predates the frontend's response transport accepts the request, fails to open a stream, and sends nothing back. With the establish timeout off by default, the request waited forever and the worker stayed routable. Default the timeout to 30 seconds when the response plane is quic or velo. tcp keeps no default, DYN_RESPONSE_STREAM_ESTABLISH_TIMEOUT_SECS=0 still disables it, and a value that does not parse now logs a warning. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The derived bitflags reader fails the whole card on an unknown name, so a frontend rejects a worker that registers a model type added in a later release alongside ones it knows. Skip unknown names and keep the rest, so a v1.7 frontend stays compatible with model types added in v1.8 and v1.9. A card left with no known type is still rejected, because an empty ModelType means a worker with no client-facing endpoint. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The Pool Relay rename changed the serialized PoolId field from dc_id to cluster_id with no alias, so v1.7 and v1.5/v1.6 state agents and routers could not read each other's advertisements or event state sources. Write dc_id, which the older releases know, and read both names. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The suite still tested a candidate against the 1.5 and 1.4 lines, so a v1.7 candidate was never paired with v1.6. Add the 1.6.2 images, move 1.5 to its latest patch, and set the default release line to 1.7. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Add the settings that narrow the N-2 guarantee in 1.7 to the Compatibility page, record the new response-stream bound and the parser-pair rejection in the deprecations ledger, and update the power-of-two and Pool Relay entries to match the code. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
…efusal Replace the prose under Worker Request Limit with a table of what a 1.7, 1.6 and 1.5 frontend returns with migration off and on, and what each does to the worker that refused. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
… write it A v1.7 frontend forwards a client cancel to the prefill worker only when the worker's model card advertises disagg_prefill_cancel_anytime_v1. v1.6 SGLang disaggregated workers already advertise it, and v1.5 workers do not. Read the runtime_config each release writes, captured from the published wheels, so a change to how the flag is read cannot silently turn the path on or off for older workers. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Writing only dc_id kept v1.5 and v1.6 peers working but left cluster_id off the wire until v1.9 and the dc_id reader in place until v1.11. Write both names instead. An older reader takes dc_id and ignores the key it does not know, and a v1.7 reader accepts either name, with cluster_id authoritative when both are present. dc_id and the wire struct can then go together in v1.9, when v1.6 leaves the N-2 window. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A frontend sent its own response plane to every worker, and a worker refused a request that named another plane. A velo frontend therefore hung on v1.5 and v1.6 workers, and a v1.7 worker on velo or quic refused every request from a tcp frontend, including v1.5 and v1.6 frontends. A worker now advertises its response plane in its discovery instance and answers over tcp as well as that plane. A frontend uses its own plane with workers that advertise the same one and tcp with the rest. Requests to a worker on the frontend's plane take the same path as before, plus one field read on the instance record. v1.6 workers can run quic and do not advertise, so a quic frontend still sends quic to a worker without an advertisement. Velo has not shipped in a release, so a worker without an advertisement cannot serve it; that only holds if this ships in the release that introduces velo. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Add --response-plane to the runtime configuration reference, rewrite the Response Plane section of the Compatibility page around the pairings that still have to match, and record the behavior change in the ledger. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
The router held a second handle to the tcp response server for workers on another plane, which added a reference-count on the clone the guarded dispatch path makes. Borrow the existing handle per request instead, so a request to a worker on the frontend's own plane adds one field read and one compare and nothing else. Default the response-stream establish bound on quic only. With the advertisement, a velo frontend can no longer pick a worker that never answers, and the bound made a velo frontend time out requests that v1.5 and v1.6 workers queue for longer than 30 seconds, where a tcp frontend waits. A quic frontend can still pick a v1.5 worker, so it keeps the bound until v1.5 leaves the window. Add tests for a tcp request reaching the engine of a quic or velo worker, for the server a velo router picks per worker, and for removal of a worker that a quic router serves over tcp. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A worker echoes the instance its state-agent host advertised, and the host compared that echo with its own instance field by field. A v1.5 or v1.6 worker drops instance fields it does not know, such as the response plane, so the host rejected its attachment intent. Compare the endpoint instance identity and transport instead. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
List every pairing that a v1.6 peer on quic still breaks, say which changes are safe in which order, note that the tcp fallback is encrypted only with DYN_TCP_TLS_*, and describe the 30-second bound as quic-only. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
A frontend refuses a model whose tool-call and reasoning parser pair it cannot serve. The worker sees nothing: it stays up and the model never becomes ready. Check the pair in the worker when it registers the model. A worker cannot see its frontend's DYN_PARSER_VERSION, so it refuses only a pair that the frontend's validation refuses at auto, 1 and 2. Both sides now take which cards are checked from one predicate on the model card. A frontend that runs a Python chat processor ignores these selectors, so a worker with such a pair served there before and exits now. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Dynamo supports mixed-version frontends and workers across an N-2 window, but several 1.7 changes break or degrade specific pairings with 1.5 and 1.6 peers, and the N-2 test suite never pairs a 1.7 candidate with 1.6. This PR fixes the pairings that can be fixed in 1.7 code, documents the ones that cannot, and points the suite at the right baselines.
Key Changes
Core logic: 295 lines (high), in seven independent fixes.
Response plane per worker. A worker advertises its response plane in its discovery instance and answers over
tcpas well as that plane. A frontend uses its own plane with workers that advertise the same one andtcpwith the rest. The rule isResponsePlaneMode::for_workerandserves, the frontend applies it inresponses_for_worker, and the worker inpush_handler.rs.release/1.7.0already has velo without the advertisement, so if 1.7.0 ships without this, a later frontend with it cannot reach 1.7.0 velo workers over velo.quicwithout the advertisement, so six pairings with a 1.6 peer onquicstill fail. The Compatibility page lists them.tcpfallback is encrypted only whenDYN_TCP_TLS_*is set. A quic deployment that secures its request plane another way, and relied on a mismatched worker being refused, now gets plaintext responses from that worker.30-second response-stream bound on quic only.
push_router.rs: a quic frontend can still pick a 1.5 worker, which accepts the request and never answers.tcpandvelokeep no default, andDYN_RESPONSE_STREAM_ESTABLISH_TIMEOUT_SECS=0disables it. The bound runs from the worker's ACK to its prologue, so a backend admission queue that holds a request longer than 30 seconds on quic times out.Tolerant model-type reader.
model_type.rs:ModelTypeskips flag names it does not know when a model card also carries one it does, so a 1.7 frontend keeps working with model types added in 1.8 and 1.9. A card with no known type is still rejected.Worker-side parser-pair check. A 1.7 frontend refuses a model whose tool-call and reasoning parser pair it cannot serve, and the worker sees nothing: it stays up and the model never becomes ready. A worker now checks its pair when it registers the model, and exits with an error that names both selectors.
validate_parser_pair_for_any_versionrefuses only a pair that the frontend's check refuses atauto,1and2, because a worker cannot see its frontend'sDYN_PARSER_VERSION. Both sides take which cards are checked fromhas_validated_parser_pair.Also in this PR:
power-of-tworouter mode maps toleast-loaded(RouterMode::effective), for both the WorkerSet build and the prefill hop.PoolIdcarries bothcluster_idanddc_id, and the reader accepts either, withcluster_idauthoritative.dc_idgoes in 1.9. refactor: rename KV DC Relay runtime component and identities to PoolRelay #17215 had renamed the field with no alias.tests/deploy/n2: add the 1.6.2 images, move 1.5 to 1.5.1, and setDEFAULT_RELEASE_LINEto1.7. All four image tags exist on NGC.--response-planeentry in the runtime configuration reference, and ledger entries for each behavior change.Not changed, documented instead:
Server overloadedreply that older frontends understand.Validation
Runs used mock workers on one host: one etcd, a frontend from one release, workers from another. "1.7.0" is
release/1.7.0at609df855e2built from source; 1.6.2 and 1.5.1 are the published wheels.Response plane. Each row is one frontend and one worker, two chat requests.
quicin place ofvelobetween two builds of this PR, and for a 1.6.2 frontend or worker onquicwith this PR onquic.quicwith a 1.6.2tcpworker (503) or a 1.5.1 worker (504 after 30.0 s); this PR ontcporvelowith a 1.6.2quicworker (503); a 1.6.2quicfrontend with this PR ontcporvelo(503).velo_transport_frames_totalrose by 129 with two velo workers, 46 with one velo and one 1.6.2 worker, and 0 with a 1.6.2 and a 1.5.1 worker.Other fixes.
--router-mode power-of-twoModel materialization failed; scheduling retry; the model never becomes readyunrecognized named flag; 1.7 has the same readerWorker parser-pair check. One frontend and three workers, all on this PR, with
DYN_PARSER_VERSIONunset.kimi_k25+kimikimi_k25+kimi_k25beam+ unsetDYN_PARSER_VERSION=2accepts the pairRegression. Basic serving (chat and completions, unary and streaming, sampling options, client disconnect), KV-aware routing (24 of 24 repeated prompts return to the first worker), and disaggregated serving pass in all five pairings of this PR with 1.6.2 and 1.5.1, on the default response plane and with velo on the frontend, the workers, or both.
Tests added, and the regression each catches:
frontend_picks_the_response_plane_each_worker_serves,worker_serves_tcp_and_its_own_response_plane: the rule for any frontend plane, worker plane, and missing advertisement.endpoint_response_plane_metadata_tolerates_older_and_newer_workers: an instance without the field, withnull, or with a plane this version does not know fails to load.quic_router_uses_tcp_for_a_worker_on_another_response_plane: a quic router names the wrong response server for a worker, loses the data of a tcp fallback, or misses its cancellation or tombstone when the worker leaves discovery.tcp_request_reaches_the_engine_on_any_response_plane: a quic or velo worker refuses a tcp request again.establish_timeout_defaults_only_for_quic:tcporvelogains a default bound, quic loses it,0stops disabling it, or an unparsable value changes the default.worker_advertised_power_of_two_routes_least_loaded,advertised_power_of_two_routes_least_loaded: the frontend fails a WorkerSet or a prefill hop whose workers advertise the removed mode.unknown_model_type_names_are_skipped: an unknown model type fails a card that also has a known one, or a card with none is admitted as a worker with no endpoint.any_version_check_refuses_only_what_every_version_refuses: the worker refuses a pair that someDYN_PARSER_VERSIONvalue accepts, such as Beam in one slot, or admits a pair that none accepts.attach_refuses_a_parser_pair_no_frontend_accepts: a worker publishes a model card whose pair no frontend accepts, or refuses a prefill worker, whose pair a frontend does not check.pool_id_writes_both_field_names_and_reads_either: a 1.5 or 1.6 reader cannot read what 1.7 writes, 1.7 cannot read what a 1.5 or 1.6 writer sends, ordc_idwins when the two names disagree.intent_from_an_older_worker_still_names_this_host: a state-agent host rejects the intent of a worker that echoes its instance without the response plane.prefill_cancel_capability_reads_from_older_worker_cards: the frontend stops reading the prefill-cancel capability from a 1.6.2 worker'sruntime_config, or starts reading it from a 1.5.1 worker's. The two payloads were captured from the published wheels.Not validated:
DiscoverySpec::into_instance.PoolIdand state-agent host changes have unit tests only, because state agents need a real engine.register_modelbinding. No sidecar and no frontend with a Python chat processor was run.Issues and Ongoing Work
release/1.7.0. Two changes have to ship in 1.7.0 itself and not a later patch: the response-plane advertisement (see above), and thePoolIdchange, because a 1.7.0 reader without it cannot readdc_id. The response-plane andPoolIdcommits need a manual port:release/1.7.0keeps those files underlib/kv-router, and about 58 struct literals gain aresponse_plane: Noneline.power-of-two, feat(runtime): add Velo TCP and UCX response transport #15231 added the Velo response plane, feat(runtime): add batched QUIC response transport #11996 added the QUIC response plane, fix(parsers): [P2] validate V1 fallback and V2 selector pairs #17475 added the parser-pair validation, and refactor: rename KV DC Relay runtime component and identities to PoolRelay #17215 renamed the Pool Relay identities. I found no open PR or issue covering these fixes.Pull Request Checklist
/pr-checklistskill.