Inference: Fix EPP Endpoint Sync and Race Condition #12810
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
kind/fix
danehans
force-pushed
the
issue_12265
branch
3 times, most recently
from
48cc1ab to
fc451d6
Compare
danehans
changed the title
- Stores endpoints via atomic.Value and adds setEndpoints/getEndpoints to snapshot safely without locks. - Updates Equals to compare endpoint snapshots without locks, fixing race condition in krt.Equal/DeepEqual. - Switches error handling to hasErrors/snapshotErrors/setErrors. The backend path now returns empty ClusterLoadAssinment when errors exist. - Updates tests to seed errors via setErrors and avoid direct field access. - Keeps DP collection returning Backend IR on empty endpoints and relaxes route pass to allow empty endpoint sets. - Passes errors to DP to ensure cluster management. Signed-off-by: Daneyon Hansen <[email protected]>
lgadban
approved these changes
Nov 14, 2025
| ) *ir.EndpointsForBackend { | ||
| // Build an endpoint list | ||
| irPool := in.ObjIr.(*inferencePool) | ||
| poolEps := irPool.resolvePoolEndpoints(podIdx) |
Contributor
There was a problem hiding this comment.
we are no longer calling resolvePoolEndpoints in this function, so we are just relying on the endpoints to be computed elsewhere in the plugin?
Contributor
Author
There was a problem hiding this comment.
Correct, the eps resolution now happens only once during ApplyForBackend(). If the pool has no endpoints during ApplyForBackend(), translation does not fail. Instead, it keeps the route valid and provides an empty subset hint so the EPP returns a 5xx.
Merged
via the queue into
kgateway-dev:v2.1.x
with commit Nov 17, 2025
531027d
41 of 44 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes an issue where the inference extension did not program the managed Envoy proxy until controller restart due to mismatched endpoint sources between DP and IR translation paths.
Changes:
Fixes: #12265
Change Type
Changelog
Additional Notes
This is not a backport from
mainsince the Envoy-based data plane is being removed in v2.2.