[API]: feat: add Topology Policy for StormService - #1846
Conversation
Summary of ChangesHello @googs1025, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a TopologyPolicy to control pod co-location for StormServices. The changes include adding the new API types, updating CRDs, and implementing the logic to inject pod affinity rules based on the policy.
The implementation is mostly solid, but I've identified a few areas for improvement:
- There is a typo in a new API constant that should be corrected.
- The validation for the new
TopologyPolicycould be strengthened to provide better feedback to users on misconfiguration. - There is some code duplication in the new utility functions that can be refactored for better maintainability.
Overall, this is a good addition. Please see my detailed comments for suggestions.
| // Key is the Kubernetes topology label key to enforce co-location on. | ||
| // Common values include: | ||
| // - "kubernetes.io/hostname" (node-level) | ||
| // - "topology.kubernetes.io/zone" (zone-level) | ||
| // Required when Scope is set. | ||
| // +optional | ||
| Key string `json:"key,omitempty"` |
There was a problem hiding this comment.
The comment for Key on line 156 states that it's "Required when Scope is set", but this is not enforced by any validation, and the field is marked as +optional. While the controller logic handles this by ignoring the policy if either Key or Scope is empty, it would be better for user experience to enforce this relationship via validation. Consider adding a validating webhook to enforce that Key is present if Scope is set. Without it, a user might specify a Scope without a Key and the policy would be silently ignored, which can be hard to debug.
There was a problem hiding this comment.
add TODO to solve in next pr
2183d34 to
0681d7b
Compare
|
local test: case 1:root@iZ6we78ovpjfbxlipufqliZ:~# kubectl get node -owide --show-labels
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS
cluster1-control-plane Ready control-plane 22h v1.32.2 172.18.0.3 <none> Debian GNU/Linux 12 (bookworm) 6.8.0-88-generic containerd://2.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
cluster1-worker Ready <none> 22h v1.32.2 172.18.0.5 <none> Debian GNU/Linux 12 (bookworm) 6.8.0-88-generic containerd://2.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gateway=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-worker,kubernetes.io/os=linux,topology.kubernetes.io/zone=zone-a
cluster1-worker2 Ready <none> 22h v1.32.2 172.18.0.4 <none> Debian GNU/Linux 12 (bookworm) 6.8.0-88-generic containerd://2.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-worker2,kubernetes.io/os=linux,topology.kubernetes.io/zone=zone-a
cluster1-worker3 Ready <none> 22h v1.32.2 172.18.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.8.0-88-generic containerd://2.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-worker3,kubernetes.io/os=linux,topology.kubernetes.io/zone=zone-b
cluster1-worker4 Ready <none> 22h v1.32.2 172.18.0.6 <none> Debian GNU/Linux 12 (bookworm) 6.8.0-88-generic containerd://2.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-worker4,kubernetes.io/os=linux,topology.kubernetes.io/zone=zone-b
root@iZ6we78ovpjfbxlipufqliZ:~# vi aa.yaml
root@iZ6we78ovpjfbxlipufqliZ:~# vi aa.yaml
root@iZ6we78ovpjfbxlipufqliZ:~# kubectl apply -f aa.yaml
stormservice.orchestration.aibrix.ai/pd-inference created
root@iZ6we78ovpjfbxlipufqliZ:~# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pd-inference-roleset-7zrrb-decode-6794f4-0 1/1 Running 0 8s 10.6.2.48 cluster1-worker <none> <none>
pd-inference-roleset-7zrrb-decode-6794f4-1 1/1 Running 0 8s 10.6.4.84 cluster1-worker2 <none> <none>
pd-inference-roleset-7zrrb-decode-6794f4-2 1/1 Running 0 8s 10.6.4.85 cluster1-worker2 <none> <none>
pd-inference-roleset-7zrrb-decode-6794f4-3 1/1 Running 0 8s 10.6.2.47 cluster1-worker <none> <none>
pd-inference-roleset-7zrrb-prefill-648798-0-0 1/1 Running 0 8s 10.6.3.52 cluster1-worker3 <none> <none>
pd-inference-roleset-7zrrb-prefill-648798-0-1 1/1 Running 0 7s 10.6.1.29 cluster1-worker4 <none> <none>
pd-inference-roleset-7zrrb-prefill-648798-1-0 1/1 Running 0 7s 10.6.3.53 cluster1-worker3 <none> <none>
pd-inference-roleset-7zrrb-prefill-648798-1-1 1/1 Running 0 7s 10.6.1.30 cluster1-worker4 <none> <none>
pd-inference-roleset-fjf5b-decode-6794f4-0 1/1 Running 0 8s 10.6.2.49 cluster1-worker <none>
... apiVersion: orchestration.aibrix.ai/v1alpha1
kind: StormService
metadata:
name: pd-inference
spec:
replicas: 3
stateful: true
selector:
matchLabels:
app: pd-inference
template:
metadata:
labels:
app: pd-inference
spec:
topologyPolicy:
scope: Role
key: topology.kubernetes.io/zone
roles:
- name: prefill
replicas: 2
podGroupSize: 2
stateful: true
template:
metadata:
labels:
role: prefill
spec:
containers:
- name: prefill
image: nginx:alpine
- name: decode
replicas: 4
stateful: true
template:
metadata:
labels:
role: decode
spec:
containers:
- name: decode
image: nginx:alpinecase 2:root@iZ6we78ovpjfbxlipufqliZ:~# cat aa.yaml
apiVersion: orchestration.aibrix.ai/v1alpha1
kind: StormService
metadata:
name: pd-inference
spec:
replicas: 3
stateful: true
selector:
matchLabels:
app: pd-inference
template:
metadata:
labels:
app: pd-inference
spec:
topologyPolicy:
scope: Role
key: topology.kubernetes.io/zone
roles:
- name: prefill
replicas: 2
podGroupSize: 2
stateful: true
template:
metadata:
labels:
role: prefill
spec:
containers:
- name: prefill
image: nginx:alpine
- name: decode
replicas: 4
stateful: true
template:
metadata:
labels:
role: decode
spec:
containers:
- name: decode
image: nginx:alpine
root@iZ6we78ovpjfbxlipufqliZ:~#
root@iZ6we78ovpjfbxlipufqliZ:~# kubectl apply -f aa.yaml
stormservice.orchestration.aibrix.ai/pd-inference created
root@iZ6we78ovpjfbxlipufqliZ:~# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pd-inference-roleset-8qw9t-decode-6794f4-0 1/1 Running 0 8s 10.6.3.56 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-decode-6794f4-1 1/1 Running 0 8s 10.6.3.59 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-decode-6794f4-2 1/1 Running 0 8s 10.6.3.57 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-decode-6794f4-3 1/1 Running 0 8s 10.6.3.58 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-prefill-648798-0-0 1/1 Running 0 8s 10.6.3.63 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-prefill-648798-0-1 1/1 Running 0 8s 10.6.3.60 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-prefill-648798-1-0 1/1 Running 0 8s 10.6.3.61 cluster1-worker3 <none> <none>
pd-inference-roleset-8qw9t-prefill-648798-1-1 1/1 Running 0 8s 10.6.3.62 cluster1-worker3 <none> <none>
pd-inference-roleset-bgqhg-decode-6794f4-0 1/1 Running 0 9s 10.6.2.54 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-decode-6794f4-1 1/1 Running 0 9s 10.6.2.51 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-decode-6794f4-2 1/1 Running 0 9s 10.6.2.53 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-decode-6794f4-3 1/1 Running 0 9s 10.6.2.52 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-prefill-648798-0-0 1/1 Running 0 9s 10.6.2.55 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-prefill-648798-0-1 1/1 Running 0 9s 10.6.2.56 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-prefill-648798-1-0 1/1 Running 0 8s 10.6.2.57 cluster1-worker <none> <none>
pd-inference-roleset-bgqhg-prefill-648798-1-1 1/1 Running 0 8s 10.6.2.58 cluster1-worker <none> <none>
pd-inference-roleset-dfwfs-decode-6794f4-0 1/1 Running 0 8s 10.6.1.36 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-decode-6794f4-1 1/1 Running 0 8s 10.6.1.34 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-decode-6794f4-2 1/1 Running 0 8s 10.6.1.33 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-decode-6794f4-3 1/1 Running 0 8s 10.6.1.35 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-prefill-648798-0-0 1/1 Running 0 8s 10.6.1.37 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-prefill-648798-0-1 1/1 Running 0 8s 10.6.1.39 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-prefill-648798-1-0 1/1 Running 0 8s 10.6.1.38 cluster1-worker4 <none> <none>
pd-inference-roleset-dfwfs-prefill-648798-1-1 1/1 Running 0 8s 10.6.1.40 cluster1-worker4 <none> <none>
|
Signed-off-by: CYJiang <googs1025@gmail.com>
0681d7b to
fdd1d4e
Compare
|
@Jeffwan /PTAL |
Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
5548953 to
3f78ff0
Compare
Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new topologyPolicy field to the StormService/RoleSet API surface and wires controller logic to inject pod-affinity terms so pods can be co-located on a chosen Kubernetes topology domain (e.g., hostname/zone), with unit and integration coverage.
Changes:
- Introduce
TopologyPolicy(scope/mode/key) in the orchestration API and regenerate deepcopy/applyconfiguration helpers. - Inject pod affinity into both direct Pods and PodSet templates when
RoleSetSpec.TopologyPolicyis set. - Add unit tests and an integration test validating default (Preferred) topology affinity injection.
Reviewed changes
Copilot reviewed 7 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/integration/controller/roleset_test.go | Integration test ensuring affinity injection for both Pod and PodSet roles. |
| pkg/controller/roleset/utils.go | Implements topology-policy label selection and affinity injection logic. |
| pkg/controller/roleset/utils_test.go | Unit tests for match-label selection and affinity injection behavior. |
| pkg/controller/roleset/podset_rollsyncer.go | Injects topology affinity into PodSet templates during creation. |
| pkg/client/applyconfiguration/utils.go | Registers applyconfiguration mapping for TopologyPolicy. |
| pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go | Generated applyconfiguration type for TopologyPolicy. |
| pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go | Adds TopologyPolicy to RoleSetSpec applyconfiguration. |
| dist/chart/crds/orchestration.aibrix.ai_stormservices.yaml | Helm CRD update to expose topologyPolicy in StormService template spec. |
| dist/chart/crds/orchestration.aibrix.ai_rolesets.yaml | Helm CRD update to expose topologyPolicy in RoleSet spec. |
| config/crd/orchestration/orchestration.aibrix.ai_stormservices.yaml | Source CRD update for topologyPolicy in StormService template spec. |
| config/crd/orchestration/orchestration.aibrix.ai_rolesets.yaml | Source CRD update for topologyPolicy in RoleSet spec. |
| api/orchestration/v1alpha1/zz_generated.deepcopy.go | Generated deepcopy updates for TopologyPolicy and RoleSetSpec. |
| api/orchestration/v1alpha1/roleset_types.go | Defines TopologyPolicy, enums, and adds it to RoleSetSpec. |
Files not reviewed (4)
- api/orchestration/v1alpha1/zz_generated.deepcopy.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go: Generated file
- pkg/client/applyconfiguration/utils.go: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| case orchestrationv1alpha1.TopologyPolicyRequired: | ||
| // avoid duplicate terms | ||
| for _, term := range spec.Affinity.PodAffinity.RequiredDuringSchedulingIgnoredDuringExecution { | ||
| if term.TopologyKey == topologyKey && | ||
| reflect.DeepEqual(term.LabelSelector.MatchLabels, matchLabels) { | ||
| return | ||
| } | ||
| } | ||
| spec.Affinity.PodAffinity.RequiredDuringSchedulingIgnoredDuringExecution = | ||
| append(spec.Affinity.PodAffinity.RequiredDuringSchedulingIgnoredDuringExecution, affinityTerm) |
There was a problem hiding this comment.
Fixed in 5386063 by guarding nil LabelSelector before comparing existing required affinity terms. Added a regression test covering existing required terms with nil selectors.
| for _, term := range spec.Affinity.PodAffinity.PreferredDuringSchedulingIgnoredDuringExecution { | ||
| if term.Weight == weightedTerm.Weight && | ||
| term.PodAffinityTerm.TopologyKey == topologyKey && | ||
| reflect.DeepEqual(term.PodAffinityTerm.LabelSelector.MatchLabels, matchLabels) { | ||
| return | ||
| } | ||
| } | ||
| spec.Affinity.PodAffinity.PreferredDuringSchedulingIgnoredDuringExecution = | ||
| append(spec.Affinity.PodAffinity.PreferredDuringSchedulingIgnoredDuringExecution, weightedTerm) |
There was a problem hiding this comment.
Fixed in 5386063 by guarding nil PodAffinityTerm.LabelSelector before comparing existing preferred affinity terms. Added a regression test covering existing preferred terms with nil selectors.
| // TopologyPolicy specifies how Pods are co-located based on Kubernetes topology keys. | ||
| type TopologyPolicy struct { | ||
| // Scope defines the granularity of co-location. | ||
| // Valid values are: | ||
| // - "StormService": All Pods in the entire StormService share the same topology value. | ||
| // - "RoleSet": All Pods within each RoleSet share the same topology value (different RoleSets may be on different domains). | ||
| // - "Role": All Pods of the same role (across all RoleSets) share the same topology value. |
There was a problem hiding this comment.
Updated the PR description with the topologyPolicy API fields, Preferred/Required scheduling semantics, an example, related issues, and the verification commands.
Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
📐 API Design
⚙️ Controller Logic
🧪 Tests
Here is a clean, structured formatting of your additional code review feedback: ⚙️ Refactoring & Code Cleanup
🔍 Dead Code & Optimization
|
Signed-off-by: CYJiang <googs1025@gmail.com>
13b8b55 to
adb5ee0
Compare
# Conflicts: # pkg/controller/roleset/utils.go
|
@googs1025 let me know once it is ready to review. |
yes, ready for review @varungup90 @Jeffwan |
|
Thanks for the work on this feature! The core logic looks solid, but I've found a few areas that need attention, particularly regarding CRD synchronization and observability. Below is my feedback broken down into actionable issues and a few non-blocking observations. 🚨 Actionable Issues1. CRD drift between key:
maxLength: 317
pattern: ^([a-z0-9]...
type: string
However, the 2. Missing observability when a topology policy is silently skipped
3. Integration test couples policy change with scale-up
💡 Observations & Nits (Non-blocking)
|
Signed-off-by: CYJiang <googs1025@gmail.com>
|
@Jeffwan /PTAL |
Preserve topology affinity injection and integration coverage while incorporating the RoleSet in-place update events, tests, and helpers added on main. Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
Resolve RoleSet topology policy conflicts with pod in-place update events and integration coverage. Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
Restore auto-merged RoleSet API, CRD, and controller updates from main while retaining topology policy changes and the two conflict resolutions. Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
|
cc @Jeffwan /PTAL |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 16 changed files in this pull request and generated 2 comments.
Files not reviewed (4)
- api/orchestration/v1alpha1/zz_generated.deepcopy.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go: Generated file
- pkg/client/applyconfiguration/utils.go: Generated file
| r.EventRecorder.Eventf( | ||
| roleSet, | ||
| corev1.EventTypeWarning, | ||
| TopologyPolicyPendingPodReplacementEventType, | ||
| "TopologyPolicy changes do not update %s or %s; they will use the new affinity after replacement or recreation.", | ||
| formatTopologyPolicyEventCount(outdatedPods, "active Pod"), | ||
| formatTopologyPolicyEventCount(outdatedPodSets, "PodSet template"), | ||
| ) |
|
|
||
| assert.NoError(t, reconciler.emitTopologyPolicyPendingReplacementEvent(ctx, roleSet)) | ||
|
|
||
| event := <-recorder.Events |
Signed-off-by: CYJiang <googs1025@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 16 changed files in this pull request and generated no new comments.
Files not reviewed (4)
- api/orchestration/v1alpha1/zz_generated.deepcopy.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go: Generated file
- pkg/client/applyconfiguration/utils.go: Generated file
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 21 changed files in this pull request and generated 1 comment.
Files not reviewed (4)
- api/orchestration/v1alpha1/zz_generated.deepcopy.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go: Generated file
- pkg/client/applyconfiguration/utils.go: Generated file
Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
a9d58c9 to
46e7a9d
Compare
|
@Jeffwan Could you please take a look at this Topology Policy feature when you have time? It is important for controlling prefill/decode and RoleSet placement across topology domains, and I have added local validation and samples to make the usage clear. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 21 changed files in this pull request and generated 2 comments.
Files not reviewed (4)
- api/orchestration/v1alpha1/zz_generated.deepcopy.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go: Generated file
- pkg/client/applyconfiguration/utils.go: Generated file
| func waitForTopologyRolePods(namespace, roleSetName, roleName string, expected int) []corev1.Pod { | ||
| var pods []corev1.Pod | ||
| gomega.Eventually(func(g gomega.Gomega) { | ||
| podList := &corev1.PodList{} | ||
| g.Expect(k8sClient.List(ctx, podList, | ||
| client.InNamespace(namespace), |
| var outdatedPods int | ||
| var outdatedPodSets int | ||
| for i := range roleSet.Spec.Roles { | ||
| role := &roleSet.Spec.Roles[i] | ||
| matchLabels, ok := getTopologyMatchLabels(roleSet, role.Name, roleSet.Spec.TopologyPolicy) | ||
| if !ok || !validateTopologyKey(roleSet, roleSet.Spec.TopologyPolicy) { | ||
| continue | ||
| } | ||
|
|
||
| pods, err := getRolePods(ctx, r.Client, roleSet.Namespace, roleSet.Name, role.Name) | ||
| if err != nil { | ||
| return err |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 21 changed files in this pull request and generated no new comments.
Files not reviewed (4)
- api/orchestration/v1alpha1/zz_generated.deepcopy.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/rolesetspec.go: Generated file
- pkg/client/applyconfiguration/orchestration/v1alpha1/topologypolicy.go: Generated file
- pkg/client/applyconfiguration/utils.go: Generated file
Comments suppressed due to low confidence (2)
pkg/controller/roleset/topology_policy_event.go:45
validateTopologyKeyis called inside the per-role loop, so an emptyTopologyPolicy.Keywill log the same warning once per role on every reconcile. Since the key is RoleSet-wide (not per-role), validate once up-front and return early to avoid log spam and wasted work.
var outdatedPods int
var outdatedPodSets int
for i := range roleSet.Spec.Roles {
role := &roleSet.Spec.Roles[i]
matchLabels, ok := getTopologyMatchLabels(roleSet, role.Name, roleSet.Spec.TopologyPolicy)
pkg/controller/roleset/roleset_controller.go:149
- Calling
emitTopologyPolicyPendingReplacementEventfromReconcileadds per-role List calls for Pods and (optionally) PodSets on every reconcile whenspec.topologyPolicyis set.syncPodsalready lists the same role Pods/PodSets during reconciliation (e.g.,rolesyncer.gocallsgetRolePods), so this can noticeably increase API-server load as role/replica counts grow. Consider gating this to only run whenTopologyPolicychanges (persist an observed hash/value on the RoleSet), or emit the event from within the existing syncers where pod/podset lists are already available.
if err := r.emitTopologyPolicyPendingReplacementEvent(ctx, roleSet); err != nil {
managedErrors = append(managedErrors, fmt.Errorf("emit topology policy event error %v", err))
}
Pull Request Description
This PR adds
topologyPolicytoRoleSetSpec, which is also available throughStormService.spec.template.spec. The policy lets operators request pod co-location by injecting Kubernetes pod affinity into both directly managed Pods and generated PodSet templates.This is the generalized form of the earlier
exclusiveTopologyproposal in #1842. Instead of only supporting a hard per-RoleSet topology constraint, the API now supports multiple co-location scopes and both soft and hard scheduling behavior.The policy has three fields:
scope: co-location granularity. Supported values areStormService,RoleSet, andRole.key: Kubernetes topology label key used for co-location, such askubernetes.io/hostnameortopology.kubernetes.io/zone.mode: scheduling strength.Preferredadds soft affinity and is the default;Requiredadds hard required affinity.Example:
With
scope: RoleSet, pods within the same RoleSet prefer to schedule on the same topology domain. This covers the Prefill/Decode same-host colocation use case from #1290 while avoiding scheduling failures by default when a single host or zone cannot satisfy the request. Users can setmode: Requiredwhen strict co-location is needed.For roles using
podGroupSize > 1, the same affinity is applied to the generated PodSet template, so PodSet-managed pods receive the same topology behavior as directly managed pods.Topology policy updates do not mutate affinity on already-created Pods because Pod affinity is immutable after Pod creation. Existing Pods and PodSet templates will pick up the new affinity after replacement or recreation.
Related Issues
Resolves: #1842
Related: #1290
Testing
go test -count=1 ./pkg/controller/roleset -run 'Test(GetTopologyMatchLabels|InjectTopologyAffinity|CreatePodSetForRole_TopologyPolicyDefaultsToPreferred|EmitTopologyPolicyPendingReplacementEvent)'go test -count=1 ./pkg/controller/podset ./api/orchestration/v1alpha1go test -count=1 ./test/integration/controller -run '^$'git diff --check