Skip to content

Commit 542c80a

Browse files
joshuakguojoshuakguo
committed
refactor to store resolved placement group in memory instead of in the nodeclass status
1 parent 6f349d3 commit 542c80a

File tree

44 files changed

+507
-585
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+507
-585
lines changed

‎charts/karpenter-crd/templates/karpenter.k8s.aws_ec2nodeclasses.yaml‎

Lines changed: 0 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -831,50 +831,6 @@ spec:
831831
instanceProfile:
832832
description: InstanceProfile contains the resolved instance profile for the role
833833
type: string
834-
placementGroups:
835-
description: PlacementGroups contains the placement group values that are available to this NodeClass.
836-
items:
837-
properties:
838-
id:
839-
description: The id for the placement group.
840-
pattern: ^pg-[0-9a-z]+$
841-
type: string
842-
name:
843-
description: The name for the placement group.
844-
type: string
845-
partitionCount:
846-
description: The partition count for the partition placement group.
847-
format: int32
848-
type: integer
849-
spreadLevel:
850-
description: The spread level for the spread placement group.
851-
enum:
852-
- host
853-
- rack
854-
type: string
855-
state:
856-
default: available
857-
description: The state of the placement group.
858-
enum:
859-
- available
860-
- pending
861-
- deleting
862-
- deleted
863-
type: string
864-
strategy:
865-
description: The strategy for the placement group.
866-
enum:
867-
- cluster
868-
- partition
869-
- spread
870-
type: string
871-
required:
872-
- id
873-
- name
874-
- state
875-
- strategy
876-
type: object
877-
type: array
878834
securityGroups:
879835
description: |-
880836
SecurityGroups contains the current security group values that are available to the

‎cmd/controller/main.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ func main() {
4040
op.AMIProvider,
4141
op.SecurityGroupProvider,
4242
op.CapacityReservationProvider,
43+
op.PlacementGroupProvider,
4344
op.InstanceTypeStore,
4445
)
4546
overlayUndecoratedCloudProvider := metrics.Decorate(awsCloudProvider)
@@ -61,7 +62,7 @@ func main() {
6162
overlayUndecoratedCloudProvider,
6263
clusterState,
6364
op.InstanceTypeStore,
64-
corecontrollers.WithRegistrationHook(registrationhooks.NewPlacementGroupRegistrationHook(op.GetClient(), op.InstanceProvider)),
65+
corecontrollers.WithRegistrationHook(registrationhooks.NewPlacementGroupRegistrationHook(op.GetClient(), op.InstanceProvider, op.PlacementGroupProvider)),
6566
)...).
6667
WithControllers(ctx, controllers.NewControllers(
6768
ctx,

‎designs/placement-groups-support.md‎

Lines changed: 12 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,7 @@ Below lists the non-goals for _this RFC design._ Each of these items represents
278278
- Add a new struct under `spec` for `placementGroupSelector` to `EC2NodeClass` for defining which Placement Group to be used for a specific `EC2NodeClass`
279279
- Each EC2NodeClass maps to exactly one Placement Group
280280
- The struct accepts either a placement group name or id as a string value
281-
- Add a new field under `status` for the resolved Placement Group by the `spec.placementGroupSelector` for the `EC2NodeClass`
281+
- The resolved placement group details are stored **in-memory** by the placement group provider
282282

283283
```yaml
284284
apiVersion: karpenter.k8s.aws/v1
@@ -300,26 +300,17 @@ status:
300300
# The EC2NodeClass is not ready if this condition is False,
301301
# blocking all launches from this NodeClass.
302302
type: PlacementGroupReady
303-
# placementGroup contains the resolved placement group details.
304-
placementGroup:
305-
# Id for the Placement Group
306-
id: String
307-
# Name of the Placement Group
308-
name: String
309-
# Number of partitions for partition Placement Groups
310-
partitionCount: int | None
311-
# Spread level for spread Placement Groups {host, rack}
312-
spreadLevel: String | None
313-
# State of the placement group
314-
# {pending, available, deleting, deleted}
315-
# Karpenter sets PlacementGroupReady to False for any state other than "available"
316-
state: String
317-
# Strategy of the placement group
318-
# {cluster, partition, spread}
319-
strategy: String
320303
```
321304

322-
This API closely follows how [DescribePlacementGroups](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribePlacementGroups.html) can filter placement groups -- allowing Karpenter to receive the server-side filtered version of the placement groups to store in its status.
305+
The placement group provider resolves placement groups from the EC2 `DescribePlacementGroups` API and stores the result in-memory, keyed by nodeclass name. The resolved placement group contains the following fields:
306+
307+
- **ID**: Placement group ID (e.g., `"pg-0123456789abcdef0"`)
308+
- **Name**: Placement group name
309+
- **PartitionCount**: Number of partitions (partition strategy only)
310+
- **SpreadLevel**: Spread level (`"rack"` or `"host"`, spread strategy only)
311+
- **Strategy**: Placement group strategy (`"cluster"`, `"partition"`, or `"spread"`)
312+
313+
All consumers (drift detection, offering resolution, launch template creation, etc.) read from the provider's in-memory store. The nodeclass reconciler is responsible for calling the provider to resolve and store the placement group on each reconciliation loop, ensuring the in-memory state is always fresh.
323314

324315
### Labels
325316

@@ -447,7 +438,7 @@ For partition placement groups, the `karpenter.k8s.aws/placement-group-partition
447438

448439
1. When a NodeClaim is created for an EC2NodeClass with a partition placement group, the instance is launched via `CreateFleet` without specifying a `PartitionNumber`, allowing EC2 to auto-assign the partition.
449440
2. During node registration, before the `karpenter.sh/unregistered` taint is removed, the `PlacementGroupRegistrationHook` runs as part of the NodeClaim lifecycle controller's registration phase.
450-
3. The hook resolves the EC2NodeClass from the NodeClaim's `spec.nodeClassRef` and checks `status.placementGroup` to determine if this is a partition placement group. If not, the hook passes through immediately.
441+
3. The hook resolves the EC2NodeClass from the NodeClaim's `spec.nodeClassRef` and queries the placement group provider's in-memory store to determine if this is a partition placement group. If not, the hook passes through immediately.
451442
4. For partition placement groups, the hook calls `DescribeInstances` to discover the EC2-assigned partition number from `Placement.PartitionNumber`.
452443
5. Once the partition number is available, the hook sets the `karpenter.k8s.aws/placement-group-partition` label on the NodeClaim and allows registration to proceed. The label is then synced to the Node as part of the normal registration sync.
453444
6. If the partition number is not yet available (e.g., the instance is still initializing), the hook returns `false`, causing the lifecycle controller to requeue after 1 second and retry.
@@ -522,7 +513,7 @@ Placement groups do not directly affect pricing, but they constrain the set of v
522513

523514
## Drift
524515

525-
Nodes are marked as drifted when their placement group ID label (`karpenter.k8s.aws/placement-group-id`) no longer matches the EC2NodeClass's resolved placement group ID (`status.placementGroup.id`). This is checked explicitly in the `isPlacementGroupDrifted` function, which compares the NodeClaim's `placement-group-id` label against the resolved PG ID from the NodeClass status.
516+
Nodes are marked as drifted when their placement group ID label (`karpenter.k8s.aws/placement-group-id`) no longer matches the EC2NodeClass's resolved placement group ID stored in the placement group provider's in-memory store. This is checked explicitly in the `isPlacementGroupDrifted` function, which compares the NodeClaim's `placement-group-id` label against the resolved PG ID from the provider.
526517

527518
| Scenario | Detection | Recovery |
528519
|----------|-----------|----------|

‎hack/docs/instancetypes_gen/main.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ below are the resources available with some assumptions and after the instance o
143143
true,
144144
),
145145
nil,
146+
nil,
146147
awscache.NewUnavailableOfferings(),
147148
instancetype.NewDefaultResolver(
148149
region,

‎hack/tools/allocatable_diff/main.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ func main() {
7979
op.AMIProvider,
8080
op.SecurityGroupProvider,
8181
op.CapacityReservationProvider,
82+
op.PlacementGroupProvider,
8283
op.InstanceTypeStore,
8384
)
8485
instanceTypes := lo.Must(cloudProvider.GetInstanceTypes(ctx, nil))

‎hack/tools/launchtemplate_counter/main.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ func main() {
7070
true,
7171
),
7272
nil,
73+
nil,
7374
awscache.NewUnavailableOfferings(),
7475
instancetype.NewDefaultResolver(
7576
region,
@@ -140,7 +141,7 @@ func main() {
140141
fmt.Printf("Got %d instance types after filtering\n", len(instanceTypes))
141142

142143
resolver := amifamily.NewDefaultResolver(region)
143-
launchTemplates, err := resolver.Resolve(nodeClass, &karpv1.NodeClaim{}, lo.Slice(instanceTypes, 0, 60), karpv1.CapacityTypeOnDemand, string(ec2types.TenancyDefault), &amifamily.Options{InstanceStorePolicy: lo.ToPtr(v1.InstanceStorePolicyRAID0)})
144+
launchTemplates, err := resolver.Resolve(nodeClass, &karpv1.NodeClaim{}, lo.Slice(instanceTypes, 0, 60), karpv1.CapacityTypeOnDemand, string(ec2types.TenancyDefault), &amifamily.Options{InstanceStorePolicy: lo.ToPtr(v1.InstanceStorePolicyRAID0)}, "", 0)
144145

145146
if err != nil {
146147
log.Fatalf("resolving launchTemplates, %s", err)

‎kwok/cloudprovider/cloudprovider.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ import (
2626
"github.com/aws/karpenter-provider-aws/pkg/providers/capacityreservation"
2727
"github.com/aws/karpenter-provider-aws/pkg/providers/instance"
2828
"github.com/aws/karpenter-provider-aws/pkg/providers/instancetype"
29+
"github.com/aws/karpenter-provider-aws/pkg/providers/placementgroup"
2930
"github.com/aws/karpenter-provider-aws/pkg/providers/securitygroup"
3031
)
3132

@@ -41,6 +42,7 @@ func New(
4142
amiProvider amifamily.Provider,
4243
securityGroupProvider securitygroup.Provider,
4344
capacityReservationProvider capacityreservation.Provider,
45+
placementGroupProvider placementgroup.Provider,
4446
instanceTypeStore *nodeoverlay.InstanceTypeStore,
4547
) *CloudProvider {
4648
return &CloudProvider{
@@ -52,6 +54,7 @@ func New(
5254
amiProvider,
5355
securityGroupProvider,
5456
capacityReservationProvider,
57+
placementGroupProvider,
5558
instanceTypeStore,
5659
),
5760
}

‎kwok/main.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ func main() {
4040
op.AMIProvider,
4141
op.SecurityGroupProvider,
4242
op.CapacityReservationProvider,
43+
op.PlacementGroupProvider,
4344
op.InstanceTypeStore,
4445
)
4546
overlayUndecoratedCloudProvider := metrics.Decorate(kwokAWSCloudProvider)

‎kwok/operator/operator.go‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,10 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
147147
lo.Must0(versionProvider.UpdateVersion(ctx))
148148
ssmProvider := ssmp.NewDefaultProvider(ssm.NewFromConfig(cfg), ssmCache)
149149
amiProvider := amifamily.NewDefaultProvider(operator.Clock, versionProvider, ssmProvider, ec2api, cache.New(awscache.DefaultTTL, awscache.DefaultCleanupInterval))
150+
placementGroupProvider := placementgroup.NewProvider(
151+
ec2api,
152+
cache.New(awscache.PlacementGroupTTL, awscache.DefaultCleanupInterval),
153+
)
150154
amiResolver := amifamily.NewDefaultResolver(cfg.Region)
151155
launchTemplateProvider := launchtemplate.NewDefaultProvider(
152156
ctx,
@@ -156,6 +160,7 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
156160
amiResolver,
157161
securityGroupProvider,
158162
subnetProvider,
163+
placementGroupProvider,
159164
lo.Must(GetCABundle(ctx, operator.GetConfig())),
160165
operator.Elected(),
161166
kubeDNSIP,
@@ -167,10 +172,6 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
167172
cache.New(awscache.DefaultTTL, awscache.DefaultCleanupInterval),
168173
cache.New(awscache.CapacityReservationAvailabilityTTL, awscache.DefaultCleanupInterval),
169174
)
170-
placementGroupProvider := placementgroup.NewProvider(
171-
ec2api,
172-
cache.New(awscache.PlacementGroupTTL, awscache.DefaultCleanupInterval),
173-
)
174175
instanceTypeProvider := instancetype.NewDefaultProvider(
175176
cache.New(awscache.InstanceTypesZonesAndOfferingsTTL, awscache.DefaultCleanupInterval),
176177
cache.New(awscache.InstanceTypesZonesAndOfferingsTTL, awscache.DefaultCleanupInterval),
@@ -179,6 +180,7 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
179180
subnetProvider,
180181
pricingProvider,
181182
capacityReservationProvider,
183+
placementGroupProvider,
182184
unavailableOfferingsCache,
183185
instancetype.NewDefaultResolver(cfg.Region),
184186
)
@@ -195,6 +197,7 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
195197
subnetProvider,
196198
launchTemplateProvider,
197199
capacityReservationProvider,
200+
placementGroupProvider,
198201
cache.New(awscache.DefaultTTL, awscache.DefaultCleanupInterval),
199202
)
200203

‎pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml‎

Lines changed: 0 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -828,50 +828,6 @@ spec:
828828
instanceProfile:
829829
description: InstanceProfile contains the resolved instance profile for the role
830830
type: string
831-
placementGroups:
832-
description: PlacementGroups contains the placement group values that are available to this NodeClass.
833-
items:
834-
properties:
835-
id:
836-
description: The id for the placement group.
837-
pattern: ^pg-[0-9a-z]+$
838-
type: string
839-
name:
840-
description: The name for the placement group.
841-
type: string
842-
partitionCount:
843-
description: The partition count for the partition placement group.
844-
format: int32
845-
type: integer
846-
spreadLevel:
847-
description: The spread level for the spread placement group.
848-
enum:
849-
- host
850-
- rack
851-
type: string
852-
state:
853-
default: available
854-
description: The state of the placement group.
855-
enum:
856-
- available
857-
- pending
858-
- deleting
859-
- deleted
860-
type: string
861-
strategy:
862-
description: The strategy for the placement group.
863-
enum:
864-
- cluster
865-
- partition
866-
- spread
867-
type: string
868-
required:
869-
- id
870-
- name
871-
- state
872-
- strategy
873-
type: object
874-
type: array
875831
securityGroups:
876832
description: |-
877833
SecurityGroups contains the current security group values that are available to the

0 commit comments

Comments
 (0)