☂️ Integration with WAS in kube-scheduler

[mimowo]

What would you like to be added:

Make sure Kueue is compatible with WAS, and can delegate node-level scheduling to kube-scheduler.

This is an umbrella issue with a lot of work we need to identify as WAS is graduating.

Some of the related items:

• TAS: Better accuracy of scheduling by tighter integration with kube-scheduler #3755
• WAS integration: run e2e tests using "main" kubernetes to proactively ensure compatibility #8858

Why is this needed:

To leverage kube-scheduler primitives for:

• Gang Scheduling
• Dynamic Resource Allocation
• Topology-Aware Scheduling

[mimowo]

cc @tenzen-y @gabesaba @sohankunkerkar @kannon92

[kannon92]

For those that are interested, there is a weekly call on Mondays at 1 pm EST to sync on WAS.

Right now my hope is to get workload controller support for gangs but there is a lot of changes for WAS.

[kannon92]

One area we may want to explore is using WAS to enforce gang scheduling for bare pods integration (https://kueue.sigs.k8s.io/docs/tasks/run/plain_pods/#running-a-group-of-pods-to-be-admitted-together).

I think most controllers should handle gang scheduling in their own way but this is the one integration that kueue may need to create its own PodGroups/Workloads so that the grouping of pods can be gang scheduled together.

[tenzen-y]

One area we may want to explore is using WAS to enforce gang scheduling for bare pods integration (https://kueue.sigs.k8s.io/docs/tasks/run/plain_pods/#running-a-group-of-pods-to-be-admitted-together).

I think most controllers should handle gang scheduling in their own way but this is the one integration that kueue may need to create its own PodGroups/Workloads so that the grouping of pods can be gang scheduled together.

Yes, exactly. But the discussion point is how to handle if controllers create PodGroups / Workloads.
We might be able to just skip handling such a group of Pods if we are not missing any stuff.

[44past4]

I've drafted a proposal outlining the next steps for the WAS and Kueue integration: https://docs.google.com/document/d/13I3bMY-abBtsWmDhCl4RdDHKchHeB3G8zMSTpplmxHA/edit?usp=sharing

Please take a look and leave your feedback directly in the document.

[mimowo]

cc @tenzen-y @gabesaba @kannon92 @sohankunkerkar ptal