Add FlavorAssignmentStrategy or flavor priorities to ClusterQueue

[alculquicondor]

What would you like to be added:

A field in ClusterQueueSpec to influence how flavors are selected when there is already some quota in use.

Why is this needed:

Currently, we strictly go in the order determined by the ClusterQueue resource. But cluster administrators might need different optimization criteria, for example:

• InOrder: current behavior. It could be used to minimize cost. Cheaper resources can be listed first.
• MinimizeBorrowing: Choose the flavor that minimizes borrowing. Since a flavor could involve multiple resources, they borrowing can be expressed as percentage of the total request. This can be used to minimize disruptions once preemption is supported.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

/kind api-change

[kerthcet]

It seems like it's part of #171, I'd like to take a look so we can release 0.2.0 soon.
/assign

[kerthcet]

Sorry, it actually should be #328, but I can also take a look of this.

[alculquicondor]

This is not part of #328. Yes, it will need to be validated, but we don't have this field yet.

[kerthcet]

I thought about it a little, I think minimizeBorrowing should be the default policy we should follow, we can borrow resources but we should avoid it if possible for sharing between clusterQueues can be break up.

Besides, rather than prioritize the flavors by the slice order, maybe we should a new field priority, this is a more common style and easy to organize the configurations. And yes, this should also be a default policy.

I don't know whether we should introduce policies like binpack/spread/balanced to FlavorAssignmentStrategy(or FlavorManageStrategy), if we do, it seems we reinvent the wheels in scheduling. I think we can let this fly for a while until community users really need it.

[kerthcet]

cc @alculquicondor for thoughts

[alculquicondor]

We can only change the default behavior if we introduce a new API version. We can probably do it as we launch v1beta1.

maybe we should a new field priority

I don't think that adds much value. Going in order is a clear API.

I don't know whether we should introduce policies like binpack/spread/balanced to FlavorAssignmentStrategy

As long as there is a valid use case, we can consider it. Otherwise, I would start with the obvious ones: InOrder, MinimizeBorrowing.

[kerthcet]

maybe we should a new field priority

I don't think that adds much value. Going in order is a clear API.

They can't express the meaning that two flavors are of the same priority. This is useful when we want to apply more policies on flavors, like minimizeBorrowing by priority.

But as you say, let's wait to see if we really need them.

[alculquicondor]

but if you use minimizeBorrowing, we would first sort by how much they borrow. If there are multiple flavors that borrow the same (or borrow nothing), we fallback to the order in the list. It is comprehensive enough, in my opinion.

[kerthcet]

I think priority should be the first decision factor, then we apply other policies.
Whatever, I'll try to implement these two strategies as mentioned and evolve gradually.

[alculquicondor]

I see what you mean. You want to do minimizeBorrowing within a group of flavors. Then, if it doesn't fit, go to the next group of flavors which have lower priority.

To keep some level of "backwards compatibility", we can add a validation rule that priorities can only be decreasing in the list.