scheduler: support prefer node hint

[saintube]

Ⅰ. Describe what this PR does

Make the koord-scheduler support the scheduling hint of the preferred nodes.

Introduce the PreferNodesPlugin in the FrameworkExtender, allowing the plugin to provide a node list that the scheduling pipeline first tries to filter before entering the regular findNodesThatPassFilters. It introduces the overhead of once RunFilterPluginsWithNominatedPlugins, which should be acceptable.

In contrast to the .status.nominatedNodeName, the hint uses a list of nodes, and the hinted pods do not account for the nodes before assuming.

Compatibilty:

When the user also enables the network-topology-aware scheduling and generates the PreFilterResult by the FindOneNode plugin according to the best plan, the preference hint is ignored.

The SchedulingHint plugin also considers the intersection of the original PreFilterResult. When the preferred nodes are not included in the PreFilterResult, the scheduler does not break the constraint of the existing PreFilter, but aborts the hint instead.

Ⅱ. Does this pull request fix one issue?

In the scenario of job queueing, the queue component may have some suggestions about which nodes the dequeued pods should try to schedule first. The patch adds the api for the queue to hint the scheduler for the preferred node when it does not want the pod to be counted on the nominated node before the scheduler predicates. We hint at a node list instead of a single node, which serves as a backup in case the most preferred one fails for some reason.

Ⅲ. Describe how to verify it

1. Set the scheduling hint annotations for the Pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    internal.scheduling.koordinator.sh/scheduling-hint: '{"preferredNodeNames": ["most-prefer-node-name", "second-prefer-node-name"]}'

2. Then check the scheduler's log if the PreferOneNodePlugin runs and returns a result.

Ⅳ. Special notes for reviews

V. Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in make test

[koordinator-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign zimengsheng after the PR has been reviewed.
You can assign the PR to them by writing /assign @zimengsheng in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• apis/OWNERS
• pkg/scheduler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 86.95652% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.94%. Comparing base (1399086) to head (2fcc579).

Files with missing lines	Patch %	Lines
pkg/scheduler/frameworkext/framework_extender.go	76.92%	8 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2787      +/-   ##
==========================================
+ Coverage   67.91%   67.94%   +0.03%     
==========================================
  Files         513      513              
  Lines       52283    52345      +62     
==========================================
+ Hits        35509    35568      +59     
- Misses      13724    13727       +3     
  Partials     3050     3050              

Flag	Coverage Δ
unittests	67.94% <86.95%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ZiMengSheng]

A very useful building block. However, I'd like to understand the queue component's requirements for us:

1. When preemption is needed, should it also prioritize preemption on certain nodes?
2. When calculating network topology, should it also prioritize trying on certain nodes first?

[saintube]

A very useful building block. However, I'd like to understand the queue component's requirements for us:

1. When preemption is needed, should it also prioritize preemption on certain nodes?
2. When calculating network topology, should it also prioritize trying on certain nodes first?

1. It is a good point. Like the nominatedNodeName in the vanilla kube-scheduler, the preference to filter is not the same as the preference for preempt. However, I believe it can be helpful if we give a chance to try some candidate nodes first when the pods are dequeued for an intended preemption. The problem is that there is no unified interface for the DryRunPreemption, so it requires us to implement this ability inside preemption plugins.
2. For the network-topology-aware scheduling, I suppose the current offering is enough. If the queue wants to nominate job pods with nominated nodes according to the topology constraints, it is OK to use this api or the nominatedNodeName. If the calculation of the NTW is to be done by the scheduler, we do not need to set any preferred node.