Support for CPU allocation in distributed mode

[ffromani]

When operating in grouped mode, we allocate CPUs using a packed algorithm borrowed from kubelet's cpumanager. We should support also the distributed allocation algorithm, in order to have feature parity with cpumanager. The simplest way is a dracpu option to change the allocation, like cpumanager does. A better approach would be allow each workload to pick and choose.

[ffromani]

/assign

[ffromani]

We can currently do manual split of CPUs to make sure they are allocated using different NUMA nodes:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: claim-cpu-capacity-20
spec:
  devices:
    requests:
    - name: numa0-cpus
      exactly:
        deviceClassName: dra.cpu
        capacity:
          requests:
            dra.cpu/cpu: "10"
        selectors:
        - cel:
            expression: device.attributes["dra.cpu"].numaNodeID == 0
    - name: numa1-cpus
      exactly:
        deviceClassName: dra.cpu
        capacity:
          requests:
            dra.cpu/cpu: "10"
        selectors:
        - cel:
            expression: device.attributes["dra.cpu"].numaNodeID ==1

(see: https://github.com/kubernetes-sigs/dra-driver-cpu/pull/47/changes#r2738021782)

However, this approach is manual and leaks assumptions about machine properties: it hardcodes the NUMA split and, differently from the cpumanager feature, won't automatically adapt if the same claim is handled by a 1-NUMA, 2-NUMA or 4-NUMA machine; the claim would need to be updated or recreated manually.

[AutuSnow]

@ffromani When I was browsing through the code today, I noticed a phenomenon where pkg/cpumanager/cpu_assignment.go:894 has implemented the takeByTopologYNUMADistributed algorithm, but it has not been exported and has never been called. When I came across this lssue, should we add a global dracpu command-line option to switch allocation policies (similar to cpumanager) or should we implement each workload to choose its own allocation policy? In my opinion, the second approach seems more suitable for this project. Can we implement this feature through YAML definition?

  apiVersion: resource.k8s.io/v1
  kind: ResourceClaim
  metadata:
    name: my-distributed-cpus
  spec:
    devices:
      config:
      - requests: ["cpus"]         #Optional, empty applies to all requests
        opaque:
          driver: dra.cpu
          parameters:
            apiVersion: dra.cpu/v1
            kind: CpuClaimConfig
            allocationPolicy: distributed   #'Packaged' (default) or 'distributed'
      requests:
      - name: cpus
        exactly:
          deviceClassName: dra.cpu
          capacity:
            requests:
              dra.cpu/cpu: "20"

[ffromani]

thanks @AutuSnow . This is an interesting approach worth investigating. From the top of my head, the problems here is the interplay with the device grouping mode and with the grouping criteria. Exposing the option to the workload can create impossible/error conditions which I'm not too happy about, so we need to evaluate this path.
Furthermore the semantics of the distributed allocation is not yet clear in the context of a DRA-based allocation.

The purpose of issues like this is to track the gaps between cpumanager and the DRA driver, because users migrating from cpumanager will very likely have these questions (some of them I have/had myself).
We should however avoid to reimplement cpumanager on top of DRA, but rather expose equivalent functionality in a DRA-native way, wherever and whenever makes sense. This is the general approach we should pursue, and the reason why we need, I believe, more research before we can confidently close this issue.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ffromani]

/remove-lifecycle stale