DRA: structured parameters

[pohly]

Enhancement Description

• One-line enhancement description (can be used as a release note): The original dynamic resource allocation (DRA) uses claim and class parameters that are opaque to Kubernetes. With structured parameters, kube-scheduler and Cluster Autoscaler can handle respectively simulate claim allocation themselves without relying on a third-party driver.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4381-dra-structured-parameters/

• Discussion Link: Batch WG meeting 20231207, DRA intro and plan in that meeting

• Primary contact (assignee): @pohly

• Responsible SIGs: SIG Node

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.30-1.31
  • Beta release target (x.y): 1.32-1.33
  • Stable release target (x.y): 1.34

• Alpha in 1.30

  • KEP (k/enhancements) update PR(s): KEP 4381: add structured parameters for dynamic resource allocation #4384
  • Code (k/k) update PR(s): DRA: structured parameters kubernetes#123516
  • Docs (k/website) update PR(s): DRA in Kubernetes 1.30: adds structured parameters website#45287

• Alpha in 1.31

  • KEP (k/enhancements) update PR(s): KEP-4381: DRA update for 1.31 #4709
  • Code (k/k) update PR(s): DRA for 1.31 kubernetes#125488
  • Docs (k/website) update PR(s): DRA documentation for 1.31 website#46816

• Beta in 1.32

  • KEP (k/enhancements) update PR(s): KEP-4381: DRA: PRR and API for beta of structured parameters in 1.32 #4869
  • Code (k/k) update PR(s):
    • DRA resourceclaims: maintain metric of total and allocated claims kubernetes#127661
    • DRA API: AdminAccess in DeviceRequestAllocationResult + DRAAdminAccess feature gate kubernetes#127266
    • DRA API: implement CEL cost limit kubernetes#128101
    • Kubelet: add DRA latency metrics kubernetes#127146
    • DRA resourceclaims: maintain metric of total and allocated claims kubernetes#127661
    • DRA 1.32 API: promotion to beta kubernetes#127511
  • Docs (k/website) update(s): DRA 1.32: beta website#48330

• Beta in 1.33

  • KEP (k/enhancements) update PR(s): KEP-4381: Update GA graduation criteria #5160
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

[pohly]

/sig node
/wg batch

The rationale behind creating a separate KEP is that this is an extension of DRA that (at least in theory) could graduate at its own pace. The KEP document itself also will be shorter, which will make discussing it easier.

[SergeyKanzhelev]

/stage alpha
/milestone v1.30

[klueska]

/milestone v1.30
/label lead-opted-in

[k8s-ci-robot]

@klueska: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.30
/label lead-opted-in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[johnbelamaric]

@SergeyKanzhelev can we get this added to the 1.30 milestone?

[johnbelamaric]

/label lead-opted-in

[k8s-ci-robot]

@klueska: Can not set label lead-opted-in: Must be member in one of these teams: [release-team-enhancements release-team-leads sig-api-machinery-leads sig-apps-leads sig-architecture-leads sig-auth-leads sig-autoscaling-leads sig-cli-leads sig-cloud-provider-leads sig-cluster-lifecycle-leads sig-contributor-experience-leads sig-docs-leads sig-instrumentation-leads sig-k8s-infra-leads sig-multicluster-leads sig-network-leads sig-node-leads sig-release-leads sig-scalability-leads sig-scheduling-leads sig-security-leads sig-storage-leads sig-testing-leads sig-windows-leads]

In response to this:

/milestone v1.30
/label lead-opted-in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SergeyKanzhelev]

/label lead-opted-in
/milestone v1.30

[salehsedghpour]

Hello @pohly 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 9th February 2024.

This enhancement is targeting for stage alpha for v1.30 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.30.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just want to update the following but it's not required for alpha stage and it will serve for future stages:

• Make sure that the KEP 4381: add structured parameters for dynamic resource allocation #4384 is merged to the k/enhancements repo.

The status of this enhancement is marked as At risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well, I couldn't find any related PR regarding this KEP in k/enhancements. Thank you!

[salehsedghpour]

With all the requirements fulfilled this enhancement is now marked as tracked for the upcoming enhancements freeze 🚀

[drewhagen]

Hello @pohly 👋, 1.30 Docs Lead here.

Does this enhancement work planned for 1.30 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.30 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday February 22nd 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[haircommander]

/label lead-opted-in
/milestone v1.33

[kannon92]

/stage beta

[dipesh-rawat]

Hello @pohly 👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage beta for v1.33 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.33.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• Update the kep.yaml to reflect the following:
  • Add latest-milestone: v1.33.

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[pacoxu]

#4381 updated the GA graduation criteria for DRA structured parameters, like below.

• At least 3 real-world driver implementations from different organizations
• Standardization of at least one attribute suitable for intra-node topology alignment across drivers (e.g., NUMA or PCIe root, as determined to be appropriate and useful).
• Allowing time for feedback
• Scalability and performance analysis done and published

@pohly @johnbelamaric IIUC, this is not going to be GA in v1.33, right?
And the current status that this is beta in v1.32 but default off. Do we plan to make it by default on in v1.33?

new updates in v1.33 until now:

• DRA: seamless driver upgrades kubernetes#129832
• DRA: CEL cost estimate during validation kubernetes#129749
• DRA CEL: add missing size estimator kubernetes#129661
• DRA API: bump maximum size of ReservedFor to 256 kubernetes#129543

[pohly]

We cannot make it default on in 1.33 because the API group has to remain default off (Kubernetes policy).

There will be some minor enhancements of this feature in 1.33, but nothing that needs to be planned now through a KEP update.

[klueska]

We talked in the last working group meeting about updating this KEP in this cycle to include more comprehensive GA graduation criteria.

[dipesh-rawat]

Hi @pohly 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #4381 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[johnbelamaric]

We talked in the last working group meeting about updating this KEP in this cycle to include more comprehensive GA graduation criteria.

Yes. I don't think it's critical to get it done before the KEP freeze though, it's just a punch list we need for 1.34, not changes going into 1.33 (some might, but they would be managed as separate issues). Some of them may need issues created.

[johnbelamaric]

So, @pacoxu in other words there are no major changes here, we just wanted to track that we are working on it incrementally.

[dipesh-rawat]

we just wanted to track that we are working on it incrementally.

@johnbelamaric, I’m wondering if we’ll still need a small change to the latest milestone (latest-milestone: "v1.33" in kep.yaml)) due to the incremental updates in v1.33 release cycle. Could you please confirm if that’s the case, or am I reading it wrong? Sorry for any confusion on my side!

[johnbelamaric]

I’m wondering if we’ll still need a small change to the latest milestone

Yes, makes sense to me, thanks. Coming soon :)

[dipesh-rawat]

Hello @pohly 👋, 1.33 Enhancements team here.

Now that PR #5160 has been merged, all the KEP requirements are in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

/label tracked/yes