feat(api): Add PodTemplateOverrides API into TrainJob by xigang · Pull Request #2785 · kubeflow/trainer

xigang · 2025-08-09T14:56:47Z

What this PR does / why we need it:

Follow-up: #2784

xigang · 2025-08-09T15:01:01Z

/cc @andreyvelich @tenzen-y @mimowo

google-oss-prow · 2025-08-09T15:01:06Z

@xigang: GitHub didn't allow me to request PR reviews from the following users: mimowo.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @andreyvelich @tenzen-y @mimowo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

astefanutti · 2025-08-11T13:03:06Z

/ok-to-test

coveralls · 2025-08-11T13:07:59Z

Pull Request Test Coverage Report for Build 18348353117

Details

48 of 48 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 54.488%

Totals
Change from base Build 18314017966:	0.3%
Covered Lines:	1214
Relevant Lines:	2228

💛 - Coveralls

xigang · 2025-08-11T23:59:28Z

@andreyvelich @tenzen-y PTAL. thanks!

andreyvelich

Thank you for this @xigang!

xigang · 2025-08-12T09:30:03Z

Alright, I’ll go ahead and update the code.

astefanutti · 2025-08-12T15:33:54Z

/ok-to-test

xigang · 2025-08-13T03:01:21Z

@andreyvelich @astefanutti The code has been updated. Please take another look, thanks.

andreyvelich

/ok-to-test

andreyvelich · 2025-10-08T00:43:09Z

Awesome, appreciate your time @xigang!

xigang · 2025-10-08T03:13:13Z

@andreyvelich @tenzen-y @mimowo PTAL.

astefanutti

Do we want to update the section about PodSpecOverrides in docs/proposals/2170-kubeflow-trainer-v2/README.md accordingly?

mimowo

LGTM, thank you so much @xigang and maintainers, happy to see it will address kubernetes-sigs/kueue#7156 in 2.1 for Kubeflow and 0.15 in Kueue 👍

xigang · 2025-10-08T07:42:22Z

Do we want to update the section about PodSpecOverrides in docs/proposals/2170-kubeflow-trainer-v2/README.md accordingly?

@astefanutti Thanks for the reminder. I’ll submit a separate PR for the PodSpecOverrides section in the 2170 proposal.

xigang · 2025-10-08T07:49:18Z

/retest

astefanutti · 2025-10-08T08:15:09Z

/lgtm

Thanks @xigang!

andreyvelich · 2025-10-08T14:00:57Z

+				if len(metadata) > 0 {
+					podTemplatePatch["metadata"] = metadata
+				}


Do we need this check ?

The check should be kept as it prevents adding empty metadata objects to the strategic merge patch. It's a defensive programming approach that ensures we only modify the patch when there's actual metadata to apply.

andreyvelich · 2025-10-08T14:02:17Z

-										{
-											Name:  "NEW_VALUE",
-											Value: "from_overrides",
+							TargetJobs: []trainer.PodTemplateOverrideTargetJob{{Name: constants.Node}},


Please also add labels/annotations to the this test case, to verify it is working.

Signed-off-by: xigang <wangxigang2014@gmail.com>

andreyvelich

Thanks for this awesome contribution @xigang!
/lgtm
/approve
/hold for @tenzen-y to lgtm

astefanutti · 2025-10-08T14:52:27Z

/lgtm

tenzen-y

Thank you for this great effort!
/lgtm
/approve

/hold cancel

google-oss-prow · 2025-10-09T10:07:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: xigang <wangxigang2014@gmail.com>

google-oss-prow Bot requested review from jinchihe and kuizhiqing August 9, 2025 14:56

google-oss-prow Bot added the size/L label Aug 9, 2025

xigang changed the title ~~Add Labels and Annotations to PodSpecOverrides~~ feat: add labels and annotations to PodSpecOverrides Aug 9, 2025

google-oss-prow Bot requested review from andreyvelich and tenzen-y August 9, 2025 15:01

astefanutti reviewed Aug 11, 2025

View reviewed changes

Comment thread pkg/runtime/core/trainingruntime.go Outdated

google-oss-prow Bot added the ok-to-test label Aug 11, 2025

andreyvelich reviewed Aug 12, 2025

View reviewed changes

Comment thread pkg/runtime/core/trainingruntime.go Outdated

Comment thread pkg/runtime/core/trainingruntime.go

xigang force-pushed the override_label_and_annotation branch from e1f8c2e to b5cec0f Compare August 12, 2025 15:23

andreyvelich reviewed Aug 13, 2025

View reviewed changes

kannon92 reviewed Aug 13, 2025

View reviewed changes

Comment thread pkg/apis/trainer/v1alpha1/trainjob_types.go Outdated

andreyvelich reviewed Aug 13, 2025

View reviewed changes

Comment thread pkg/runtime/core/trainingruntime.go Outdated

xigang force-pushed the override_label_and_annotation branch 2 times, most recently from 18651dd to 41371bb Compare August 14, 2025 00:49

astefanutti reviewed Aug 14, 2025

View reviewed changes

Comment thread pkg/runtime/core/trainingruntime.go Outdated

xigang force-pushed the override_label_and_annotation branch 2 times, most recently from e09ce3f to bdb0e55 Compare August 14, 2025 09:26

andreyvelich reviewed Aug 14, 2025

View reviewed changes

Comment thread pkg/runtime/core/trainingruntime.go Outdated

xigang force-pushed the override_label_and_annotation branch from bdb0e55 to 1f50738 Compare August 14, 2025 09:35

andreyvelich reviewed Aug 14, 2025

View reviewed changes

Comment thread pkg/runtime/core/trainingruntime.go Outdated

xigang force-pushed the override_label_and_annotation branch from 1f50738 to aebc0d3 Compare August 14, 2025 09:55

xigang force-pushed the override_label_and_annotation branch from 448de4f to 888dd64 Compare October 8, 2025 02:44

astefanutti reviewed Oct 8, 2025

View reviewed changes

Comment thread test/integration/webhooks/trainjob_test.go Outdated

mimowo reviewed Oct 8, 2025

View reviewed changes

xigang force-pushed the override_label_and_annotation branch from 888dd64 to 546331a Compare October 8, 2025 07:43

google-oss-prow Bot assigned astefanutti Oct 8, 2025

google-oss-prow Bot added the lgtm label Oct 8, 2025

xigang mentioned this pull request Oct 8, 2025

feat: Add PodTemplateOverrides into TrainJob V2 API #2882

Merged

andreyvelich reviewed Oct 8, 2025

View reviewed changes

feat(api): Add PodTemplateOverrides API into TrainJob

cbf2266

Signed-off-by: xigang <wangxigang2014@gmail.com>

xigang force-pushed the override_label_and_annotation branch from 546331a to cbf2266 Compare October 8, 2025 14:37

google-oss-prow Bot removed the lgtm label Oct 8, 2025

andreyvelich reviewed Oct 8, 2025

View reviewed changes

google-oss-prow Bot added do-not-merge/hold lgtm approved labels Oct 8, 2025

tenzen-y reviewed Oct 9, 2025

View reviewed changes

google-oss-prow Bot removed the do-not-merge/hold label Oct 9, 2025

google-oss-prow Bot assigned tenzen-y Oct 9, 2025

google-oss-prow Bot merged commit d56715c into kubeflow:master Oct 9, 2025
32 of 34 checks passed

mimowo mentioned this pull request Oct 10, 2025

Extend the PodTemplateOverrides structure with "manager" field #2856

Closed

alexxfan pushed a commit to red-hat-data-services/trainer that referenced this pull request Nov 24, 2025

feat(api): Add PodTemplateOverrides API into TrainJob (kubeflow#2785)

d870ccb

Signed-off-by: xigang <wangxigang2014@gmail.com>

mahdikhashan pushed a commit to mahdikhashan/trainer that referenced this pull request Dec 29, 2025

feat(api): Add PodTemplateOverrides API into TrainJob (kubeflow#2785)

1448089

Signed-off-by: xigang <wangxigang2014@gmail.com>

astefanutti mentioned this pull request Mar 10, 2026

feat(api): Replace PodTemplateOverrides with RuntimePatches API #3199

Closed

Conversation

xigang commented Aug 9, 2025

Uh oh!

xigang commented Aug 9, 2025

Uh oh!

google-oss-prow Bot commented Aug 9, 2025

Uh oh!

Uh oh!

astefanutti commented Aug 11, 2025

Uh oh!

coveralls commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18348353117

Details

💛 - Coveralls

Uh oh!

xigang commented Aug 11, 2025

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xigang commented Aug 12, 2025

Uh oh!

astefanutti commented Aug 12, 2025

Uh oh!

xigang commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreyvelich commented Oct 8, 2025

Uh oh!

xigang commented Oct 8, 2025

Uh oh!

astefanutti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mimowo left a comment

Choose a reason for hiding this comment

Uh oh!

xigang commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xigang commented Oct 8, 2025

Uh oh!

astefanutti commented Oct 8, 2025

Uh oh!

andreyvelich Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

xigang Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

xigang Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

astefanutti commented Oct 8, 2025

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow Bot commented Oct 9, 2025

Uh oh!

Uh oh!

Reviewers

coveralls commented Aug 11, 2025 •

edited

Loading

xigang commented Aug 13, 2025 •

edited

Loading

xigang commented Oct 8, 2025 •

edited

Loading