feat: KEP-2437 - PodGroup Creation for Volcano Scheduler by Doris-xm · Pull Request #2729 · kubeflow/trainer

Doris-xm · 2025-07-14T01:01:11Z

What this PR does / why we need it:
This PR implements PodGroup creation and integration to support the Volcano Scheduler. It introduces a Volcano plugin that:

Creates PodGroup CR for each TrainJob based on the scheduling policy defined in TrainingRuntime.
Calculates and sets scheduling parameters:
- minMember: the total minimum number of Pods required to start the job.
- minTaskMember: minimum required replicas per PodSet (task).
- minResources: total resource request across all PodSets.
Sets optional scheduling hints, like queue, priorityClassName, and networkTopology, if explicitly defined.
Unit Tests

Which issue(s) this PR fixes :
Part of #2671

Checklist:

Docs included if any changes are user facing

Signed-off-by: Xinmin Du <2812493086@qq.com>

Doris-xm · 2025-07-14T01:02:26Z

/area gsoc

google-oss-prow · 2025-07-14T02:01:03Z

@Doris-xm: GitHub didn't allow me to request PR reviews from the following users: rudeigerc.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @Electronic-Waste @rudeigerc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Doris-xm · 2025-07-14T02:01:17Z

/cc @Electronic-Waste @rudeigerc

google-oss-prow · 2025-07-14T02:01:20Z

@Doris-xm: GitHub didn't allow me to request PR reviews from the following users: rudeigerc.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @Electronic-Waste @rudeigerc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Xinmin Du <2812493086@qq.com>

# Conflicts: # pkg/runtime/framework/plugins/registry.go

Signed-off-by: Xinmin Du <2812493086@qq.com>

Electronic-Waste

@Doris-xm Thanks for this. I left my very initial reviews.

Electronic-Waste · 2025-08-01T12:47:12Z

Can you rebase this branch to solve the conflicts? @Doris-xm

Electronic-Waste · 2025-08-01T14:25:19Z

/cc @kubeflow/kubeflow-trainer-team @astefanutti @rudeigerc @Monokaix @JesseStutler

Signed-off-by: Xinmin Du <2812493086@qq.com>

astefanutti

Only left a nit, otherwise LGTM once @tenzen-y comments will be addressed.

astefanutti · 2025-10-01T15:42:39Z

+	TrainingRuntimeContainerRuntimeClassKey                   = ".trainingRuntimeSpec.jobSetTemplateSpec.replicatedJobs.podTemplateSpec.runtimeClassName"
+	ClusterTrainingRuntimeContainerRuntimeClassKey            = ".clusterTrainingRuntimeSpec.jobSetTemplateSpec.replicatedJobs.podTemplateSpec.runtimeClassName"


These could probably be const.

It seems our tests need to intentionally override these keys to simulate missing indexers.

trainer/pkg/runtime/framework/core/framework_test.go

Lines 140 to 143 in b918411

if tc.emptyCoSchedulingIndexerTrainingRuntimeContainerRuntimeClassKey {

originTrainingRuntimeRuntimeKey := coscheduling.TrainingRuntimeContainerRuntimeClassKey

coscheduling.TrainingRuntimeContainerRuntimeClassKey = ""

t.Cleanup(func() {

Signed-off-by: Xinmin Du <2812493086@qq.com>

andreyvelich

Thanks for this effort @Doris-xm!
Overall lgtm
just small nit.

andreyvelich · 2025-10-02T20:26:05Z

+		})
+	}
+
+	// Test Validate()


I think for Validate test, we can have another function, like we did for torch:

trainer/pkg/runtime/framework/plugins/torch/torch_test.go

Line 1412 in 497a0b6

func TestValidate(t *testing.T) {

We don't have it for coscheduling, since we don't have any validation for that plugin.

Yes, I've tested it separately.

trainer/pkg/runtime/framework/plugins/volcano/volcano_test.go

Line 314 in 4030ce5

func TestValidate(t *testing.T) {

And the UT only have two fucntions, TestVolcano and TestValidate, now.

Signed-off-by: Xinmin Du <2812493086@qq.com>

andreyvelich · 2025-10-03T07:53:17Z

/lgtm
/approve
/hold for the final approval from @tenzen-y and @astefanutti

astefanutti · 2025-10-03T13:17:26Z

/lgtm

Thanks

Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com>

Signed-off-by: Xinmin Du <2812493086@qq.com>

…cano-podgroup-build

Signed-off-by: Xinmin Du <2812493086@qq.com>

andreyvelich

@tenzen-y @Doris-xm Are there any additional comments before we can merge this PR?

tenzen-y

Otherwise lgtm, thanks!

Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com>

tenzen-y

@Doris-xm Thank you 👍
/lgtm
/approve

I have seen @andreyvelich and @astefanutti added LGTM.

/hold cancel

google-oss-prow · 2025-10-06T05:50:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* feat: api for volcano scheduling plugin Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: init volcano-plugin Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: init test file Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: register volcano plugin Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: deal with minTaskMember, minMember, NetworkTopo Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: calculate of minResource Signed-off-by: Xinmin Du <2812493086@qq.com> * test: build PodGroup test Signed-off-by: Xinmin Du <2812493086@qq.com> * refactor: separate to 2 prs（build&handler） Signed-off-by: Xinmin Du <2812493086@qq.com> * test: add test for new&reconcile_builder Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: typo Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: trainer/v2 import Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: networktopo type Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: OpenAPI validation errors Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: remove minTaskMembers Signed-off-by: Xinmin Du <2812493086@qq.com> * test: test coverage 100% Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: update apis Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: replace testify Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: registry Volcano CRDs to the scheme Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: add volcano to scheme Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: fix networktopo schema Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: add networktopo spec in trainer Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: unit test Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: import networkTopo directly Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: make generate Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: make generate Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: golangci-lint Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: golangci-lint Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: add volcano installation in integration test Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: filter volcano api Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: get volcano.podgroup with local version Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: init test env with volcano podgroup installed Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: check plugin in enforcePodgroupPolicy Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: group-name label in unit test Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: ReconcilerBuilders Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: add PodGroupHandler Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: unit test for handlers Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: group name annotation Signed-off-by: Xinmin Du <2812493086@qq.com> * Update hack/swagger/main.go Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * fix: no need to delete RBAC Signed-off-by: Xinmin Du <2812493086@qq.com> * Update pkg/runtime/framework/plugins/volcano/indexer.go Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * fix: nil checking for trainjob Signed-off-by: Xinmin Du <2812493086@qq.com> * Update pkg/runtime/framework/plugins/volcano/volcano.go Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * fix: make generate Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: index conflict Signed-off-by: Xinmin Du <2812493086@qq.com> * Update pkg/runtime/framework/plugins/coscheduling/coscheduling.go Co-authored-by: Shao Wang <2690692950@qq.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * fix: update volcano to v1.12.2 Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: re-use indexer Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: add validate Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: no scheduler when coscheduling is nil Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: put group-name in annotations Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: validate if priorityClass installed Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: propagate annotations to pod Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: integration test for volcano Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: golangci-lint check Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: use shared indexer Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: remove indexer to runtime/ Signed-off-by: Xinmin Du <2812493086@qq.com> * Update hack/swagger/main.go Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * Update hack/swagger/main.go Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * fix: append owner reference & missing import Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: rewrite volcano UT Signed-off-by: Xinmin Du <2812493086@qq.com> * feat: add copyright Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: sync RBAC to Helm charts Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: refactor UTs Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: test validation separately Signed-off-by: Xinmin Du <2812493086@qq.com> * Update hack/swagger/main.go Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> * fix: refactor TestVolcano Signed-off-by: Xinmin Du <2812493086@qq.com> * fix: refactor TestValidate Signed-off-by: Xinmin Du <2812493086@qq.com> * Update pkg/runtime/framework/plugins/volcano/volcano_test.go Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com> --------- Signed-off-by: Xinmin Du <2812493086@qq.com> Signed-off-by: Du Xinmin <2812493086@qq.com> Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: Shao Wang <2690692950@qq.com> Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

Doris-xm added 9 commits June 29, 2025 14:33

feat: api for volcano scheduling plugin

8867e71

Signed-off-by: Xinmin Du <2812493086@qq.com>

feat: init volcano-plugin

0cd20fc

Signed-off-by: Xinmin Du <2812493086@qq.com>

feat: init test file

2e911ef

Signed-off-by: Xinmin Du <2812493086@qq.com>

feat: register volcano plugin

ed0425f

Signed-off-by: Xinmin Du <2812493086@qq.com>

feat: deal with minTaskMember, minMember, NetworkTopo

fca9f40

Signed-off-by: Xinmin Du <2812493086@qq.com>

fix: calculate of minResource

8ab5d71

Signed-off-by: Xinmin Du <2812493086@qq.com>

test: build PodGroup test

bd60987

Signed-off-by: Xinmin Du <2812493086@qq.com>

refactor: separate to 2 prs（build&handler）

ec6c5a8

Signed-off-by: Xinmin Du <2812493086@qq.com>

test: add test for new&reconcile_builder

f182b65

Signed-off-by: Xinmin Du <2812493086@qq.com>

google-oss-prow Bot requested review from astefanutti and kuizhiqing July 14, 2025 01:01

google-oss-prow Bot added the size/XL label Jul 14, 2025

google-oss-prow Bot added the area/gsoc label Jul 14, 2025

google-oss-prow Bot requested a review from Electronic-Waste July 14, 2025 02:01

Doris-xm added 3 commits July 14, 2025 20:53

fix: typo

ffa27f9

Signed-off-by: Xinmin Du <2812493086@qq.com>

Merge branch 'refs/heads/master' into volcano-podgroup-build

f8ea7dd

# Conflicts: # pkg/runtime/framework/plugins/registry.go

fix: trainer/v2 import

4fa3b6a

Signed-off-by: Xinmin Du <2812493086@qq.com>

Doris-xm changed the title ~~KEP-2437: PodGroup creation for Volcano Scheduler~~ feat: KEP-2437 - PodGroup Creation for Volcano Scheduler Jul 14, 2025

Doris-xm added 4 commits July 14, 2025 22:27

fix: networktopo type

1a247da

Signed-off-by: Xinmin Du <2812493086@qq.com>

fix: OpenAPI validation errors

73bb476

Signed-off-by: Xinmin Du <2812493086@qq.com>

fix: remove minTaskMembers

b92383a

Signed-off-by: Xinmin Du <2812493086@qq.com>

test: test coverage 100%

fe6ffd0

Signed-off-by: Xinmin Du <2812493086@qq.com>

Electronic-Waste reviewed Aug 1, 2025

View reviewed changes

Comment thread go.mod

Comment thread pkg/apis/trainer/v1alpha1/trainingruntime_types.go Outdated

Comment thread pkg/runtime/framework/plugins/volcano/volcano.go

tenzen-y reviewed Aug 1, 2025

View reviewed changes

Comment thread go.mod Outdated

Doris-xm added 4 commits October 1, 2025 22:12

fix: append owner reference & missing import

2fad831

Signed-off-by: Xinmin Du <2812493086@qq.com>

fix: rewrite volcano UT

f0e5a4c

Signed-off-by: Xinmin Du <2812493086@qq.com>

feat: add copyright

9c1eae1

Signed-off-by: Xinmin Du <2812493086@qq.com>

fix: sync RBAC to Helm charts

e185049

Signed-off-by: Xinmin Du <2812493086@qq.com>

astefanutti reviewed Oct 1, 2025

View reviewed changes

fix: refactor UTs

497a0b6

Signed-off-by: Xinmin Du <2812493086@qq.com>

andreyvelich reviewed Oct 2, 2025

View reviewed changes

fix: test validation separately

4030ce5

Signed-off-by: Xinmin Du <2812493086@qq.com>

google-oss-prow Bot added lgtm approved labels Oct 3, 2025

google-oss-prow Bot assigned astefanutti Oct 3, 2025

tenzen-y reviewed Oct 3, 2025

View reviewed changes

Update hack/swagger/main.go

c886583

Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com>

google-oss-prow Bot removed the lgtm label Oct 4, 2025

Doris-xm added 3 commits October 5, 2025 11:57

fix: refactor TestVolcano

451bf6e

Signed-off-by: Xinmin Du <2812493086@qq.com>

Merge remote-tracking branch 'origin/volcano-podgroup-build' into vol…

49bd5b9

…cano-podgroup-build

fix: refactor TestValidate

4175fef

Signed-off-by: Xinmin Du <2812493086@qq.com>

Doris-xm force-pushed the volcano-podgroup-build branch from 53e8d51 to 4175fef Compare October 5, 2025 04:55

andreyvelich reviewed Oct 5, 2025

View reviewed changes

tenzen-y reviewed Oct 6, 2025

View reviewed changes

Comment thread pkg/runtime/framework/plugins/volcano/volcano_test.go Outdated

Update pkg/runtime/framework/plugins/volcano/volcano_test.go

1f05304

Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> Signed-off-by: Du Xinmin <2812493086@qq.com>

tenzen-y reviewed Oct 6, 2025

View reviewed changes

google-oss-prow Bot removed the do-not-merge/hold label Oct 6, 2025

google-oss-prow Bot assigned tenzen-y Oct 6, 2025

google-oss-prow Bot added the lgtm label Oct 6, 2025

google-oss-prow Bot merged commit 4e17ce5 into kubeflow:master Oct 6, 2025
26 checks passed

		TrainingRuntimeContainerRuntimeClassKey = ".trainingRuntimeSpec.jobSetTemplateSpec.replicatedJobs.podTemplateSpec.runtimeClassName"
		ClusterTrainingRuntimeContainerRuntimeClassKey = ".clusterTrainingRuntimeSpec.jobSetTemplateSpec.replicatedJobs.podTemplateSpec.runtimeClassName"

	if tc.emptyCoSchedulingIndexerTrainingRuntimeContainerRuntimeClassKey {
	originTrainingRuntimeRuntimeKey := coscheduling.TrainingRuntimeContainerRuntimeClassKey
	coscheduling.TrainingRuntimeContainerRuntimeClassKey = ""
	t.Cleanup(func() {

Conversation

Doris-xm commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Doris-xm commented Jul 14, 2025

Uh oh!

google-oss-prow Bot commented Jul 14, 2025

Uh oh!

Doris-xm commented Jul 14, 2025

Uh oh!

google-oss-prow Bot commented Jul 14, 2025

Uh oh!

Electronic-Waste left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Electronic-Waste commented Aug 1, 2025

Uh oh!

Uh oh!

Electronic-Waste commented Aug 1, 2025

Uh oh!

astefanutti left a comment

Choose a reason for hiding this comment

Uh oh!

astefanutti Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Doris-xm Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Doris-xm Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich commented Oct 3, 2025

Uh oh!

astefanutti commented Oct 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tenzen-y left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

google-oss-prow Bot commented Oct 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Doris-xm commented Jul 14, 2025 •

edited

Loading

tenzen-y left a comment •

edited

Loading