Skip to content

feat(sdk): update PipelineConfig to reflect new workspace Protobuf changes #11934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(sdk): update PipelineConfig to reflect new workspace Protobuf changes #11934

wants to merge 1 commit into from

Conversation

VaniHaripriya
Copy link
Contributor

Description of your changes:

This PR updates the dsl.PipelineConfig class to support configuring a pipeline workspace using the newly introduced WorkspaceConfig and KubernetesWorkspaceConfig message types in pipeline_spec.proto

Checklist:

@google-oss-prow Google OSS Prow
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@google-oss-prow Google OSS Prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot requested review from DharmitD and zazulam May 27, 2025 20:26
@VaniHaripriya VaniHaripriya force-pushed the dsl_pipeline_config_update branch from cfc2a88 to ecd09b8 Compare May 27, 2025 20:50
@VaniHaripriya VaniHaripriya marked this pull request as ready for review May 27, 2025 23:26
@google-oss-prow google-oss-prow bot requested a review from droctothorpe May 27, 2025 23:26
@mprahl
Copy link
Contributor

mprahl commented May 28, 2025

@VaniHaripriya I think the tests will fail until kfp-pipeline-spec is released on PyPi and sdk/python/requirements.txt is updated to use the new version.

@mprahl
Copy link
Contributor

mprahl commented May 28, 2025

@VaniHaripriya you'll need to update _merge_pipeline_config similar to what @rimolive 's PR is doing here:
https://github.com/kubeflow/pipelines/pull/11269/files#diff-42a6e18d166f33d4a0992ab9609966d24389846045972a99e9f286bba08d4014R2064-R2071

@VaniHaripriya VaniHaripriya force-pushed the dsl_pipeline_config_update branch from ecd09b8 to b0941f6 Compare May 30, 2025 14:41
@VaniHaripriya
Copy link
Contributor Author

@VaniHaripriya you'll need to update _merge_pipeline_config similar to what @rimolive 's PR is doing here: https://github.com/kubeflow/pipelines/pull/11269/files#diff-42a6e18d166f33d4a0992ab9609966d24389846045972a99e9f286bba08d4014R2064-R2071

Thank you @mprahl , updated the pipeline_config and added a test.

@VaniHaripriya VaniHaripriya force-pushed the dsl_pipeline_config_update branch from 9c780f9 to 3c4dfd2 Compare May 30, 2025 15:21
mprahl
mprahl reviewed May 30, 2025
View reviewed changes
sdk/python/kfp/dsl/pipeline_config.py
@@ -13,11 +13,35 @@
# limitations under the License.
"""Pipeline-level config options."""

from typing import Any, Dict, Optional

class KubernetesWorkspaceConfig:
Copy link
Contributor

@mprahl mprahl May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you make these classes importable from kfp.dsl (e.g. from kfp.dsl import KubernetesWorkspaceConfig) just like PipelineConfig? See sdk/python/kfp/dsl/__init__.py for the PipelineConfig example.

mprahl
mprahl reviewed May 30, 2025
View reviewed changes
sdk/python/kfp/compiler/pipeline_spec_builder.py
# {'pipelineConfig': {
# '<some pipeline config option>': pipelineConfig.<get that value>,
# }}, platformSpec.platforms['kubernetes'])
workspace = pipelineConfig.workspace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to guard against pipelineConfig.workspace being None/not set below?

mprahl
mprahl reviewed May 30, 2025
View reviewed changes
sdk/python/kfp/dsl/pipeline_config.py Outdated
class WorkspaceConfig:
"""Configuration for a shared workspace that persists during the pipeline run."""

def __init__(self, size: str = "", kubernetes: Optional[KubernetesWorkspaceConfig] = None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add documentation for the expected size format?

mprahl
mprahl reviewed May 30, 2025
View reviewed changes
sdk/python/kfp/dsl/pipeline_config.py Outdated
class WorkspaceConfig:
"""Configuration for a shared workspace that persists during the pipeline run."""

def __init__(self, size: str = "", kubernetes: Optional[KubernetesWorkspaceConfig] = None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should make size required.

Suggested change
def __init__(self, size: str = "", kubernetes: Optional[KubernetesWorkspaceConfig] = None):
def __init__(self, size: str, kubernetes: Optional[KubernetesWorkspaceConfig] = None):

@VaniHaripriya VaniHaripriya force-pushed the dsl_pipeline_config_update branch from 3c4dfd2 to 6165e1f Compare May 30, 2025 15:31
@google-oss-prow google-oss-prow bot added size/L and removed size/M labels May 30, 2025
@VaniHaripriya VaniHaripriya force-pushed the dsl_pipeline_config_update branch 3 times, most recently from 20bedd5 to 2069008 Compare May 30, 2025 19:05
@VaniHaripriya VaniHaripriya force-pushed the dsl_pipeline_config_update branch from 2069008 to 8f89824 Compare May 30, 2025 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mprahl mprahl mprahl left review comments

@DharmitD DharmitD Awaiting requested review from DharmitD

@zazulam zazulam Awaiting requested review from zazulam

@droctothorpe droctothorpe Awaiting requested review from droctothorpe

Assignees
No one assigned
Labels
size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@VaniHaripriya @mprahl