feat(runtimes): implement clusterTrainingRuntime deprecation process by tdn21 · Pull Request #2791 · kubeflow/trainer

tdn21 · 2025-08-20T18:50:57Z

What this PR does

Implements the ClusterTrainingRuntime deprecation policy by surfacing admission warnings when:

Creating a ClusterTrainingRuntime labeled trainer.kubeflow.org/support=deprecated.
Creating/updating a TrainJob that references a deprecated ClusterTrainingRuntime.

Why we need it

This makes users aware of deprecated runtimes at the time of creating ClusterTrainingRuntimes or TrainJobs per the runtime deprecation policy:
https://www.kubeflow.org/docs/components/trainer/operator-guides/runtime/#runtime-deprecation-policy

Which issue(s) this PR fixes:

Fixes #2789

Checklist:

Docs included if any changes are user facing

Notes

Warning message includes the policy URL for user guidance
SDK-side warnings (listing/referencing runtimes) will be addressed in a separate PR in the SDK repo

astefanutti · 2025-08-21T08:02:43Z

+	LabelDeprecated string = "trainer.kubeflow.org/deprecated"
+
+	// DeprecatedTrueValue is the value indicating deprecation when used with LabelDeprecated.
+	DeprecatedTrueValue string = "true"


Almost a nit, but I wonder if this should be treated as a "flag" and only check for the label existence not its value. I don't think trainer.kubeflow.org/deprecated=false really makes sense in practice.

And to be sure that's future proof, are we foreseeing the need for other values in the lifecycle of runtimes like alpha, stable ...

@andreyvelich @tenzen-y @Electronic-Waste WDYT?

Sure, that make sense, but do you prefer that we use other label to identify lifecycle of a runtime (e.g. alpha, stable, deprecated) ?

If we think it's the right way to go, yes I was thinking something like trainer.kubeflow.org/support="alpha|stable|deprecated"

Thank you @astefanutti and @andreyvelich for the thoughtful input.
I also think that updating the label to something like trainer.kubeflow.org/support or trainer.kubeflow.org/lifecycle would be a more future-proof approach, as it better conveys the runtime’s lifecycle stage.

(Once we reach a consensus on the final label name, I’ll go ahead and update the PR accordingly and also open a separate PR to adjust the related documentation.)

Do we see any advantages to introduce alpha? Official CTRs maintained by us are CustomResource, not CRD, which needs to maintain the API and display maturity level to users. And I think, the maturity of CRs is determined by CRD, since we always make the optimal design decision for the CR resource based on the APIs provided by CRD.

Given these facts, I think trainer.kubeflow.org/deprecated=true is enough. If CTRs does not have this label or have its value set to true, we consider it having the same maturity level with their CRD, ClusterTrainingRuntime. WDYT?

I would distinguish maturity of CRD APIs and maturity of supported runtimes, since the CRD might be in the v1 stage, but the runtime is unstable and can introduce breaking changes. For example, we can decide to remove some packages from the runtime (e.g. transformers).

However, we can end up with never introducing the maturity level for the runtimes, if we don't see strong signal from users.

All in all, my suggestion is to keep trainer.kubeflow.org/support=deprecated label for now to make sure we can introduce other support levels in the future if that is required.

@tenzen-y Any thoughts ?

I would also differentiate the maturity of the CRD APIs from that of the runtimes.

For the maturity levels of the runtimes, it's likely that downstream projects / companies do not use the upstream runtimes directly but productize their own runtimes and can define those levels according to their own policies.

So I'd also suggest to "standardize" the trainer.kubeflow.org/support label without enforcing values and let those projects / companies the freedom to set it freely.

For upstream we can start with trainer.kubeflow.org/support=deprecated.

+1 on trainer.kubeflow.org/support=deprecated

SGTM. Let's use trainer.kubeflow.org/support=deprecated

Thank you! updated. (51b95cf)

andreyvelich

Thank you @tdn21!
/ok-to-test

andreyvelich · 2025-08-21T13:09:18Z

+	LabelDeprecated string = "trainer.kubeflow.org/deprecated"
+
+	// DeprecatedTrueValue is the value indicating deprecation when used with LabelDeprecated.
+	DeprecatedTrueValue string = "true"


Sure, that make sense, but do you prefer that we use other label to identify lifecycle of a runtime (e.g. alpha, stable, deprecated) ?

andreyvelich · 2025-08-21T13:15:36Z

+	testingutil "github.com/kubeflow/trainer/v2/pkg/util/testing"
+)
+
+func TestClusterTrainingRuntime_ValidateCreate_DeprecatedWarning(t *testing.T) {


@tdn21 @astefanutti @tenzen-y @Electronic-Waste Do we prefer to create unit or integration test to verify it ?
https://github.com/kubeflow/trainer/blob/2a059ec3c93d3a09210a44d74677a53600576787/test/integration/webhooks/clustertrainingruntime_webhook_test.go

Maybe we should have both like for TrainJob webhook ?

Agree, having both would be better. Let me add integration test as well.

Thanks a lot for the suggestion @andreyvelich. I have added the integration tests.
Just a quick note:

In the integration tests, we aren’t asserting warnings since controller-runtime’s client doesn’t expose admission warnings. These tests only validate that resource creation succeeds and isn’t blocked.

The actual warning text is validated in the unit tests.

Agree, having both would be better. Let me add integration test as well.

I don't think so. The webhook integration test for verification webhook behavior and marker validatiion.
The only UTs should be better.

+1 on UTs. That will be enough:)

Sounds good, thank you! 🙇
I've removed earlier added integration tests. (62f7931)

coveralls · 2025-08-21T13:21:15Z

Pull Request Test Coverage Report for Build 18546363197

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

9 of 28 (32.14%) changed or added relevant lines in 3 files are covered.
192 unchanged lines in 10 files lost coverage.
Overall coverage increased (+0.01%) to 52.147%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/util/trainingruntime/trainingruntime.go	0	6	0.0%
pkg/runtime/core/clustertrainingruntime.go	0	13	0.0%

Files with Coverage Reduction	New Missed Lines	%
pkg/runtime/framework/plugins/registry.go	2	0.0%
pkg/runtime/framework/plugins/jobset/builder.go	7	0.0%
pkg/runtime/runtime.go	7	72.12%
pkg/runtime/core/clustertrainingruntime.go	11	33.33%
pkg/runtime/framework/core/framework.go	15	84.69%
pkg/runtime/framework/plugins/torch/torch.go	22	84.18%
pkg/controller/trainjob_controller.go	24	0.0%
pkg/runtime/framework/plugins/coscheduling/coscheduling.go	30	33.33%
pkg/runtime/framework/plugins/jobset/jobset.go	31	52.97%
pkg/runtime/core/trainingruntime.go	43	72.02%

Totals
Change from base Build 17469034215:	0.01%
Covered Lines:	1263
Relevant Lines:	2422

💛 - Coveralls

tenzen-y · 2025-08-28T15:44:01Z

+	testingutil "github.com/kubeflow/trainer/v2/pkg/util/testing"
+)
+
+func TestClusterTrainingRuntime_ValidateCreate_DeprecatedWarning(t *testing.T) {


Suggested change

func TestClusterTrainingRuntime_ValidateCreate_DeprecatedWarning(t *testing.T) {

func ValidateCreate(t *testing.T) {

The UTs should be for the top level exposed function. We should avoid implementing UTs each for action.

@tdn21 Splitting with _ is python coding styling. In golang, we use camel style. And it will also be better to use a tidy and short name:)

Thank you. Updated! (ref)

tenzen-y · 2025-08-28T15:44:36Z

+	_, ctx := ktesting.NewTestContext(t)
+	ctx, cancel := context.WithCancel(ctx)
+	t.Cleanup(cancel)
+
+	obj := testingutil.MakeClusterTrainingRuntimeWrapper("test-runtime").
+		RuntimeSpec(trainer.TrainingRuntimeSpec{
+			Template: trainer.JobSetTemplateSpec{
+				Spec: func() jobsetv1alpha2.JobSetSpec {
+					js := testingutil.MakeJobSetWrapper("", "")
+					js.Replicas(1, constants.DatasetInitializer, constants.ModelInitializer, constants.Node)
+					return js.Obj().Spec
+				}(),
+			},
+		}).Obj()
+
+	if obj.Labels == nil {
+		obj.Labels = map[string]string{}
+	}
+	obj.Labels[constants.LabelDeprecated] = constants.DeprecatedTrueValue
+
+	validator := &ClusterTrainingRuntimeWebhook{}
+	warnings, err := validator.ValidateCreate(ctx, obj)
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	var want admission.Warnings
+	want = append(want, "ClusterTrainingRuntime \"test-runtime\" is deprecated and will be removed in a future release of Kubeflow Trainer. See runtime deprecation policy: "+constants.RuntimeDeprecationPolicyURL)
+	if diff := cmp.Diff(want, warnings); diff != "" {
+		t.Fatalf("unexpected warnings (-want, +got): %s", diff)
+	}


It would be better to follow the another test case.

tenzen-y · 2025-08-28T15:46:04Z

+				Obj(),
+			wantError:    nil,
+			wantWarnings: admission.Warnings{"Referenced ClusterTrainingRuntime \"test-runtime\" is deprecated and will be removed in a future release of Kubeflow Trainer. See runtime deprecation policy: " + constants.RuntimeDeprecationPolicyURL},
+			deprecate:    true,


Suggested change

deprecate: true,

clusterTrainingRuntime: ...

Handling test data by the boolean is not good way. For more descriptive approach, we shoul define the clusterTrainingRuntime object.

Makes sense, thank you! (updated, ref)

tenzen-y · 2025-08-28T15:47:13Z

+	var warnings admission.Warnings
+	if clusterTrainingRuntime.Labels != nil {
+		if val, ok := clusterTrainingRuntime.Labels[constants.LabelDeprecated]; ok && val == constants.DeprecatedTrueValue {
+			warnings = append(warnings, fmt.Sprintf(
+				"Referenced ClusterTrainingRuntime \"%s\" is deprecated and will be removed in a future release of Kubeflow Trainer. See runtime deprecation policy: %s",
+				clusterTrainingRuntime.Name,
+				constants.RuntimeDeprecationPolicyURL,
+			))
+		}
+	}


I don't think that we should have these duplicate validations.

Thank you so much @tenzen-y and @Electronic-Waste for your feedback. 🙇
Could you please help me better understand what we mean by "duplicate validations" here?

Currently, we warn in two different scenarios:

when creating a deprecated ClusterTrainingRuntime (via webhook), and

when a TrainJob references a deprecated ClusterTrainingRuntime (runtime validation via the TrainJob webhook).

Since these are separate user touchpoints, I feel it makes sense to keep both. That said, I completely agree we should deduplicate the code by using a helper. (added it here)

I think the "duplicate validations" means that you implement validation in both runtime plugin and webhook. Could we only examine the whether clustertrainingruntime is deprecated in webhook? Maybe that's enough:)

WDYT? @tdn21 @tenzen-y @andreyvelich @astefanutti

My understanding is the warnings should only be emitted for TrainJobs that reference deprecated training runtimes, not for the creation of "deprecated" training runtimes themselves (which somewhat does not make much sense).

In that case, only the logic in the runtime is necessary, and the validation in the ClusterTrainingRuntime webhook is not necessary.

Electronic-Waste

@tdn21 Thanks for this. Can you follow up with this PR?

Electronic-Waste · 2025-09-06T06:24:23Z

+	LabelDeprecated string = "trainer.kubeflow.org/deprecated"
+
+	// DeprecatedTrueValue is the value indicating deprecation when used with LabelDeprecated.
+	DeprecatedTrueValue string = "true"


SGTM. Let's use trainer.kubeflow.org/support=deprecated

Electronic-Waste · 2025-09-06T06:26:09Z

+	var warnings admission.Warnings
+	if clusterTrainingRuntime.Labels != nil {
+		if val, ok := clusterTrainingRuntime.Labels[constants.LabelDeprecated]; ok && val == constants.DeprecatedTrueValue {
+			warnings = append(warnings, fmt.Sprintf(
+				"Referenced ClusterTrainingRuntime \"%s\" is deprecated and will be removed in a future release of Kubeflow Trainer. See runtime deprecation policy: %s",
+				clusterTrainingRuntime.Name,
+				constants.RuntimeDeprecationPolicyURL,
+			))
+		}
+	}


Electronic-Waste · 2025-09-06T06:27:58Z

+	testingutil "github.com/kubeflow/trainer/v2/pkg/util/testing"
+)
+
+func TestClusterTrainingRuntime_ValidateCreate_DeprecatedWarning(t *testing.T) {


+1 on UTs. That will be enough:)

Electronic-Waste · 2025-09-06T06:29:25Z

+	testingutil "github.com/kubeflow/trainer/v2/pkg/util/testing"
+)
+
+func TestClusterTrainingRuntime_ValidateCreate_DeprecatedWarning(t *testing.T) {


@tdn21 Splitting with _ is python coding styling. In golang, we use camel style. And it will also be better to use a tidy and short name:)

review-notebook-app · 2025-09-06T08:42:48Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

…ce constant Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

…ainingRuntime Create ClusterTrainingRuntime with deprecated label; assert creation succeeds (warning is non-blocking) Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

…terTrainingRuntime Mark ClusterTrainingRuntime deprecated; create TrainJob referencing it; assert creation succeeds (warning is non-blocking) Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

…lidateObjects Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

… webhooks Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

…on check Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

astefanutti · 2025-09-08T07:31:37Z

+	var warnings admission.Warnings
+	if labelutil.IsSupportDeprecated(clTrainingRuntime.Labels) {
+		warnings = append(warnings, fmt.Sprintf(
+			"ClusterTrainingRuntime \"%s\" is deprecated and will be removed in a future release of Kubeflow Trainer. See runtime deprecation policy: %s",
+			clTrainingRuntime.Name,
+			constants.RuntimeDeprecationPolicyURL,
+		))
+	}
+	return warnings, validateReplicatedJobs(clTrainingRuntime.Spec.Template.Spec.ReplicatedJobs).ToAggregate()


This is not necessary. Warnings should only be emitted for TrainJobs referencing deprecated runtimes, not for the creation of ClusterTrainingRuntimes themselves.

@astefanutti According to our policy, we also show an error when user creates a deprecated Runtime: https://www.kubeflow.org/docs/components/trainer/operator-guides/runtime/#runtime-deprecation-policy
Do you prefer that we don't do this ?

@andreyvelich ah you're right, I missed this. That makes sense.

astefanutti · 2025-09-08T07:32:10Z

@@ -0,0 +1,12 @@
+package labels


Maybe:

Suggested change

package labels

package support

I would prefer we call this package as runtime, similar to trainjob utils: https://github.com/kubeflow/trainer/blob/master/pkg/util/trainjob/trainjob.go

That's a good idea.

andreyvelich · 2025-09-24T09:25:46Z

@tdn21 Did you get a chance to address the remaining comments ?
/milestone v2.1

andreyvelich · 2025-10-14T18:06:27Z

ping @tdn21, we are planning to release v2.1.0-rc.0 this week.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich · 2025-10-16T00:24:41Z

@astefanutti @Electronic-Waste @tdn21 @tenzen-y I've updated the package according to this comment: #2791 (comment)
Do we have any other changes we would like to make before merging this PR ?

astefanutti · 2025-10-16T07:47:57Z

/lgtm

Thanks @tdn21 and everyone!

andreyvelich

/approve

google-oss-prow · 2025-10-16T10:50:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ubeflow#2791) * constants: add deprecation label constants for runtimes Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * webhooks: warn on creation of deprecated ClusterTrainingRuntime Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * runtime: warn when TrainJob references deprecated ClusterTrainingRuntime Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * tests: add unit test coverage for deprecation warnings Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * webhooks/runtime: update deprecation warning wording and tests Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * constants/webhooks/runtime: centralize runtime policy URL and reference constant Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * tests(integration/webhooks): add create case for deprecated ClusterTrainingRuntime Create ClusterTrainingRuntime with deprecated label; assert creation succeeds (warning is non-blocking) Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * tests(integration/webhooks): add TrainJob create with deprecated ClusterTrainingRuntime Mark ClusterTrainingRuntime deprecated; create TrainJob referencing it; assert creation succeeds (warning is non-blocking) Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * tests/runtime: cover deprecation warning in ClusterTrainingRuntime.ValidateObjects Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * tests(integration): remove deprecation-warning cases for CTR/TrainJob webhooks Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * feat(runtimes): switch to support=deprecated and centralize deprecation check Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * tests(webhooks): focus UTs on entrypoints and clean up tests/test data Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> * Rename package from labels to trainingruntime Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Tarun Duhan <itarunduhan@gmail.com> Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-prow Bot added the do-not-merge/work-in-progress label Aug 20, 2025

google-oss-prow Bot requested review from jinchihe and kuizhiqing August 20, 2025 18:51

google-oss-prow Bot added the size/L label Aug 20, 2025

tdn21 force-pushed the issue-2789/implement-cluster-training-runtimes-deprecation-process branch from c5b0011 to 2a059ec Compare August 20, 2025 18:53

tdn21 marked this pull request as ready for review August 21, 2025 06:58

google-oss-prow Bot removed the do-not-merge/work-in-progress label Aug 21, 2025

google-oss-prow Bot requested a review from astefanutti August 21, 2025 06:58

astefanutti reviewed Aug 21, 2025

View reviewed changes

andreyvelich reviewed Aug 21, 2025

View reviewed changes

google-oss-prow Bot added the ok-to-test label Aug 21, 2025

tdn21 requested a review from andreyvelich August 22, 2025 13:12

tenzen-y reviewed Aug 28, 2025

View reviewed changes

Electronic-Waste reviewed Sep 6, 2025

View reviewed changes

tdn21 force-pushed the issue-2789/implement-cluster-training-runtimes-deprecation-process branch from 7439fc6 to c880b0c Compare September 6, 2025 08:42

google-oss-prow Bot added size/XXL and removed size/L labels Sep 6, 2025

tdn21 marked this pull request as draft September 6, 2025 08:57

google-oss-prow Bot added the do-not-merge/work-in-progress label Sep 6, 2025

tdn21 added 6 commits September 6, 2025 14:31

constants: add deprecation label constants for runtimes

3feab6e

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

webhooks: warn on creation of deprecated ClusterTrainingRuntime

653c264

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

runtime: warn when TrainJob references deprecated ClusterTrainingRuntime

74c6bf6

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tests: add unit test coverage for deprecation warnings

9b13818

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

webhooks/runtime: update deprecation warning wording and tests

16ddbe4

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

constants/webhooks/runtime: centralize runtime policy URL and referen…

35f7f4b

…ce constant Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tdn21 added 6 commits September 6, 2025 14:31

tests(integration/webhooks): add create case for deprecated ClusterTr…

492d8e2

…ainingRuntime Create ClusterTrainingRuntime with deprecated label; assert creation succeeds (warning is non-blocking) Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tests(integration/webhooks): add TrainJob create with deprecated Clus…

f20985b

…terTrainingRuntime Mark ClusterTrainingRuntime deprecated; create TrainJob referencing it; assert creation succeeds (warning is non-blocking) Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tests/runtime: cover deprecation warning in ClusterTrainingRuntime.Va…

2e90081

…lidateObjects Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tests(integration): remove deprecation-warning cases for CTR/TrainJob…

62f7931

… webhooks Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

feat(runtimes): switch to support=deprecated and centralize deprecati…

51b95cf

…on check Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tests(webhooks): focus UTs on entrypoints and clean up tests/test data

c74ae97

Signed-off-by: Tarun Duhan <itarunduhan@gmail.com>

tdn21 force-pushed the issue-2789/implement-cluster-training-runtimes-deprecation-process branch from c880b0c to c74ae97 Compare September 6, 2025 09:02

google-oss-prow Bot added size/L and removed size/XXL labels Sep 6, 2025

tdn21 marked this pull request as ready for review September 6, 2025 09:30

google-oss-prow Bot removed the do-not-merge/work-in-progress label Sep 6, 2025

tdn21 requested a review from Electronic-Waste September 6, 2025 09:30

astefanutti reviewed Sep 8, 2025

View reviewed changes

google-oss-prow Bot added this to the v2.1 milestone Sep 24, 2025

Rename package from labels to trainingruntime

3d69a67

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-prow Bot assigned astefanutti Oct 16, 2025

google-oss-prow Bot added the lgtm label Oct 16, 2025

andreyvelich reviewed Oct 16, 2025

View reviewed changes

google-oss-prow Bot added the approved label Oct 16, 2025

google-oss-prow Bot merged commit b17c6c9 into kubeflow:master Oct 16, 2025
31 checks passed

	func TestClusterTrainingRuntime_ValidateCreate_DeprecatedWarning(t *testing.T) {
	func ValidateCreate(t *testing.T) {

Conversation

tdn21 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Why we need it

Which issue(s) this PR fixes:

Checklist:

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18546363197

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste left a comment

Choose a reason for hiding this comment

Uh oh!

tdn21 commented Aug 20, 2025 •

edited

Loading

coveralls commented Aug 21, 2025 •

edited

Loading