feat(backend): add gRPC metrics to api-server (RPS/latency), optimize execution spec reporting

[ntny]

Description of your changes:

• Add standard gRPC RPS and latency metrics to each API server endpoint. There are a lot of manually added metrics in run_server.go.
  This PR replaces most of them with standardized gRPC metrics.
  Additionally, many services in the api-server previously had no metrics at all — this PR adds basic observability for them as well.
  I kept the old metrics to maintain backward compatibility with existing dashboards.

• Add metrics to measure the delay between Argo Workflow creation for recurring runs and its reporting to the API server by the persistence agent

• Optimize patchExistingTasks MySQL query by using the runId field to leverage the existing index (since podName is not covered by the index)

ReportWorkflowV1 Performance After Optimization (1.5M Tasks in MySQL):

Metrics will be available after the PR is merged.

This PR was motivated by an issue that emerged as the number of tasks in our system increased.
Recurring runs were created successfully, but their status was not updated for an extended period of time.
The same issue also affected regular (one-off) runs, which also experienced delays in status reporting.

[google-oss-prow]

Hi @ntny. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[github-actions]

🚫 This command cannot be processed. Only organization members or owners can use the commands.

[zazulam]

/ok-to-test

[zazulam]

lgtm, just curious if there is a reason behind sticking with this go-grpc-prometheus package when its github page has it listed as deprecated.

This project is depreacted and archived as the functionality moved to go-grpc-middleware repo since provider/[email protected] release. You can pull it using go get github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus. The API is simplified and morernized, yet functionality is similar to what v1.2.0 offered. All questions and issues you can submit here.

Is it possible to leverage the modules/structs from the middleware package instead?

[ntny]

lgtm, just curious if there is a reason behind sticking with this go-grpc-prometheus package when its github page has it listed as deprecated.

This project is depreacted and archived as the functionality moved to go-grpc-middleware repo since provider/[email protected] release. You can pull it using go get github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus. The API is simplified and morernized, yet functionality is similar to what v1.2.0 offered. All questions and issues you can submit here.

Is it possible to leverage the modules/structs from the middleware package instead?

lgtm, just curious if there is a reason behind sticking with this go-grpc-prometheus package when its github page has it listed as deprecated.

This project is depreacted and archived as the functionality moved to go-grpc-middleware repo since provider/[email protected] release. You can pull it using go get github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus. The API is simplified and morernized, yet functionality is similar to what v1.2.0 offered. All questions and issues you can submit here.

Is it possible to leverage the modules/structs from the middleware package instead?

1. I was a bit concerned about that too, but decided to stick with the v1 version since the project already uses github.com/grpc-ecosystem/go-grpc-middleware v1.3.0. Using go-grpc-prometheus/v2 would require upgrading to the v2 middleware stack, which I figured would be better handled in a separate merge request with proper testing.
2. Theoretically, yes but this requires implementing all metrics manually with matching naming and building a custom interceptor. That adds some maintenance overhead. It might be an option to explore if there are concerns about using a package.

[zazulam]

1. I was a bit concerned about that too, but decided to stick with the v1 version since the project already uses github.com/grpc-ecosystem/go-grpc-middleware v1.3.0. Using go-grpc-prometheus/v2 would require upgrading to the v2 middleware stack, which I figured would be better handled in a separate merge request with proper testing.
2. Theoretically, yes but this requires implementing all metrics manually with matching naming and building a custom interceptor. That adds some maintenance overhead. It might be an option to explore if there are concerns about using a package.

Makes sense, I'm okay with using the go-grpc-prometheus package if we can get an issue for tracking the upgrade of it / the middleware stack.

Any thoughts from your end @HumairAK?

[mprahl]

@ntny I made a PR to update go-grpc-middleware, so once that's merged, you can rebase your PR to use v2:
#12043

I prefer this approach since it seems straight forward to update it rather than add another unsupported library to KFP.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from humairak. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ntny]

/hold

need to configure metric registration with the new go-grpc-middleware/providers/prometheus api

[ntny]

/unhold

[zazulam]

Great work @ntny!
/lgtm

[juliusvonkohout]

@HumairAK for approval