Skip to content

stats/opentelemetry: record retry attempts from clientStream #8342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: master
Choose a base branch
from

Conversation

vinothkumarr227
Copy link
Contributor

@vinothkumarr227 vinothkumarr227 commented May 19, 2025

Fixes: #8299

RELEASE NOTES:

  • stats/opentelemetry: Retry attempts (grpc.previous-rpc-attempts) are now recorded as span attributes for non-transparent client retries.

Copy link

codecov bot commented May 19, 2025

Codecov Report

❌ Patch coverage is 84.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.48%. Comparing base (85240a5) to head (9b13b21).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
stats/opentelemetry/client_tracing.go 55.55% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8342      +/-   ##
==========================================
- Coverage   82.48%   82.48%   -0.01%     
==========================================
  Files         413      413              
  Lines       40518    40546      +28     
==========================================
+ Hits        33422    33445      +23     
- Misses       5738     5743       +5     
  Partials     1358     1358              
Files with missing lines Coverage Δ
stats/opentelemetry/client_metrics.go 85.16% <100.00%> (+0.08%) ⬆️
stats/opentelemetry/opentelemetry.go 77.90% <ø> (ø)
stats/opentelemetry/trace.go 90.47% <100.00%> (+1.58%) ⬆️
stats/opentelemetry/client_tracing.go 82.85% <55.55%> (-4.65%) ⬇️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@purnesh42H purnesh42H added Type: Bug Area: Observability Includes Stats, Tracing, Channelz, Healthz, Binlog, Reflection, Admin, GCP Observability labels May 19, 2025
@purnesh42H purnesh42H added this to the 1.73 Release milestone May 19, 2025
Copy link
Contributor

@purnesh42H purnesh42H left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember we had separate tests for retries. This change should only affect that test. This change shouldn't affect tests which are doing only single attempt.

Copy link
Contributor

@purnesh42H purnesh42H left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vinothkumarr227 have you tested this to ensure its working correctly? I was under impression that TestTraceSpan_WithRetriesAndNameResolutionDelay will need changes for expected values. I think its not working as expected because you are not setting the count back to ctx after incrementing.

@vinothkumarr227 vinothkumarr227 removed their assignment Jul 28, 2025
@vinothkumarr227 vinothkumarr227 removed their assignment Jul 29, 2025
@arjan-bal arjan-bal self-assigned this Jul 30, 2025
Copy link
Contributor

@arjan-bal arjan-bal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, adding a second reviewer. Please address the minor comment.

@@ -1548,12 +1547,12 @@ const delayedResolutionEventName = "Delayed name resolution complete"
// TestTraceSpan_WithRetriesAndNameResolutionDelay verifies that
// "Delayed name resolution complete" event is recorded in the call trace span
// only once if any of the retry attempt encountered a delay in name resolution
func (s) TestTraceSpan_WithRetriesAndNameResolutionDelay(t *testing.T) {
func TestTraceSpan_WithRetriesAndNameResolutionDelay(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the (s) back.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@arjan-bal arjan-bal requested a review from dfawley July 30, 2025 07:40
@arjan-bal
Copy link
Contributor

@vinothkumarr227 there is a failure in the updated test reported by codecov: https://github.com/grpc/grpc-go/actions/runs/16616024027/job/47008766111?pr=8342

PTAL and ensure the test is not flaky.

@vinothkumarr227
Copy link
Contributor Author

@vinothkumarr227 there is a failure in the updated test reported by codecov: https://github.com/grpc/grpc-go/actions/runs/16616024027/job/47008766111?pr=8342

PTAL and ensure the test is not flaky.

Sure, I’ll check it out.

@vinothkumarr227
Copy link
Contributor Author

@vinothkumarr227 there is a failure in the updated test reported by codecov: https://github.com/grpc/grpc-go/actions/runs/16616024027/job/47008766111?pr=8342

PTAL and ensure the test is not flaky.
Hi @arjan-bal , I tested the test locally and everything passed consistently. I’ve now merged the latest changes from master, and all test cases are passing.

@vinothkumarr227 vinothkumarr227 removed their assignment Aug 1, 2025
span.SetAttributes(
attribute.Bool("Client", rs.Client),
attribute.Bool("FailFast", rs.FailFast),
attribute.Int64("previous-rpc-attempts", int64(ai.previousRPCAttempts)),
// TODO: Remove "previous-rpc-attempts" and "transparent-retry"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a TODO?

Isn't this simply?

attribs := []attribute.KeyValue{...Bool("Client", rs.Client), /* etc */}
if ai.previousRPCAttempts != nil {
	attribs = append(attribs, attribute.Int64("prev...", ..), ...)
}

Copy link
Contributor Author

@vinothkumarr227 vinothkumarr227 Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Per @arjan-bal recommendation, this will be handled in a separate PR, which is why it's a TODO. I've created an issue for reference: #8430

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand why. If these attributes are not supposed to be present, then wouldn't it be incorrect to include them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Doug. I've made changes and pushed the fix.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh it was already setting these things?

Also what is this about "Go was always setting these attributes even though they aren't in the spec"?

Are we intentionally not implementing the spec? Why are we mentioning the OC spec in the OT plugin? I'm kind of lost here, TBH.

@vinothkumarr227 vinothkumarr227 requested a review from dfawley August 6, 2025 12:31
@dfawley dfawley removed their assignment Aug 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Observability Includes Stats, Tracing, Channelz, Healthz, Binlog, Reflection, Admin, GCP Observability Type: Bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Retry Attempts are not available in client-stats
6 participants