Transport error: define Request.GetBody to avoid this error

[lukasmalkmus]

Description

We are seeing the following error from clients that send traces to our custom collector implementation:

Post https://[...]/v1/traces: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

Environment

• OS: Linux/macOS
• Architecture: x86
• Go Version: 1.18
• opentelemetry-go version: v1.6.3

Steps To Reproduce

Wasn't able to reproduce it with a local example that is not using our custom collector.

Expected behavior

I never expected to see this error. Should probably implement Request.GetBody to avoid this as suggested by the error message?

[MrAlias]

Without code to reproduce this error there is no way to validate it has been resolved. Can you please add a minimal example that will generate the error?

[lukasmalkmus]

Ok, I'll try to come up with something over the weekend.

[lukasmalkmus]

Similar issue: stripe/stripe-go#710 and the solution: https://github.com/stripe/stripe-go/pull/711/files#diff-f34528b48beca228ad5851464b3e20496f8c34c1f7b663e7f109fa9f5ed02fc4R328-R338.

[DavidS-ovm]

hitting the same thing where a process just stops sending traces with that message repeating in the logs:

2023/02/02 13:16:19 traces export: Post "https://api.honeycomb.io/v1/traces": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

[DavidS-ovm]

this could be happening when honeycomb engages its rate limits. the library still should be able to deal with that gracefully

[MrAlias]

I'm not able to verify if this has been fixed in recent releases. Without example code showing the error we will need to close this as it is not reproducible. Please reopen this if a testable workflow can be provided.

[kasun-bandara]

This issue still exists for otlpmetrichttp exporter latest (v1.25.0) version due to the issue in go/http2 lib here

The otlpmetricgrpc library does not have the underlying issue, and I believe the payload is properly set (Request.GetBody) for gRPC retry requests.

[MrAlias]

@kasun-bandara please open a new bug to address your issue. Please provide example code that reproduces the error in your issue.