otelgrpc: Allow adding custom attributes to a request's metrics based on its payload

[pembletonj2-sonrai]

Problem Statement

It's not possible to add custom attributes to the metrics generated by otelgrpc based on the data in the request payloads.

Proposed Solution

A new option can be added to pass a function which will be run on each payload received to generate metric attributes for that request. Example:

otelgrpc.NewServerHandler(otelgrpc.WithMetricAttributesFromPayload(func(payload any) []attribute.KeyValue {
    data1 := extractSomeDataFromPayload(payload)
    data2 := extractOtherDataFromPayload(payload)
    return []attribute.KeyValue{
        attribute.String("key1", data1),
        attribute.String("key2", data2),
    }
}))

key1 and key2 will then be included in the metrics' attributes for this request.

Alternatives

We found an alternative way to add the needed attributes to spans. The span's attributes can be modified from an interceptor:

span := trace.SpanFromContext(ctx)
span.SetAttributes(attribute.String("key", value))

The payload can also be accessed from an interceptor, so its data can be used to set a span's attribute. However, it's not possible to do this with metrics.

Prior Art

#3894 discusses setting constant values for metric/span attributes and a PR has been made for it: #5133. This issue builds on this by allowing custom attributes to be computed on a per-request basis rather than being set to a constant value.

[dmathieu]

Similar solution for the otelhttp instrumentation: #5876

[dhowden]

Looking at the HTTP implementation (#5876), feels like we can make a very similar change, but also passing the incoming ctx too (so that callers can extract metadata etc).

The associated option then might look like:

func WithMetricAttributesFn(metricAttributesFn func(ctx context.Context,  payload any) []attribute.KeyValue) Option {
// ...
}

I've made a sketch implementation here (in draft): #6092, mostly because the grpc.StatsHandler pattern makes it a bit messier and there might be strong opinions about how this is wired in. Happy to carry this through if there is consensus on this approach.

[scorpionknifes]

Took a quick look, just wondering, why not use stats.RPCStats or *stats.InPayload instead of just passing payload any?

MetricAttributesFn func(ctx context.Context, rpcStats stats.RPCStats) []attribute.KeyValue
// or just *stats.InPayload if we scoping the feature to only request
MetricAttributesInPayloadFn func(ctx context.Context, inPayload *stats.InPayload) []attribute.KeyValue

However I see the issue of with developer experience for stats.RPCStats which requires the user to type casting. We might want something more complex (some of these might be out of scope).

MetricAttributesInPayloadFn func(ctx context.Context, inPayload *stats.InPayload) []attribute.KeyValue
MetricAttributesOutPayloadFn func(ctx context.Context, outPayload *stats.OutPayload) []attribute.KeyValue
MetricAttributesOutHeaderFn func(ctx context.Context, inPayload *stats.OutHeader) []attribute.KeyValue
MetricAttributesEndFn func(ctx context.Context, inPayload *stats.End) []attribute.KeyValue

Also what's the purpose of passing the ctx in? I feel like we can achieve the ctx metadata -> attributes with something else

Just dropping some initial thoughts for discussion.

[broguinn]

Bumping this, as it seems really useful.

[vladComan0]

We're interested in this feature as our company has a need for it and we'd like to contribute. Before starting, we wanted to check if there's already any ongoing work or design considerations we should align with and also whether this feature is already planned for an upcoming release. Thanks!

[dmathieu]

This should follow the same design as with the otelhttp WithMetricAttributesFn method.
https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#WithMetricAttributesFn

[vladComan0]

@dmathieu I think that the WithMetricAttributesFn design does not actually fit well in here because of the following reasons:

1. For the payload any solution, if I understood correctly, and by looking at @dhowden draft PR above, we cannot guarantee the statement there, i.e., the "Run this once on InPayload and record the attributes for the entire RPC." comment, in the current form because I think that runs N times in streaming; the solution for this may end up a little weird, but nothing major I think. However, we also won't have access to multiple fields from other phases, e.g., error.

2. The solution proposed also by @scorpionknifes, although more complete, may create a weird user experience due to multiple possible callbacks (assuming we don't enrich the API with the methods proposed above, which I think does not provide clarity), e.g.,

This would also let us access all of the attributes (from multiple phases), although the code might become a little weird to synchronize between the phases:

WithMetricAttributesFn(func(ctx context.Context, rs stats.RPCStats) []attribute.KeyValue {
    switch s := rs.(type) {
    case *stats.InPayload:
        if req, ok := s.Payload.(*pb.MyRequest); ok {
            return []attribute.KeyValue{
                attribute.String(<some-label>, req.GetField()),
            }
        }
    case *stats.End:
        if s.Error != nil {
            return []attribute.KeyValue{
                attribute.String("error.type", <some-error>),
            }
        }
    }
    return nil
})

The end user should know quite a bit about the internal implementation to be able to gather all the needed fields, depending on the phase.

---

The question is: Would a Labeler pattern from the same otelhttp package be a better approach for this use-case and would integrate better with the codebase? It seems to be a compromise between the two approaches from above, but I might be missing something.

[flc1125]

My initial thought is to implement a middleware similar to the following, which modifies ctx, and then the underlying layer retrieves certain information from the context through some methods.

For different stats.RPCStats, it is necessary to define different states to temporarily store this information, so that the underlying layer can extract it from ctx.

type HandleRPCHandler func(ctx context.Context, rs stats.RPCStats) context.Context

type HandleRPCMiddleware func(next HandleRPCHandler) HandleRPCHandler

Even simplified to the point of removing the concept of middleware, only allowing a single callback to return the context.

[flc1125]

For record only: Labeler name(only), see: #8107 (comment)