feat: adding configurable deployment.name in otel

[vanhtuan0409]

Motivation

Adding configuration to tag otel trace and metrics with deployment.environment attribute. Supporting signoz environment
https://signoz.io/docs/faqs/general/#q-how-to-setup-signoz-across-different-environments

Solution

As above

[vanhtuan0409]

I'm being conservative not to add an attribute if the operator does not explicitly set the config. It is up for discussion whether we want to use a default value or keep the conservative approach

[svix-jbrown]

This doesn't seem unreasonable, but a couple of questions to think about...

Question 1: I see that signoz recommends deployment.environment, but the official opentelemetry docs recommend deployment.environment.name; do you have any ideas as to which key is more commonly-used? Does signoz also support the "official" one?

Question 2: Do we want to special case deployment.environment, or add something like opentelemetry_resource_attributes: HashMap<String, String> to let people set other tags? Most otel applications just read from $OTEL_RESOURCE_ATTRIBUTES (although ours does not), which would be another option.

[svix-james]

Question 2: Do we want to special case deployment.environment, or add something like opentelemetry_resource_attributes: HashMap<String, String> to let people set other tags? Most otel applications just read from $OTEL_RESOURCE_ATTRIBUTES (although ours does not), which would be another option.

I would argue that we leave this fairly well defined for now, ie, don't allow attributes that we haven't explicitly set through configuration parameters.

[vanhtuan0409]

@svix-james searching through signoz source code, I dont think they support deployment.environment.name. Acknowledge that otel semconv is not strictly followed

Anw should we proceed with this PR

[svix-james]

Ok, apologies, I didn't read @svix-jbrown's previous message closely. We should definitely not support a non-standard resource name here.

[svix-james]

Honestly at this point I would just argue for using the original proposal (#2110), i.e., passing by environment variable, and being done with it. The reason we wanted to avoid this is to prevent inadvertently sharing OTEL parameters between Svix and user-specific OTEL streams downstream, but I think we can avoid that if we're careful.

@vanhtuan0409 Have you confirmed that setting the OTEL env var actually works for your non-standard attribute name? If so, please update this PR (no need to close/open another one) with the changes from #2110, and we can get this in.