Adding resource attributes post-creation (e.g. via auto-discovery)

[vovencij]

What are you trying to achieve?

I would like to have a possibility to add resource attributes via some auto-discovery mechanism, where attribute values are not known at the Resource creation time, but rather become known during application initialization. In case of auto-instrumentation Otel SDK is created very early in process runtime. Some useful information about the process may not be accessible yet.

An example may be a process of discovering an app server name and its version (#1143) by instrumentation, rather than setting these values via environment. Even without instrumentation, discovering and setting a version of an application (or whatever component that makes sense at the Resource level) at runtime is much easier than obtaining it before an application starts (as is needed for setting attributes via environment).

[Oberon00]

I think there are several possible routes we can go here:

• Allow adding attributes to resources at any time (or rather, merging new resources with Tracer/MeterProvider at any time). Removing or changing attributes should IMHO not be allowed.
• Add a second "dynamic" resource that can be changed at any time, keeping the original resource as an actually immutable potion. This may be useful for backends/caching/etc.
• There may be another kind of resource required, for things like pod labels, IP addresses, that can actually change during runtime. But this is not really related to this issue.
• Alternatively, report things that don't work well with Resources as span attributes instead, as suggested in Adding engine definition under resources semantic conventions #1293 (comment) and as proposed in Allow cloud.* ad faas.* resource attributes to be set on span #1265

[Oberon00]

Note, this was also briefly discussed in #515 (comment)

[anuraaga]

I've always wished Resource is an interface to allow implementing a dynamic one. Some things are resolved lazily, or change very rarely, but not never, being able to use Resource for them would be nice.

[Oberon00]

Let's try to not get too much into implementation details of how to allow changing resources (interface / struct / sync / async / auto-detected / explicit) at first and maybe discuss change in semantics and impact on backends/collectors/maybe exporters.

I think this has the potential to break assumptions and optimizations. The resource is supposed to capture identification information about the entity (runtime/process/container/...) producing telemetry. It would be very annoying if the backends identifies the resource as belonging to different entities over the lifetime of a telemetry producer just because some identification information is missing.

For example, a backend might have an entity describing a particular WildFly cluster, which may consist of multiple instances. If the cluster name is only detected later, any spans reported earlier would be mapped to the wrong entity, or otherwise the backend would need to retroactively move previously reported spans to a different entity.
Same thing applies to the faas.id (AWS ARN), host.name, ...

[anuraaga]

Hmm - I've always considered Resource as a means of compression. Instead of sending the same attribute over and over, just send it once. Similar to gzip but doesn't take extra resources to achieve since it's in the protocol.

I never considered Resource semantically and didn't really expect to. There's no unique ID on it (well maybe this was the reason for the service ID I didn't understand well? 😅). So I figured it's just a bag of attributes and that keeps things simple IMO. Yes if for some reason some spans got sent before the resource got resolved the bag would have less attributes, but I don't know if that could really have any worse effect on the backend than the loss of data, which was inevitable since the data didn't exist at the time.

[iNikem]

As an example, we already have host.name resource attribute. And this can easily change during process run time.

[Oberon00]

I assume you mean the actual value of the hostname may change. The resource attribute will always be what was initially set, won't it?

[Oberon00]

Yes if for some reason some spans got sent before the resource got resolved the bag would have less attributes, but I don't know if that could really have any worse effect on the backend than the loss of data, which was inevitable since the data didn't exist at the time.

The loss of data can in theory be prevented at the client side by delaying the sending of spans/metrics (buffering them) until all (expected) resources are available. Whether this is more desirable than data loss is questionable though (What if a problem prevents the resource from ever being detected? Aren't e.g. memory metrics especially interesting also during startup?).

For Dynatrace the effect of the missing attributes is definitely worse. We have a quite entity-centric UI so users will primarily interact with entities like "MyLambdaFunction". If the ARN is missing from a few spans, we will detect a different entity like "AwsLambdaMainClass Java process" for these. This is arguably "wrong data" which is worse than missing data. Experience with customer complaints about similar things tells us that this is something undesirable.

I guess this is more of a problem of the particular attribute's detectability than of the Resource being mutable though. Even if the resource stays mutable, it may be sensible to require some attributes to be set already at SDK startup.

[anuraaga]

Even if the resource stays mutable, it may be sensible to require some attributes to be set already at SDK startup.

I definitely agree with this - it's semantics, e.g., service.name is so important for most backends that we really need it to be static. But at the same time there are not 100% static, but mostly static data or 100% static after the first request, not startup attributes. I think they should be able to take advantage of our protocol's compression strategy.

[Oberon00]

I agree. But I think we should distinguish at least the "immutable and known from the beginning" part from the others. I think we should also distinguish "immutable" from "might change".

Another advantage of treating the mutable/delayed attributes as a separate concept is that we stay 100% compatible with backends that e.g. receive OTLP but do not expect the data in the existing resource protobuf field to change.