Should event fields be part of the global attribute registry?

[martinkuba]

This issue is intended as a discussion about the approach of defining event fields.

Background

Events are a new concept in OpenTelemetry that is in the process of being defined. An event will consist of a name and a body (payload). The name will be represented as the event.name attribute. The body payload will typically consist of a collection of fields (name/value pairs). An event may also have additional attributes that are separate from the body.

The discussion about field names in this issue refers to fields that are part of the event body, not the additional attributes. Additional attributes will follow the same rules as attributes on spans.

Discussion

Should fields in the event body be defined in the global attributes registry?

Here is an example event representing a browser page load (from this PR). The field names are not namespaced and their definition would be only in the scope of this specific event.

{
	"attributes": {
		"event.name": "browser.page_view" 
	},
	"body": {
		"referrer": "",
		"type": "",
		"title": "",
		"url": "",
	}
}

Pros:

• shorter names
• simpler to define (because they would be vetted only by experts familiar with the event, no discussion necessary how these fit in the global registry)
• global registry will not be polluted with many one-off attributes

Cons:

• some fields would be duplicated
• definition of event fields is handled differently than attributes

Fields defined in the attribute registry

If the fields for the same event were defined in the registry, they could hypothetically look like this:

{
	"browser.document.referrer": "",
	"browser.page.type": "",
	"browser.page.title": "",
	"url.full": "",
}

Note that both url.full is an existing attribute that is re-used in this event. However, the browser.document.referrer also contains a full URL, but must have a different name here to distinguish from page URL and also to provide the meaning of the field.

Pros:

• consistent with how metrics use attributes
• field names could be reused without duplicating definition of the format and content

Cons:

• there might be many fields in the registry that are used only once
  • this might make the registry harder to manage
  • might make it harder to define payloads of new events
• larger payload size on the wire

Hybrid

There is a third option where an event payload could contain both fields defined only for the specific event and attributes from the global registry.

{
	"referrer": "",
	"type": "",
	"title": "",
	"url.full": "", 
}

Event fields defined externally

There might be cases when it's better to not define the content of a payload in OpenTelemetry semantic conventions, but instead link to an external definition. An example of this might be the data from the W3C ResourceTiming API. In this case, if the field definitions were duplicated in the attributes registry, they may get out of sync with the external definition.

If fields defined externally are used in the event body, they would not be namespaced.

[MSNev]

Just to document my take on this and this will include some generic definitions on how "events" should be defined.

By Default for the payload (fields)

• The event fields SHOULD not be required to be fully namespaced (and MUST not be mandated)
• The fields SHOULD not be required to be in the global registry (ie. an event may include whatever fields it needs to describe its payload)
• The fields included into the payload (for OpenTelemetry defined events) MUST be defined in semantic conventions, with the event.name and each field MUST be described (required / optional) and any description.
• Each field MAY reference out to a definition of the global semantic convention, if the field needs to conform to an already well defined semantic convention, but does not want / need to use the same fully qualified name (eg. due to runtime constraints , there are multiple fields that would confirm to the same definition or because the field names are defined by an external definition but OpenTelemetry has already documented the definition)
• Different events MAY include the same field name within their own payload and this DOES NOT infer that they are the same concept as all fields of a payload MUST be considered to be unique to each specific event.name. However, as identified above the field definition above MAY link out to a global semantic convention to provide this weak assertion (if required).
  • Even with the assertion fields of different events SHOULD NOT generally be a candidate for grouping (unless otherwise specified between specific events event.name, eg. Event A MAY defined the same field as Event B AND define these as representing the same value, so they could be used for some form of grouping / metric (if required).

Optionally

• Each domain specific event event.name is free to define its own set of payload fields
• Each field MAY use the fully namespaced global registry defined semantic convention, it is up to the domain specific group defining the event
• The payload of an event is NOT REQUIRED, again, it's up to the specific event domain to determine how to best represent the event (eg. there may be some events that do not require a payload and thus do not need to specify an empty payload)
• As part of the event definition (whether defined by OpenTelemetry, a vender or application event), MAY identify top level attributes (ie not withing the payload) that MAY be expected to be included with the event
  • These additional (globally namespaced) attributes MAY also be defined as required / optional (as required by the event definition)

Optional (additional) Attributes (not the payload fields)

• These attributes SHOULD obey the normal Semantic Convention rules for Attributes
• An event MAY include as many additional attributes as required by the end-user of that event

[jack-berg]

I’ve stated my opinion on this before and engaged in lengthy discussions in my Event payload proposal document. I’ll recap my stance and offer some old and new arguments:

Event fields should be part of the global attribute registry and should be subject to the same naming rules as other attributes.

In a nutshell, the discussion comes down to a question of whether the constrained environments (limited network bandwidth, CPU, package size) of certain client scenarios should influence how OpenTelemetry models semantic conventions for Events (and presumably other signals as well). The answer should be no. We should clearly decouple modeling the shape of telemetry (i.e. semantic conventions) from how telemetry is sent over a network. Accepting that Event fields are different from attributes on other Signals would result in:

• Additional context required to interpret fields. The most important characteristic of the global attribute registry is that it gives the ability to interpret all attributes in the same way regardless of the signal or context. Special casing Events breaks this, requiring the user and backends to understand the semantics of a particular field in the context of the event.name it is attached to. This is a bad user experience and backend experience.
• A lack of ability to correlate fields between signals. Spans, Metric, and Log attributes require namespaces and any overlap between Event fields and Span / Metric / Log attributes would be inconsistent. I.e. Event field “foo” would have to be something like “namespace.foo” on a Span / Metric / Log. This breaks part of the core design philosophy of OpenTelemetry, which prioritizes cross cutting concerns with all observability signals sharing certain concepts and working together to form a complete picture.
• A performance-justified race to zero. If constrained environments didn’t exist, this issue wouldn’t exist. It would be obvious to treat Event fields the same as other attributes. Letting the constraints of a particular type of environment jeopardize the design coherence of the broader semantic conventions is a slippery slope. If we start omitting namespaces from Event fields justified by smaller payloads, why wouldn’t we take it a step further and question the length of the name of every field? E.g. we could abbreviate “click” with “c” to save 4 characters. Why not take it a step further and apply the same logic to Span / Metric / Log attributes? Surely the constrained environment conditions that justify short field names for Events also prefer short field names for Spans and Metrics? We’re obviously going to include Spans / Metrics / Logs in constrained environments and will have to solve protocol level issues related to payloads for these signals. We should solve the protocol problem and apply the solution (whatever it may be) uniformly to Spans, Metrics, Logs, and Events. We should clearly separate the problem of transmitting data in constrained environments from the data modeling problem of semantic conventions. Semantic conventions should prioritize coherence and clarity, naming fields in a way that helps humans understand their meaning rather than to minimize bytes transmitted.

Another aspect of the discussion is whether including Event fields in the global attribute registry is unnecessarily cumbersome for people looking to contribute Event semantic conventions. Do we want a bunch of domain-specific one-off attributes? Do we want contributors of Event semantic conventions to have to understand the universe of attributes such that they re-use an existing attribute when the concept has the same semantics? Do we want domain experts to have to define their Events in collaboration with semantic convention maintainers? Yes, yes, and yes. The process of defining semantic conventions is intended to be low overhead and accommodate domain specific concepts. Defining Event semantic conventions should be subject to the same process and scrutiny as any other convention. There’s no difference between a group wanting to define domain specific Event conventions and wanting to define domain specific Span conventions. Both require defining how domain specific concepts fit into semantic conventions.

Up to this point, I’ve been expressing my own opinion. However, we talked about this issue in the 11/15/23 TC meeting and there was agreement among the present TC members that Event fields should follow the same process as other attributes defined in semantic-conventions. We discussed that OTLP as currently defined may not be suitable for certain extremely resource constrained environments, and that the Client WG in collaboration with the Event WG should work to prototype changes to the protocol to accommodate these environments. Ideas included various versions of the symbol dictionary, or using a columnar format like the OpenTelemetry Arrow Project, or even a stateful domain specific protocol. Note that this opinion from the TC reflects consistency on a previous vote about whether metric attributes should be namespaced. The last comment on that issue is particularly relevant here:

I think the main decision here was that they are conceptually namespaced and that's how we will describe them in semantic conventions. On this basis, any transmission optimizations can still be built.

An interesting aspect of this problem is that it is not limited in scope to the Client WG, or to Event WG. It involves questions of data modeling that squarely intersect with the Semantic Convention WG, it has implications on how the different telemetry signals interact with each other which is a clear specification level issue which is maintained by the TC, and it has implications on all maintainers since Events will be used outside of the client instrumentation space.

So far, we’ve heard a lot from the perspective of the Client and Event WG, and we’ve discussed this in the TC. I’m interested in whether there are any additional perspectives from other semantic-conventions contributors (other than the maintainers who are also on the TC) and from language maintainers? Are all the arguments on the table or are there additional things we haven’t yet considered?

[MSNev]

Up to this point, I’ve been expressing my own opinion. However, we talked about this issue in the 11/15/23 TC meeting and there was agreement among the present TC members that Event fields should follow the same process as other attributes defined in semantic-conventions. We discussed that OTLP as currently defined may not be suitable for certain extremely resource constrained environments, and that the Client WG in collaboration with the Event WG should work to prototype changes to the protocol to accommodate these environments. Ideas included various versions of the symbol dictionary, or using a columnar format like the OpenTelemetry Arrow Project, or even a stateful domain specific protocol. Note that this opinion from the TC reflects consistency on a previous vote about #51 The #51 (comment) on that issue is particularly relevant here:

I think the main decision here was that they are conceptually namespaced and that's how we will describe them in semantic conventions. On this basis, any transmission optimizations can still be built.

I want to document my VERY STRONG disagreement with this approach on several levels that I'm not quite sure where to start...

Strongly agree:

• OTLP as currently defined may not be suitable for certain extremely resource constrained environments
• Yes, we should look at creating a "better" (smaller) transport payload, but not one that requires any form of special dynamic tables to be calculated during payload serialization. In my opinion OTLP (could) be made better (and slightly more workable) by introducing a few tweaks and defaults (although these may not be supported by existing protobuf definitions / implementations)

Strongly disagree:

• Event fields should follow the same process as other attributes defined in semantic-conventions.
  • Not all events will be defined by OpenTelemetry
  • There will be 1000's more "Custom" events then there will be OpenTelemetry defined events and requiring that every "Custom" event MUST have every "field" for the custom event prefixed is just not viable for most applications.
  • This will "force" application custom level events to end up using AnyValue just to avoid the prefix for the EXACT same reasons that I'm pushing back on this for the OpenTelemetry defined events.
• Prefixing every field of an event with name of the event (ie. the event.name) is what you are proposing here as that is the ONLY viable option, prefixing with just the namespace of the event WILL cause clashes with different fields in the same domain namespace group.
  • Bundle sizes in some languages (runtime constraints on the size of the code - ie browsers), strings are not compressable
  • Prefixing values to runtime defined fields just causes additional memory (and CPU) usage which are then batch (holding onto that in memory) and then during transport serialization this generates further excessive bytes (further compounding the OTLP transport problems), and using some sort of client side "compressing" (symbol disctionary, etc) does not help this situation when you have LOTS of unique events with unique namespaces as every field name is now unique and has to be listed in the symbol table (where it's most likely only EVERY used once).
• Prototyping some form of client side "compression" during transport
  • Transport payload serialization is a SUPER CRITICAL time sensitive operation and additional "time" taken to "compress" MUST be avoided at all costs.
  • There are also situations where the final payload "size" is limited (by the runtime) and can be non-deterministic (a maximum of 64kb but may be a lot smaller) so "if" there is some form of "compression" (symbol table, gzip, etc) this can require multiple "passes" (because if the payload is too large you have to perform the entire operation again (potentially checking the usage of every field to see if any "symbol" should be included), and this may need to be repeated multiple times.
• Serialization / deserialization can also represent security issues for the libraries that are performing either, to the point that some organizations (including Microsoft) require to usage of only certified serialization libraries, with the preference for JS runtimes to use the platform JSON implementations, which also has the advantage of being the "fastest" (using the native serialization) conversions and therefore having the lowest impact (not withstanding the above comments about the creating of the long strings and the current OTLP format) on any "waiting" user -- server environments DON'T have this issue ALL client environment DO (or CAN - some can offload to another thread -- but not all).

Possible "tweaks" which would make the "logical" prefixing of an event payload "fields"

• All of the payload fields for the events are "grouped" as the payload for an event and when present in the global registry they are "listed" with the prefix, BUT are NOT sent in this manner, they are still SENT (without the prefix) in a payload Attribute (event.data / body)

A "general" OTLP transport improvement(s)

• All "Attributes" with the same namespace (including ALL of the ones defined in the registry) as sent in a containing AnyValue (should really be a new special type) where the name of the top level Attribute is the namespace with the value being a Map of attributes with the namespace removed (just the name)
• eg. All http conventions would be in a sub-object http: { method: "GET", ... } etc
• Remove the "need" for every attribute to have it's type (at least for the JSON representation - binary formats will always need to define the "encoding")). That way "strings", "numbers" (caveat with int / float), "arrays" "map" (represented as an object) and all other types would still "need" there "type. There is also a "special" need required for when transmitting fields that "may" requied metadata, to identify fields that may contain PII information so that backends can handle as required, depending on the provider / vender.

I'm going to (try to) stop now and I'm sure we will (and need) to discuss further in the Event WG (and other SIGs), for reference I have create a draft white paper (not yet public -- being reviewed and may not be able to be fully public) on why Microsoft is not yet using OpenTelemetry for the Web (some of this is summarized above, and I've talked in lots of SIG's about the reasons), this draft internal paper is 17 pages long with all of the issues and what is required.

And for clients breaking things down and trying to address (and / or punt) different concerns down the road doesn't work because they are all inter-related to different degrees because

• the specifications drive the internal representations in-memory representation and the code required to support
• the transport format along with the specification also drives the in-memory representations and the code required to support
• the combination of the above drives the conversions (DTO's) required to go from the user (application) input, to the enforced specifications (readonly in some cases) and the into a form that can be serialized and the serialization process can then require further "processing"

To Date OpenTelemetry (appears) (IMHO) to have been driven by the "final" storage format used by many organizations and then creating wrappings around these "storage" formats for creating on servers to enable the data to be quickly dumped, and for high volume servers this is an excellent approach -- it just doesn't work for ALL client scenarios / languages.

[jack-berg]

There will be 1000's more "Custom" events then there will be OpenTelemetry defined events and requiring that every "Custom" event MUST have every "field" for the custom event prefixed is just not viable for most applications. This will "force" application custom level events to end up using AnyValue just to avoid the prefix for the EXACT same reasons that I'm pushing back on this for the OpenTelemetry defined events.

No. The attribute naming rules use strong normative language OpenTelemetry authors, but we can't enforce what application authors do for custom events. At best, we can define recommendations. For events defined outside OpenTelemetry and bridged over, we should minimally transform the original payload with mapping rules. The naming rules for are OpenTelemetry semantic conventions.

Bundle sizes in some languages (runtime constraints on the size of the code - ie browsers), strings are not compressable
Transport payload serialization is a SUPER CRITICAL time sensitive
Prefixing values to runtime defined fields just causes additional memory (and CPU) usage which are then batch (holding onto that in memory) and then during transport serialization this generates further excessive bytes

We shouldn't jeopardize the coherence and clarity of the telemetry semantic conventions for the requirements of extremely constrained environments. We must decouple data modeling from transmission, and work to accommodate constrained environments with transmission level solutions.

Possible "tweaks" which would make the "logical" prefixing of an event payload "fields". All of the payload fields for the events are "grouped" as the payload for an event and when present in the global registry they are "listed" with the prefix, BUT are NOT sent in this manner, they are still SENT (without the prefix) in a payload Attribute (event.data / body)

This is an example of a "transmission optimization" described in this comment, which is compatible with the notion of treating event fields as attributes.

Remove the "need" for every attribute to have it's type (at least for the JSON representation - binary formats will always need to define the "encoding")).

You're describing a breaking change to the OTLP JSON protobuf encoding. A variation which doesn't include the type is possible with a new protocol. This is still possible, since as this comment points out, "any transmission optimizations can still be built".

the specifications drive the internal representations in-memory representation and the code required to support
the transport format along with the specification also drives the in-memory representations and the code required to support
the combination of the above drives the conversions (DTO's) required to go from the user (application) input, to the enforced specifications (readonly in some cases) and the into a form that can be serialized and the serialization process can then require further "processing"

It sounds like your issues go well beyond the representation of Event fields, bubbling into the SDK specification. The SDK specification are stable of course, and its hard to imagine evolving them in a way that would accommodate the extreme requirements you describe. If the issues exist for all client use cases, it may suggest that a domain specific SDK might need to be developed for the specialized conditions which is not subject to the normal SDK specification language. If the issue only occurs in client application of a certain scale (or under certain constraints), it may be worth considering developing a custom SDK for your specific use case. The OpenTelemetry decoupling of API and SDK was designed for this purpose.

[martinkuba]

My view is the following -
I agree that we should decouple the conversation about data modeling and performance overhead. There is no guarantee that attribute names without namespaces will be enough to address performance requirements of some applications. This should be addressed by choosing different exporters/protocols and perhaps even different SDK implementations, and application owners can choose which makes the most sense for their requirements. I also think that we need to just focus on data modeling at this point to make progress.

With that said, I am not 100% convinced that event payload fields need to be in the same global registry as regular attributes. Currently, attribute semantic conventions have a global scope, while event fields are scoped to the particular event. The combination of the event name and a field name should be enough to understand what the field represents. And in order to enable reusability, perhaps it should be allowed to use an attribute from the global registry as an event field.

I want to give one example. Assume that there is an event that represents navigation in a browser environment. We want to capture the previous URL and the current URL. We cannot reuse url.full here because its name does not capture the meaning of those fields (even though the value it holds does). The fields should be something like previouls.url and current.url. Does it add a value to add this to the global registry?

I would like to hear from @open-telemetry/specs-semconv-maintainers and @open-telemetry/specs-semconv-approvers about this topic. If there is a consensus that event fields should be in the global registry from this group and the TC, then I will be in favor as well in order to make progress.

[pyohannes]

Should fields in the event body be defined in the global attributes registry?

If the answer to this is "no", where else would those fields be defined? Wouldn't they be defined at all? Or would there be another event-specific registry independent of the semantic conventions attribute registry?

[AlexanderWert]

I agree with everything that @jack-berg wrote in #505 (comment).

Some additional perspective:

• We are merging Elastic Common Schema (ECS) into OTel semantic conventions
• A big portion of ECS (and the value it provides to the ECS community) comes from the area of describing events and their payload cross signal and cross use cases. And it's specifically useful to have a common, standardised way of mapping data for common, repetitive use cases into semantic fields that have a common meaning also across single use cases (e.g. a client.geo.* has always the same meaning independent whether it's on a span, comes from an NGINX access log, or is attached to a mobile error event).
• So, as a learning from the ECS and it's community (that we are merging into the OTel community with the ECS donation) standardising on common fields / attributes (also in payloads) is something that provides high value and is very well appreciated by the community.
• Of course there can be custom events with custom fields that are not documented and not standardised. But, when considering common use cases that we explicitly write semantic conventions for (e.g. how to parse NGINX logs into a structured event) we should use semantic attributes.

I want to give one example. Assume that there is an event that represents navigation in a browser environment. We want to capture the previous URL and the current URL. We cannot reuse url.full here because its name does not capture the meaning of those fields (even though the value it holds does). The fields should be something like previouls.url and current.url. Does it add a value to add this to the global registry?

@martinkuba In ECS, we have the concept of reusing field definitions in other namespaces to address exactly this use case. For example, we define geo.* fields here and in addition specify in which other namespaces these fields can be nested. So, on a single event you could have something like client.geo.location and server.geo.location to differentiate the context.

We also discussed in the SemConv WG to introduce a similar concept in OTel semantic conventions and have agreement that it's something we want to do. (Next steps is to work on a proposal for the syntax and tooling to realize it). Maybe that would help with the above concern.

[scheler]

A big portion of ECS (and the value it provides to the ECS community) comes from the area of describing events and their payload cross signal and cross use cases.

@AlexanderWert can you please point to an example in ECS describing the fields of payload for a given event type?

[AlexanderWert]

@AlexanderWert can you please point to an example in ECS describing the fields of payload for a given event type?

@scheler Sure, it's mainly defined by the Elastic integrations which are with their documentation pages to some extent comparable to the semantic conventions in OTel.

Let's consider a very concrete example: NGINX Ingress Controller. That page describes in very detail, how an access log from the ingress controller is parsed into ECS fields. In OTel terms, that page describes the semantic conventions for NGINX Ingress Controller logs.

(Please bear in mind that with ECS there's no differentiation between top-level fields, attributes and payload as it is with OTel / OTLP. But, that doesn't make a difference for the below reasoning regarding payload).

In the "Exported fields table" you will find both:

• many ECS fields (related to the above discussion: this is comparable with the attributes registry)
• but also NGINX-specific fields (these would be the use case specific fields)

Some inspirations we can get from this example are:

1. where meaningful, we should use generic semantic conventions fields
2. sometimes it makes sense to have use-case specific fields
   • BUT: those should still have a meaningful namespace / prefix to be easily searchable / filterable without the need of additional attributes to define the context. (e.g. nginx_ingress_controller.access.upstream.name instead of just upstream.name)
   • AND: even the domain specific fields need to be documented in semantic conventions in a consistent way (as it helps users to better understand and consume data)

As part of the ECS contribution we also hope to contribute semantic conventions and parsers / integrations for many of these integrations to the OTel ecosystem (which will turn the ECS addition into actual, consumable value for the community). The event / logs signal will be crucial for that.

[martinkuba]

If the answer to this is "no", where else would those fields be defined? Wouldn't they be defined at all? Or would there be another event-specific registry independent of the semantic conventions attribute registry?

@pyohannes They would still be defined here, but along with the definition of the event. For every event name, there would be a list of fields that should be included in the payload for that event. I suppose this is similar to how attributes are associated with metric names right now.