Add a log observed timestamp processor that preserves request time in context #4503

Pre-filing checklist

• I searched existing issues and didn't find a duplicate

Component(s)

Rust OTAP dataflow (rust/otap-dataflow/)

Is your feature request related to a problem?

Yes. Log records sent through OTAP Dataflow pipelines can arrive without an observed timestamp. When observed_time_unix_nano is missing or zero, downstream consumers lose the time the log was received. This makes it harder to measure ingestion delay, order or troubleshoot late-arriving data, and tell delayed collection apart from an incorrect time_unix_nano.

Receivers may record the time a request was received, but no processor currently uses that value to fill missing log observed timestamps. If observed timestamps are filled later from the current time instead, records from the same request can get different values. That value also reflects when processing happened, not when the collector received the request.

Downstream processors also have no consistent way to read the same request-received time. For example, processors that copy it into an attribute or use it for routing or diagnostics could see different values.

This is especially visible with HTTP/JSON and SDK sources that legitimately omit observed_time_unix_nano, expecting the collector to set it at ingestion.

Proposed Solution

Proposed behavior
For each incoming logs pdata message:

1. Read the configured named request-context timestamp, expressed as Unix nanoseconds.
2. If the receiver already captured request receipt time under that name, preserve and reuse it.
3. If the context entry is absent, capture the current time once and store it under that name.
4. Use that timestamp to populate each log record's missing, null, or zero observed_time_unix_nano.
5. Preserve existing nonzero observed timestamps and event time_unix_nano values.
6. Forward the context timestamp downstream so other processors can read it.

All records needing population within one pdata message receive the same timestamp. An existing context timestamp is never replaced by a new clock reading.

Timestamp meaning: actual request receipt time must be captured by the receiver. If it was not captured, this processor can only record the time it observes the pdata message; it cannot reconstruct the original receipt time. The fallback must be documented as processor observation time.

Metrics and traces pass through unchanged, including their context. The processor does not add log fields or generate timestamp context entries for those signals.

Proposed configuration

type: processor:log_observed_timestamp
config:
  name: request_received

• name selects the context entry to read or create.
• Default: observed_timestamp.
• An empty or whitespace-only name is invalid.
• Including the processor in a pipeline enables it; no separate enabled setting is necessary.

Example composition

Place the timestamp processor after ingestion and before any attribute processor that consumes its context entry:

nodes:
  observed-time:
    type: processor:log_observed_timestamp
    config:
      name: request_received

  // Proposed extension from issue # 4504
  request-attributes:
    type: processor:attribute
    config:
      apply_to: [resource]
      actions:
        - action: insert
          key: request.received_unix_nano
          value:
            from_context: request_received
          signals: [logs]

connections:
  - from: observed-time
    to: request-attributes

This is an illustrative pipeline fragment. The attribute action depends on the separately proposed context-source enhancement, but the timestamp processor must work independently.

Existing nonzero log observed timestamps may differ from the request context value. Preserving those values is intentional; the resource attribute represents the message's captured/generated request timestamp, not necessarily every record's original observed time.

Implementation considerations

• Use shared pdata context APIs and preserve the timestamp through downstream forwarding.
• Generate at most one fallback timestamp per logs pdata message, even when it contains multiple resources, scopes, or records.
• Populate both absent timestamp columns and missing values in existing columns for OTAP inputs.
• Preserve payload/schema metadata and nonzero timestamps.
• Avoid rebuilding timestamp columns when no record needs population. Where context creation is the only change, preserve the payload representation when feasible.
• Explicitly handle clock failures and values outside the supported timestamp range; do not silently wrap or substitute a success-shaped value.
• Preserve thread-per-core locality and follow upstream instrumentation, configuration, and error/ack/nack conventions.

Acceptance criteria
[ ] Receiver-captured context timestamps are reused without replacement.
[ ] Absent context timestamps are generated once per logs message and forwarded downstream.
[ ] Missing, null, and zero log observed timestamps are populated.
[ ] Existing nonzero observed timestamps and event timestamps remain unchanged.
[ ] Multiple resources/scopes/records share one fallback value per message.
[ ] Context is established even when all log records already have observed timestamps.
[ ] Metrics and traces pass through unchanged.
[ ] OTLP protobuf and OTAP Arrow inputs are tested, including absent timestamp columns.
[ ] Invalid configuration, clock/range errors, and context propagation are tested.
[ ] Documentation explains request receipt time versus processor-time fallback.
[ ] Documentation, examples, and a user-facing changelog entry are added.

Alternatives Considered

We considered adding this as a setting on processor:attribute, alongside the proposed context-sourced attribute values. We rejected it because the attribute processor would then also change log record fields and generate context values. A dedicated processor keeps responsibilities clear, and the attribute processor can still read the timestamp through from_context.

Have each receiver populate observed_time_unix_nano directly
Receivers are the right place to capture the time a request was received. However, writing the log field in every receiver would duplicate decode/re-encode logic across receivers. It would also add per-record work to the ingestion path, and pipelines would have no single, configurable place to control or disable the behavior. Optionally, capturing the time in the receiver and populating the field in a processor keeps receivers lightweight.

Use the transform processor (OPL/KQL)
An expression could set observed_time_unix_nano when it is missing. However, this would require exposing request-context values to the query languages. It also gives no guarantee that one timestamp is generated per message and stored for downstream processors. That is a broader change than this use case needs.

Fall back to the event time_unix_nano
Copying the event time into the observed time is simple. However, it conflates two OpenTelemetry fields with different meanings, so ingestion latency can no longer be measured or late-arriving data detected.

Generate the current time per record
Calling the clock for each record is straightforward. However, records from the same request would get inconsistent values, it adds per-record overhead, and no value is left in context for downstream processors.

Defer to the backend or exporter
Backends can stamp data on arrival. However, that time includes collector buffering, batching, retries, and export delay, so it does not represent when the collector received the data.

Additional Context

OpenTelemetry semantics

The OpenTelemetry Logs Data Model defines ObservedTimestamp as the time the collection system observed the event. For externally sourced events, it is measured by the OpenTelemetry component's clock. In OTLP, a value of 0 means the timestamp is unknown or missing. This processor fills only those missing values and leaves any nonzero value set by an SDK or upstream collector unchanged.

• Logs Data Model: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-observedtimestamp
• OTLP LogRecord.observed_time_unix_nano: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto

Relationship to other work

• Companion issue: # TBD, which proposes from_context / from_headers value sources for processor:attribute. That issue can project this processor's context timestamp into an attribute, but neither issue depends on the other.
• This processor relies on the pdata context's named request values (request_context_u64 / set_request_context_u64). If a receiver records the time it received the request under the configured name, this processor reuses that value. Otherwise, it records the processor's own observation time.

Pipeline placement
Place the processor as early as possible after ingestion: after receivers and decoding, and before batching, routing, or buffering. Placing it later adds pipeline latency to any fallback timestamp. Any processor that reads the timestamp from context must run after this one.

Scope
Out of scope for this issue:

• changing time_unix_nano;
• adding observed timestamps to metrics or traces, whose data models have no equivalent field;
• changing receiver capture behavior.