diff --git a/specs/agents/log-correlation.md b/specs/agents/log-correlation.md index 9c5ff22a..e22e3716 100644 --- a/specs/agents/log-correlation.md +++ b/specs/agents/log-correlation.md @@ -1,7 +1,64 @@ ## Log correlation -Agents should provide instrumentation/hooks for popular logging libraries in order to decorate structured log records with trace context. In particular, logging that occurs within the context of a transaction should add the fields `trace.id` and `transaction.id`; logging that occurs within a span should add the fields `trace.id`, `span.id`, and optionally `transaction.id`. +Agents should provide instrumentation/hooks for popular logging libraries in order to decorate structured log records with trace context. +In particular, logging that occurs within the context of a transaction should add the fields `trace.id` and `transaction.id`; +logging that occurs within a span should add the fields `trace.id` and optionally `transaction.id`. By adding trace context to log records, users will be able to move between the APM UI and Logs UI. -See also the [log-reformatting](log-reformatting.md) spec. +Logging frameworks and libraries may provide a way to inject key-value pairs in log messages, +this allows to reuse those fields in log message formats (for example in plain text). + +Log correlation relies on two sets of fields: +- [metadata fields](#service-correlation-fields) + - They allow to build the per-service logs view in UI. + - They are implicitly provided when using log-sending by the agent metadata. + - When using ECS logging, they might be set by the application. +- [per-log-event fields](#trace-correlation-fields): `trace.id`, `transaction.id` and `error.id` + - They allow to build the per-trace/transaction/error logs view in UI. + - They are added to the log event + - They must be written in each log event document + +The values for those fields can be set in two places: +- when using [ecs-logging](https://github.com/elastic/ecs-logging) directly in the application +- when the agent reformats a log event + +The values set at the application level have higher priority than the values set by agents. +Agents must provide fallback values if they are not explicitly set by the application. + +In case the values set in the application and agent configuration differ, the resulting log +messages won't correlate to the expected service in UI. In order to prevent such inconsistencies +agents may issue a warning when there is a mis-configuration. + +### Service correlation fields + +They allow to build the per-service logs view in UI. +They are implicitly provided when using log-sending by the agent metadata. +When using ECS logging, they might be set by the application in ECS logging configuration. + +- `service.name`: + - used to filter/link log messages to a given service. + - must be provided even if there is no active transaction +- `service.version`: + - only used for service metadata correlation + - must be provided even if there is no active transaction +- `service.environment`: + - allows to filter/link log messages to a given service/environment. + - must be provided even if there is no active transaction + + +The `container.id` field can also be used as a fallback to provide service-level correlation in UI, however agents ARE NOT expected to set it: + +- log collector (filebeat) is expected to do that when ingesting logs. +- all data sent through agent intake implicitly provides `container.id` through metadata, which also includes the log events that may be sent to apm-server. + +### Trace correlation fields + +They allow to build the per-trace/transaction/error logs view in UI. +They allow to navigate from the log event to the trace/transaction/error in UI. +They should be added to the log event. +They must be written in each log event document they relate to, either reformatted or sent by the agent. + +- `trace.id` +- `transaction.id` +- `error.id` diff --git a/specs/agents/log-reformatting.md b/specs/agents/log-reformatting.md index f54e866d..4a94d1f4 100644 --- a/specs/agents/log-reformatting.md +++ b/specs/agents/log-reformatting.md @@ -65,26 +65,17 @@ The following fields are required: ## Recommended fields -The following fields are important for a good user experience in Kibana, -but will not cause errors if they are omitted: - ### `service.name` -Agents should always populate `service.name` even if there is not an active transaction. - -When using [ecs-logging](https://github.com/elastic/ecs-logging) directly in the application, the logs are already -written to ECS format, however the agent has to provide a fallback value if `service.name` is not set by the application. - -The `service.name` is used to be able to add a logs tab to the service view in -the UI. This lets users quickly get a stream of all logs for a particular -service. +See [Log correlation](log-correlation.md) ### `service.version` -Agents may populate the `service.version` if a value is set in the agent configuration. +See [Log correlation](log-correlation.md) + +### `service.environment` -When using [ecs-logging](https://github.com/elastic/ecs-logging) directly in the application, the logs are already -written to ECS format, however the agent should provide a fallback value if `service.version` is not set by the application. +See [Log correlation](log-correlation.md) ### `event.dataset` diff --git a/specs/agents/logging.md b/specs/agents/logging.md index 15fae292..df342e01 100644 --- a/specs/agents/logging.md +++ b/specs/agents/logging.md @@ -43,7 +43,7 @@ Also, the documentation should not mention the old log levels going forward. Agents may provide the following log-related features: -- [Log correlation](log-correlation.md): inject current trace/transaction/error ID into logs. +- [Log correlation](log-correlation.md): inject service metadata, trace IDs and error IDs in log events. - [Log reformatting](log-reformatting.md): reformat plain-text logs to ECS, equivalent to using [ecs logging](https://github.com/elastic/ecs-logging) without modifying the application nor its dependencies. - [Log sending](log-sending.md): send logs directly to APM server.