Spec service.environment for logs

specs/agents/log-correlation.md

@@ -1,7 +1,64 @@
 ## Log correlation
 
-Agents should provide instrumentation/hooks for popular logging libraries in order to decorate structured log records with trace context. In particular, logging that occurs within the context of a transaction should add the fields `trace.id` and `transaction.id`; logging that occurs within a span should add the fields `trace.id`, `span.id`, and optionally `transaction.id`.
+Agents should provide instrumentation/hooks for popular logging libraries in order to decorate structured log records with trace context.
+In particular, logging that occurs within the context of a transaction should add the fields `trace.id` and `transaction.id`;
+logging that occurs within a span should add the fields `trace.id` and optionally `transaction.id`.
 
 By adding trace context to log records, users will be able to move between the APM UI and Logs UI.
 
-See also the [log-reformatting](log-reformatting.md) spec.
+Logging frameworks and libraries may provide a way to inject key-value pairs in log messages,
+this allows to reuse those fields in log message formats (for example in plain text).
+
+Log correlation relies on two sets of fields:
+- [metadata fields](#service-correlation-fields)
+  - They allow to build the per-service logs view in UI.
+  - They are implicitly provided when using log-sending by the agent metadata.
+  - When using ECS logging, they might be set by the application.
+- [per-log-event fields](#trace-correlation-fields): `trace.id`, `transaction.id` and `error.id`
+  - They allow to build the per-trace/transaction/error logs view in UI.
+  - They are added to the log event
+  - They must be written in each log event document
+
+The values for those fields can be set in two places:
+- when using [ecs-logging](https://github.com/elastic/ecs-logging) directly in the application
+- when the agent reformats a log event
+
+The values set at the application level have higher priority than the values set by agents.
+Agents must provide fallback values if they are not explicitly set by the application.
+
+In case the values set in the application and agent configuration differ, the resulting log
+messages won't correlate to the expected service in UI. In order to prevent such inconsistencies
+agents may issue a warning when there is a mis-configuration.
+
+### Service correlation fields
+
+They allow to build the per-service logs view in UI.
+They are implicitly provided when using log-sending by the agent metadata.
+When using ECS logging, they might be set by the application in ECS logging configuration.
+
+- `service.name`:
+  - used to filter/link log messages to a given service.
+  - must be provided even if there is no active transaction
+- `service.version`:
+  - only used for service metadata correlation
+  - must be provided even if there is no active transaction
+- `service.environment`:
+  - allows to filter/link log messages to a given service/environment.
+  - must be provided even if there is no active transaction
+
+
+The `container.id` field can also be used as a fallback to provide service-level correlation in UI, however agents ARE NOT expected to set it:
+
+- log collector (filebeat) is expected to do that when ingesting logs.
+- all data sent through agent intake implicitly provides `container.id` through metadata, which also includes the log events that may be sent to apm-server.
+
+### Trace correlation fields
+
+They allow to build the per-trace/transaction/error logs view in UI.
+They allow to navigate from the log event to the trace/transaction/error in UI.
+They should be added to the log event.
+They must be written in each log event document they relate to, either reformatted or sent by the agent.
+
+- `trace.id`
+- `transaction.id`
+- `error.id`

specs/agents/log-reformatting.md

@@ -65,26 +65,17 @@ The following fields are required:
 
 ## Recommended fields
 
-The following fields are important for a good user experience in Kibana,
-but will not cause errors if they are omitted:
-
 ### `service.name`
 
-Agents should always populate `service.name` even if there is not an active transaction.
-
-When using [ecs-logging](https://github.com/elastic/ecs-logging) directly in the application, the logs are already
-written to ECS format, however the agent has to provide a fallback value if `service.name` is not set by the application.
-
-The `service.name` is used to be able to add a logs tab to the service view in
-the UI. This lets users quickly get a stream of all logs for a particular
-service.
+See [Log correlation](log-correlation.md)
 
 ### `service.version`
 
-Agents may populate the `service.version` if a value is set in the agent configuration.
+See [Log correlation](log-correlation.md)
+
+### `service.environment`
 
-When using [ecs-logging](https://github.com/elastic/ecs-logging) directly in the application, the logs are already
-written to ECS format, however the agent should provide a fallback value if `service.version` is not set by the application.
+See [Log correlation](log-correlation.md)
 
 ### `event.dataset`
 

specs/agents/logging.md

@@ -43,7 +43,7 @@ Also, the documentation should not mention the old log levels going forward.
 
 Agents may provide the following log-related features:
 
-- [Log correlation](log-correlation.md): inject current trace/transaction/error ID into logs.
+- [Log correlation](log-correlation.md): inject service metadata, trace IDs and error IDs in log events.
 - [Log reformatting](log-reformatting.md): reformat plain-text logs to ECS, equivalent to using [ecs logging](https://github.com/elastic/ecs-logging) 
 without modifying the application nor its dependencies.
 - [Log sending](log-sending.md): send logs directly to APM server.