data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 3
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 4
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
The definition of ObservabilityPipelineOcsfMapperProcessorMappingMapping object.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 8
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 9
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 10
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileEncodingType object. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileKeyItemsComparison object. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileSchemaItemsType object. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 11
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 12
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before applying throttling.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
The number of events to allow before throttling is applied.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 4
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 5
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 6
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
name [required]
\nstring
Name of the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 3
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 4
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
The definition of ObservabilityPipelineOcsfMapperProcessorMappingMapping object.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 8
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 9
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 10
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileEncodingType object. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileKeyItemsComparison object. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileSchemaItemsType object. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 11
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 12
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before applying throttling.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
The number of events to allow before throttling is applied.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 4
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 5
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 6
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 3
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 4
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
The definition of ObservabilityPipelineOcsfMapperProcessorMappingMapping object.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 8
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 9
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 10
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileEncodingType object. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileKeyItemsComparison object. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileSchemaItemsType object. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 11
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 12
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before applying throttling.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
The number of events to allow before throttling is applied.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 4
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 5
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 6
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 3
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 4
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
The definition of ObservabilityPipelineOcsfMapperProcessorMappingMapping object.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 8
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 9
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 10
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileEncodingType object. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileKeyItemsComparison object. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileSchemaItemsType object. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 11
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 12
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before applying throttling.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
The number of events to allow before throttling is applied.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 4
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 5
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 6
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines
data [required]
\nobject
Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]
\nobject
Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]
\nobject
Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]
\n[ <oneOf>]
A list of destination components where processed logs are sent.
Option 1
\nobject
The datadog_logs destination forwards logs to Datadog Log Management.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The destination type. The value should always be datadog_logs. \nAllowed enum values: datadog_logs
default: datadog_logs
Option 2
\nobject
The google_chronicle destination sends logs to Google Chronicle.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
customer_id [required]
\nstring
The Google Chronicle customer ID.
encoding
\nenum
The encoding format for the logs sent to Chronicle. \nAllowed enum values: json,raw_message
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
log_type
\nstring
The log type metadata associated with the Chronicle destination.
type [required]
\nenum
The destination type. The value should always be google_chronicle. \nAllowed enum values: google_chronicle
default: google_chronicle
Option 3
\nobject
The new_relic destination sends logs to the New Relic platform.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The New Relic region. \nAllowed enum values: us,eu
type [required]
\nenum
The destination type. The value should always be new_relic. \nAllowed enum values: new_relic
default: new_relic
Option 4
\nobject
The sentinel_one destination sends logs to SentinelOne.
id [required]
\nstring
The unique identifier for this component.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
region [required]
\nenum
The SentinelOne region to send logs to. \nAllowed enum values: us,eu,ca,data_set_us
type [required]
\nenum
The destination type. The value should always be sentinel_one. \nAllowed enum values: sentinel_one
default: sentinel_one
processors [required]
\n[ <oneOf>]
A list of processors that transform or enrich log data.
Option 1
\nobject
The filter processor allows conditional processing of logs based on a Datadog search query. Logs that match the include query are passed through; others are discarded.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs should pass through the filter. Logs that match this query continue to downstream components; others are dropped.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be filter. \nAllowed enum values: filter
default: filter
Option 2
\nobject
The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
field [required]
\nstring
The name of the log field that contains a JSON string.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be parse_json. \nAllowed enum values: parse_json
default: parse_json
Option 3
\nobject
The Quota Processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
drop_events [required]
\nboolean
If set to true, logs that matched the quota filter and sent after the quota has been met are dropped; only logs that did not match the filter query continue through the pipeline.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
ignore_when_missing_partitions
\nboolean
If true, the processor skips quota checks when partition fields are missing from the logs.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
name [required]
\nstring
Name for identifying the processor.
overrides
\n[object]
A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]
\n[object]
A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
limit [required]
\nobject
The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]
\nenum
Unit for quota enforcement in bytes for data size or events for count. \nAllowed enum values: bytes,events
limit [required]
\nint64
The limit for quota enforcement.
partition_fields
\n[string]
A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
type [required]
\nenum
The processor type. The value should always be quota. \nAllowed enum values: quota
default: quota
Option 4
\nobject
The add_fields processor adds static key-value fields to logs.
fields [required]
\n[object]
A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]
\nstring
The field name.
value [required]
\nstring
The field value.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (for example, as the input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be add_fields. \nAllowed enum values: add_fields
default: add_fields
Option 5
\nobject
The remove_fields processor deletes specified fields from logs.
fields [required]
\n[string]
A list of field names to be removed from each log event.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
The PipelineRemoveFieldsProcessor inputs.
type [required]
\nenum
The processor type. The value should always be remove_fields. \nAllowed enum values: remove_fields
default: remove_fields
Option 6
\nobject
The rename_fields processor changes field names.
fields [required]
\n[object]
A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]
\nstring
The field name to assign the renamed value to.
preserve_source [required]
\nboolean
Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]
\nstring
The original field name in the log event that should be renamed.
id [required]
\nstring
A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this component.
type [required]
\nenum
The processor type. The value should always be rename_fields. \nAllowed enum values: rename_fields
default: rename_fields
Option 7
\nobject
The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mappings [required]
\n[object]
A list of mapping rules to convert events to the OCSF format.
include [required]
\nstring
A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]
\n<oneOf>
The definition of ObservabilityPipelineOcsfMapperProcessorMappingMapping object.
Option 1
\nenum
Predefined library mappings for common log formats. \nAllowed enum values: CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication,Palo Alto Networks Firewall Traffic
type [required]
\nenum
The processor type. The value should always be ocsf_mapper. \nAllowed enum values: ocsf_mapper
default: ocsf_mapper
Option 8
\nobject
The add_env_vars processor adds environment variable values to log events.
id [required]
\nstring
The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
type [required]
\nenum
The processor type. The value should always be add_env_vars. \nAllowed enum values: add_env_vars
default: add_env_vars
variables [required]
\n[object]
A list of environment variable mappings to apply to log fields.
field [required]
\nstring
The target field in the log event.
name [required]
\nstring
The name of the environment variable to read.
Option 9
\nobject
The dedupe processor removes duplicate fields in log events.
fields [required]
\n[string]
A list of log field paths to check for duplicates.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
mode [required]
\nenum
The deduplication mode to apply to the fields. \nAllowed enum values: match,ignore
type [required]
\nenum
The processor type. The value should always be dedupe. \nAllowed enum values: dedupe
default: dedupe
Option 10
\nobject
The enrichment_table processor enriches logs using a static CSV file or GeoIP database.
file
\nobject
Defines a static enrichment table loaded from a CSV file.
encoding [required]
\nobject
File encoding format.
delimiter [required]
\nstring
The encoding delimiter.
includes_headers [required]
\nboolean
The encoding includes_headers.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileEncodingType object. \nAllowed enum values: csv
key [required]
\n[object]
Key fields used to look up enrichment values.
column [required]
\nstring
The items column.
comparison [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileKeyItemsComparison object. \nAllowed enum values: equals
field [required]
\nstring
The items field.
path [required]
\nstring
Path to the CSV file.
schema [required]
\n[object]
Schema defining column names and their types.
column [required]
\nstring
The items column.
type [required]
\nenum
The definition of ObservabilityPipelineEnrichmentTableFileSchemaItemsType object. \nAllowed enum values: string,boolean,integer,float,date,timestamp
geoip
\nobject
Uses a GeoIP database to enrich logs based on an IP field.
key_field [required]
\nstring
Path to the IP field in the log.
locale [required]
\nstring
Locale used to resolve geographical names.
path [required]
\nstring
Path to the GeoIP database file.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
target [required]
\nstring
Path where enrichment results should be stored in the log.
type [required]
\nenum
The processor type. The value should always be enrichment_table. \nAllowed enum values: enrichment_table
default: enrichment_table
Option 11
\nobject
The reduce processor aggregates and merges logs based on matching keys and merge strategies.
group_by [required]
\n[string]
A list of fields used to group log events for merging.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
merge_strategies [required]
\n[object]
List of merge strategies defining how values from grouped events should be combined.
path [required]
\nstring
The field path in the log event.
strategy [required]
\nenum
The merge strategy to apply. \nAllowed enum values: discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array,longest_array,flat_unique
type [required]
\nenum
The processor type. The value should always be reduce. \nAllowed enum values: reduce
default: reduce
Option 12
\nobject
The throttle processor limits the number of events that pass through over a given time window.
group_by
\n[string]
Optional list of fields used to group events before applying throttling.
id [required]
\nstring
The unique identifier for this processor.
include [required]
\nstring
A Datadog search query used to determine which logs this processor targets.
inputs [required]
\n[string]
A list of component IDs whose output is used as the input for this processor.
threshold [required]
\nint64
The number of events to allow before throttling is applied.
type [required]
\nenum
The processor type. The value should always be throttle. \nAllowed enum values: throttle
default: throttle
window [required]
\ndouble
The time window in seconds over which the threshold applies.
sources [required]
\n[ <oneOf>]
A list of configured data sources for the pipeline.
Option 1
\nobject
The kafka source ingests data from Apache Kafka topics.
group_id [required]
\nstring
Consumer group ID used by the Kafka client.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
librdkafka_options
\n[object]
Optional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]
\nstring
The name of the librdkafka configuration option to set.
value [required]
\nstring
The value assigned to the specified librdkafka configuration option.
sasl
\nobject
Specifies the SASL mechanism for authenticating with a Kafka cluster.
mechanism
\nenum
SASL mechanism used for Kafka authentication. \nAllowed enum values: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
topics [required]
\n[string]
A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]
\nenum
The source type. The value should always be kafka. \nAllowed enum values: kafka
default: kafka
Option 2
\nobject
The datadog_agent source collects logs from the Datadog Agent.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be datadog_agent. \nAllowed enum values: datadog_agent
default: datadog_agent
Option 3
\nobject
The amazon_data_firehose source ingests logs from AWS Data Firehose.
auth
\nobject
AWS authentication credentials used for accessing AWS services such as S3.\nIf omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_role
\nstring
The Amazon Resource Name (ARN) of the role to assume.
external_id
\nstring
A unique identifier for cross-account role assumption.
session_name
\nstring
A session identifier used for logging and tracing the assumed role session.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be amazon_data_firehose. \nAllowed enum values: amazon_data_firehose
default: amazon_data_firehose
Option 4
\nobject
The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
auth [required]
\nobject
GCP credentials used to authenticate with Google Cloud Storage.
credentials_file [required]
\nstring
Path to the GCP service account key file.
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
project [required]
\nstring
The GCP project ID that owns the Pub/Sub subscription.
subscription [required]
\nstring
The Pub/Sub subscription name from which messages are consumed.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be google_pubsub. \nAllowed enum values: google_pubsub
default: google_pubsub
Option 5
\nobject
The http_client source scrapes logs from HTTP endpoints at regular intervals.
auth_strategy
\nenum
Optional authentication strategy for HTTP requests. \nAllowed enum values: basic,bearer
decoding [required]
\nenum
The decoding format used to interpret incoming logs. \nAllowed enum values: bytes,gelf,json,syslog
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
scrape_interval_secs
\nint64
The interval (in seconds) between HTTP scrape requests.
scrape_timeout_secs
\nint64
The timeout (in seconds) for each scrape request.
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be http_client. \nAllowed enum values: http_client
default: http_client
Option 6
\nobject
The logstash source ingests logs from a Logstash forwarder.
id [required]
\nstring
The unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
tls
\nobject
Configuration for enabling TLS encryption.
ca_file
\nstring
Path to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]
\nstring
Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_file
\nstring
Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
type [required]
\nenum
The source type. The value should always be logstash. \nAllowed enum values: logstash
default: logstash
name [required]
\nstring
Name of the pipeline.
id [required]
\nstring
Unique identifier for the pipeline.
type [required]
\nstring
The resource type identifier. For pipeline resources, this should always be set to pipelines.
default: pipelines