[DOCS-10653] Obs Pipelines OCSF Custom Configuration #28892
+427
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Preview links (active after the
|
maycmlee
commented
Apr 22, 2025
| |------------|-----------|-----------------------------------------------------------------------------------------------------------------------| | ||
| | `version` | Yes | Must be set to `1` to indicate the mapping descriptor format version. | | ||
| | `metadata` | Yes | Contains a set of hard-coded description fields about the event class. See [Metadata section](#metadata-section) for details. | | ||
| | `preprocess` | No | Lists an ordered series of preprocessing steps. The preprocessors rework the data to allow the field-to-field mappings. Each entry is an object consisting of a `function` name (required) and parameters associated with that function. See [Preprocessors](#preprocessors) for more information. | |
There was a problem hiding this comment.
What does "rework" mean here? And how does it allow field-to-field mappings?
There was a problem hiding this comment.
These are operations that may be necessary to transform incoming raw data from original sources into an object suitable for the mapping steps below. "Reformat" might be a better descriptor.
content/en/observability_pipelines/processors/remap_ocsf/custom_configuration.md
Outdated
Show resolved
Hide resolved
content/en/observability_pipelines/processors/remap_ocsf/custom_configuration.md
Outdated
Show resolved
Hide resolved
|
|
||
| The `parse_csv` preprocessor: | ||
|
|
||
| 1. Extracts a `source` field. |
There was a problem hiding this comment.
An example source field that gets extracted could be helpful.
content/en/observability_pipelines/processors/remap_ocsf/custom_configuration.md
Outdated
Show resolved
Hide resolved
| All enumerated name or label fields identified in the OCSF schema are converted to their sibling `id` field. For example, the string field `severity` is automatically converted to the numeric field `severity_id` based on the standard enum value table defined in the OCSF schema. If no matching value is found in the lookup table, the `id` field is set to `99` to represent `Other`. | ||
|
|
||
| If one of the listed `profiles` in the metadata section is `datetime`, the mapping | ||
| automatically has all numeric timestamps identified in the OCSF schema converted into the sibling field `{DEST}_dt`. For example, the numeric `time` field is converted into `time_dt`, which contains a string representation of that timestamp. No additional work is required to support the `datetime` profile. |
There was a problem hiding this comment.
Is there a link to the OCSF schema we can add here?
There was a problem hiding this comment.
OCSF is composed of multiple schemas, the choice of which is specified by the version, class, and profiles given in the metadata section. The complete schema can be browsed here: https://schema.ocsf.io/
content/en/observability_pipelines/processors/remap_ocsf/custom_configuration.md
Outdated
Show resolved
Hide resolved
content/en/observability_pipelines/processors/remap_ocsf/custom_configuration.md
Outdated
Show resolved
Hide resolved
|
|
||
| ### Mapping Functions | ||
|
|
||
| A function applies an operation to the value extracted from the source, before assigning to the destination. |
There was a problem hiding this comment.
Before assigning the extracted value to the destination? Is that correct?
content/en/observability_pipelines/processors/remap_ocsf/custom_configuration.md
Outdated
Show resolved
Hide resolved
github-actions
bot
added
Images
github-actions
bot
removed
Images
maycmlee
removed request for
a team,
TovRudyy,
RomainGuarinoni and
vitor-de-araujo
maycmlee
commented
Apr 24, 2025
| 1. Parses `source` as CSV and maps the values to the `columns` listed. | ||
| 1. Inserts the mapped data in the `dest` field. | ||
|
|
||
| Columns with a `null` value are dropped. One of the columns can have a wildcard (`*`), in which case it is assigned a string containing all the text from the fields that remain after those before and after have been mapped. |
There was a problem hiding this comment.
Need clarification on what this means: "assigned a string containing all the text from the fields that remain after those before and after have been mapped."
There was a problem hiding this comment.
@bruceg could you clarify that part of the sentence?
There was a problem hiding this comment.
See the example posted earlier. Effectively, the output column is assigned everything that is left after handling the remainder of the fields after the wildcard.
drichards-87
requested changes
Apr 24, 2025
| | `version` | Yes | Must be set to `1` to indicate the mapping descriptor format version. | | ||
| | `metadata` | Yes | Contains a set of hard-coded description fields about the event class. See [Metadata section](#metadata-section) for details. | | ||
| | `preprocess` | No | An ordered list of preprocessing steps. The preprocessors reformat raw data from the sources so that the data can be converted to OCSF format based on `mapping`. Each preprocessor entry consists of a `function` and parameters associated with that function. See [Preprocessors](#preprocess-section) for more information. | | ||
| | `mapping` | Yes | An ordered list of field-to-field assignments, where a `source` field is assigned to a `dest` field in the output OCSF event. See for more information. Each [mapping](#mapping-section) may have a `conversion` specified by a [lookup table](#mapping-lookup-tables) or post-processing [mapping function](#mapping-functions). | |
There was a problem hiding this comment.
I think there might be a missing link in the mapping description. (See for more information.)
|
|
||
| ### Preprocess section | ||
|
|
||
| The `preprocess` section lists the preprocessors that reformats the data to allow field-to-field mappings. Each entry in this section consists of a `function` and the parameters associated with that function. |
There was a problem hiding this comment.
Suggested change
| The `preprocess` section lists the preprocessors that reformats the data to allow field-to-field mappings. Each entry in this section consists of a `function` and the parameters associated with that function. | |
| The `preprocess` section lists the preprocessors that reformat the data to enable field-to-field mappings. Each entry in this section consists of a `function` and the parameters associated with that function. |
|
|
||
| ### Mapping section | ||
|
|
||
| The `mapping` section is an ordered list of field-to-field assignments. Each mapping entry consists of a `dest` path that refers to the destination field in the OCSF event and either a `source` path that refers to a field in the source event or a `value` that contains a literal constant to insert at that destination. |
There was a problem hiding this comment.
Suggested change
| The `mapping` section is an ordered list of field-to-field assignments. Each mapping entry consists of a `dest` path that refers to the destination field in the OCSF event and either a `source` path that refers to a field in the source event or a `value` that contains a literal constant to insert at that destination. | |
| The `mapping` section is an ordered list of field-to-field assignments. Each mapping entry consists of a `dest` path that refers to the destination field in the OCSF event, and either a `source` path that refers to a field in the source event, or a `value` that contains a literal constant to insert at that destination. |
|
|
||
| #### Implicit mappings | ||
|
|
||
| All enumerated name or label fields identified in the OCSF schema are converted to their sibling `id` field. For example, the string field `severity` is automatically converted to the numeric OSCF field `severity_id` based on the values defined in the OCSF schema. See `severity_id` in [Authentication [3002]][1] for the OCSF values. If no matching value is found in the lookup table, the `id` field is set to `99` to represent `Other`. |
There was a problem hiding this comment.
Suggested change
| All enumerated name or label fields identified in the OCSF schema are converted to their sibling `id` field. For example, the string field `severity` is automatically converted to the numeric OSCF field `severity_id` based on the values defined in the OCSF schema. See `severity_id` in [Authentication [3002]][1] for the OCSF values. If no matching value is found in the lookup table, the `id` field is set to `99` to represent `Other`. | |
| All enumerated name or label fields identified in the OCSF schema are converted to their sibling `id` field. For example, the string field `severity` is automatically converted to the numeric OSCF field `severity_id` based on the values defined in the OCSF schema. See `severity_id` in [Authentication][1] for the OCSF values. If no matching value is found in the lookup table, the `id` field is set to `99` to represent `Other`. |
|
|
||
| ### `reshape_array` | ||
|
|
||
| The `reshape_array` function extracts data from a source array to create a new array of values. The function filters only array elements containing a field that matches a condition from the list in [Mapping lookup tables](#mapping-lookup-tables), and extracts another field into the output array. |
There was a problem hiding this comment.
Suggested change
| The `reshape_array` function extracts data from a source array to create a new array of values. The function filters only array elements containing a field that matches a condition from the list in [Mapping lookup tables](#mapping-lookup-tables), and extracts another field into the output array. | |
| The `reshape_array` function extracts data from a source array to create a new array of values. The function filters array elements, selecting only those that contain a field matching a condition from the [Mapping lookup tables](#mapping-lookup-tables), and extracts another field into the output array. |
There was a problem hiding this comment.
This might be because I'm not very familiar with OCSF or Observability Pipelines, but I'm not sure I understand what 'another field' refers to in the phrase 'and extracts another field into the output array.'"
There was a problem hiding this comment.
That is a reference to a field within elements of the array. i.e. if the array contains [{"a":1},{"a":2}] then you could indicate you want to extract field a and it would result in [1, 2].
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Merge instructions
Merge readiness:
For Datadog employees:
Merge queue is enabled in this repo. Your branch name MUST follow the
<name>/<description>convention and include the forward slash (/). Without this format, your pull request will not pass in CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.If your branch doesn't follow this format, rename it or create a new branch and PR.
To have your PR automatically merged after it receives the required reviews, add the following PR comment:
Additional notes