feat(collectors): adding OpenTelemetryCollector #3499
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In this PR I am proposing a new feature to enable
dltcompatibility with OpenTelemetry through adding a new collector that just extends theLogCollectorand utilices existingdltcomponents to delineate traces, and to export metrics.Metrics are kept lean, communicating just total rows, duration, tables extracted and pipelines run. This is so that we don't explode the metrics sent, but also so that it is easier to aggregate across multiple pipeline deployments.
In the traces we enrich them with attributes corresponding to the rows added in tables during the normalized step, which can be parsed with thing such as Tempo, so build single pipeline dashboards.
Furthermore, this is also compatible with other OTeL collectors, which allow us to fully enrich traces with other libraries' telemetry. We have found this was specifically useful with databases and requests, were we can see the traces of requests and database interactions, by just installing the required telemetry libraries. See more at OTeL docs
Related Issues