Feat: Custom metrics in the incremental transform #3117
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs canceled.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | 7e6d35a | Commit Preview URL Branch Preview URL |
Oct 02 2025, 08:04 AM |
anuunchin
force-pushed
the
feat/3072-incremental-custom-metrics
branch
2 times, most recently
from
a5ba867 to
c55ce51
Compare
anuunchin
commented
Sep 23, 2025
anuunchin
force-pushed
the
feat/3072-incremental-custom-metrics
branch
5 times, most recently
from
080c70d to
3c12fe9
Compare
anuunchin
commented
Sep 25, 2025
anuunchin
commented
Sep 25, 2025
tests/extract/test_incremental.py
Outdated
|
|
||
| # 6. run with two new items as a single batch | ||
| load_id = _run_with_items([{"id": 6, "value": "6.1"}, {"id": 6, "value": "6.2"}], True) | ||
| _assert_custom_metrics(load_id, 5, 2, 3, 0, 2) |
Contributor
Author
There was a problem hiding this comment.
@rudolfix my intuition here was to get 0 as the initial_unique_hashes_count, but since it's retrieved from the state it makes sense. However, if the boundary deduplication is off and unique hashes becomes 0, shouldn't this also reset the hash count in state ? 👀
Collaborator
There was a problem hiding this comment.
yes, it should see my comment above
rudolfix
requested changes
Sep 30, 2025
Collaborator
rudolfix
left a comment
rudolfix
left a comment
There was a problem hiding this comment.
I changed the TypeVar for the metrics:
- added default type so we are backward compatible if any user derives from
ItemTransform - added type bound to
Mapping
pls take a look at review comments. also two last test cases are suspicious or I do not understand input arguments
tests/extract/test_incremental.py
Outdated
|
|
||
| # 6. run with two new items as a single batch | ||
| load_id = _run_with_items([{"id": 6, "value": "6.1"}, {"id": 6, "value": "6.2"}], True) | ||
| _assert_custom_metrics(load_id, 5, 2, 3, 0, 2) |
Collaborator
There was a problem hiding this comment.
yes, it should see my comment above
anuunchin
force-pushed
the
feat/3072-incremental-custom-metrics
branch
from
128009c to
7e6d35a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the following three custom metrics to the incremental transform:
unfiltered_items_countunfiltered_batches_countunique_hashes_countSpecifically, in the
calldunder of theIncrementalclass.An appropriate test was added.