Skip to content

prevent accidental wrapping of sources in resources when using adapters #1645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sh-rp merged 8 commits into from
Jul 30, 2024
Merged

prevent accidental wrapping of sources in resources when using adapters #1645

Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f86c90d
prevent accidental wrapping of sources in resources when using adapters
sh-rp Jul 29, 2024
bc3030a
fix typo
sh-rp Jul 29, 2024
fd45350
fix qdrant zendesk example
sh-rp Jul 29, 2024
1a91659
another fix to the qdrant example
sh-rp Jul 29, 2024
1d6a18f
rename utils function
sh-rp Jul 30, 2024
4b00b4b
add logger warning when setting default name for resource
sh-rp Jul 30, 2024
8f37964
only use selected resources in get_resource_for_adapter
sh-rp Jul 30, 2024
bccbbba
switch to value error
sh-rp Jul 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion dlt/destinations/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from typing import Any, cast, Tuple, Dict, Type

from dlt.destinations.exceptions import DatabaseTransientException
from dlt.extract import DltResource, resource as make_resource
from dlt.extract import DltResource, resource as make_resource, DltSource

RE_DATA_TYPE = re.compile(r"([A-Z]+)\((\d+)(?:,\s?(\d+))?\)")

Expand All @@ -23,6 +23,12 @@ def ensure_resource(data: Any) -> DltResource:
"""Wraps `data` in a DltResource if it's not a DltResource already."""
if isinstance(data, DltResource):
return data
# prevent accidentally wrapping sources with adapters
if isinstance(data, DltSource):
Copy link
Collaborator

@rudolfix rudolfix Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you check if source selects just one resource and then adapt it? look here: #1644 I think the user's thinking is OK

raise Exception(
"You are trying to use an adapter on a dlt source. You can only use adapters on pure"
Copy link
Contributor

@VioletM VioletM Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this function only used with adapters? The error message seems to be out of context for ensure_resource function. Also, TypeError maybe?

Copy link
Collaborator Author

@sh-rp sh-rp Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it only is used in adapters, I checked that.

" data or dlt resources."
)
resource_name = None if hasattr(data, "__name__") else "content"
Copy link
Contributor

@VioletM VioletM Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this else close should throw a warning. Otherwise, you get some unrecognizable table in the destination with the name Content out of nowhere 😱

return cast(DltResource, make_resource(data, name=resource_name))

Expand Down