Skip to content

Add snowflake to application parameter to configuration #1266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rudolfix merged 13 commits into from
Apr 24, 2024
Merged

Add snowflake to application parameter to configuration #1266

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a10e72d
Add snowflake to application parameter to configuration
sultaniman Apr 23, 2024
0649112
Set default application parameter if it is not specified
sultaniman Apr 23, 2024
2b0d155
Adjust tests and for connection params
sultaniman Apr 23, 2024
af9d625
Use empty string to skip setting the application parameter
sultaniman Apr 23, 2024
ca39008
Set default value for application parameter
sultaniman Apr 23, 2024
5e89bd6
Fix if check bug
sultaniman Apr 23, 2024
b215754
Uppercase SNOWFLAKE_APPLICATION_ID and re-use in tests
sultaniman Apr 23, 2024
a035941
Add note in docs about application parameter for snowflake
sultaniman Apr 23, 2024
b426923
Update text for snowflake's application connection parameter
sultaniman Apr 24, 2024
683ffe5
Fix typo
sultaniman Apr 24, 2024
8a77a62
Update docs/website/docs/dlt-ecosystem/destinations/snowflake.md
sultaniman Apr 24, 2024
f27b026
Update snowflake.md
sultaniman Apr 24, 2024
874c08b
Update doc
VioletM Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions dlt/destinations/impl/snowflake/configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ def _read_private_key(private_key: str, password: Optional[str] = None) -> bytes
)


snowflake_application_id = "dltHub_dlt"
Copy link
Collaborator

@rudolfix rudolfix Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the application id we picked for dlt? @VioletM ?

Copy link
Collaborator

@sh-rp sh-rp Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make the var uppercase and import it into the test so we don't have any magic strings there

Copy link
Contributor Author

@sultaniman sultaniman Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.



@configspec(init=False)
class SnowflakeCredentials(ConnectionStringCredentials):
drivername: Final[str] = dataclasses.field(default="snowflake", init=False, repr=False, compare=False) # type: ignore[misc]
Expand All @@ -60,6 +63,7 @@ class SnowflakeCredentials(ConnectionStringCredentials):
authenticator: Optional[str] = None
private_key: Optional[TSecretStrValue] = None
private_key_passphrase: Optional[TSecretStrValue] = None
application: Optional[str] = None

__config_gen_annotations__: ClassVar[List[str]] = ["password", "warehouse", "role"]

Expand All @@ -85,6 +89,13 @@ def to_url(self) -> URL:
query["warehouse"] = self.warehouse
if self.role and "role" not in query:
query["role"] = self.role

if "application" not in query:
if self.application is None:
query["application"] = snowflake_application_id
elif self.application and self.application != "skip":
query["application"] = self.application

return URL.create(
self.drivername,
self.username,
Expand All @@ -109,8 +120,15 @@ def to_connector_params(self) -> Dict[str, Any]:
role=self.role,
private_key=private_key,
)

if self.authenticator:
conn_params["authenticator"] = self.authenticator

if self.application is None:
conn_params["application"] = snowflake_application_id
elif self.application and self.application != "skip":
conn_params["application"] = self.application

return conn_params


Expand Down
3 changes: 3 additions & 0 deletions tests/load/snowflake/test_snowflake_configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,14 @@ def test_snowflake_credentials_native_value(environment) -> None:
)
# set password via env
os.environ["CREDENTIALS__PASSWORD"] = "pass"
os.environ["CREDENTIALS__APPLICATION"] = "dlt"
c = resolve_configuration(
SnowflakeCredentials(),
explicit_value="snowflake://user1@host1/db1?warehouse=warehouse1&role=role1",
)
assert c.is_resolved()
assert c.password == "pass"
assert "application=dlt" in str(c.to_url())
# # but if password is specified - it is final
c = resolve_configuration(
SnowflakeCredentials(),
Expand All @@ -126,6 +128,7 @@ def test_snowflake_credentials_native_value(environment) -> None:
)
assert c.is_resolved()
assert c.private_key == "pk"
assert "application=dlt" in str(c.to_url())


def test_snowflake_configuration() -> None:
Expand Down