feat: snowflake clustering key modifications
#3365
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | 5bb2365 | Commit Preview URL Branch Preview URL |
Nov 25 2025, 07:28 AM |
Collaborator
Author
|
Hey @rudolfix, could you review this PR? |
rudolfix
requested changes
Nov 23, 2025
Collaborator
rudolfix
left a comment
rudolfix
left a comment
There was a problem hiding this comment.
this is excellent! end to end test is missing though: where we actually run the queries on snowflake. you can add a pipeline testing this in test_snowflake_pipeline.py.
Collaborator
Author
|
@rudolfix Good call! I added a pipeline test and it exposed an issue with my implementation: it relied on Long story short, I switch to reading the So, could you review again? |
rudolfix
approved these changes
Nov 25, 2025
Collaborator
rudolfix
left a comment
rudolfix
left a comment
There was a problem hiding this comment.
LGTM! thanks for adding super clear end to end test!
sh-rp
added a commit
that referenced
this pull request
Nov 26, 2025
* override local marimo theme for dashboard (#3337) This ensures custom CSS is always readable. * Fix DocSearch v4 styles (#3338) * Fix DocSearch v4 styles * Fix search input styles for light and dark modes * docs: update weaviate destination docs and version (#3352) * Redshift feature: Include STS session token in COPY CREDENTIALS. (#3307) * Redshift feature: Include STS session token in COPY CREDENTIALS. If aws_session_token is present, append the session token. Keeps IAM_ROLE path and long-lieved keys unchanged --------- Co-authored-by: Tim Hable <[email protected]> * fixes sqlglot from find (#3357) * fixes athena refresh mode (#3313) * adds filter to exclude dropped tables in staging destination, implements for athena * enables refresh mode tests for athena, fixes tests * fixes staging_allowed_local_path on databricks, bumps databricks connector in lockfile * passes dropped tables schemas to filter, adjust athena filter * allows to disable lake formation * fix: backwards compatible traces (#3354) * makes trace backward compat with 1.17.0 and earlier * skips trace if any error in unpickle * always saves merged pipeline trace to have consistent pipeline.last_trace property * tests for past traces, broken traces and other improvements * (docs) adds community destinations (#3326) * adds community destinations * Apply suggestions from code review applies crate fixes Co-authored-by: Andreas Motl <[email protected]> --------- Co-authored-by: Andreas Motl <[email protected]> * fix: dashboard no longer crashes on broken home cell (#3348) * split home and workspace render methods * header row dry-er * catch-all errors in home()-cell * local try-catch for broken traces * e2e test for broken trace * removes this * shows navigation on pipeline attach error --------- Co-authored-by: Marcin Rudolf <[email protected]> * (fix) use sparse checkout for dlt init dlthub (#3356) * adds option to sparse checkout repo * use sparse checkout for llm context * fixes sqlglot from find * adds checkout after sparse clone * explains unknown path tests * Fix: The child table column remains in the schema as a partial column with seen-null-first=True (#3131) * child table column removed from parent * A utility functin that checks whether a column has seen-null-first set * Improved comments and docstrings, separate method in worker * null column not inferred if exists as compound * Column level x-normalizer cleaning moved outside of worker * Test for empty column becoming compound * Test clean_seen_null_first_hint * Uncalled source in pipeline.run( (#3369) * fix flaky dashboard tests (#3370) * improves dashboard multi schema test * closes and waits for sections in multi-schema test * removes command line snippet with generic text in exceptions * disables transformers pokeapi test * feat: `Schema.to_mermaid()` (#3364) * Add dlt.Schema.to_mermaid() method --------- Co-authored-by: jayant <[email protected]> * Refactor boundary timestamp handling in SqlMergeFollowupJob and SqlalchemyMergeFollowupJob to ensure current load package creation time is used when no boundary timestamp is provided. Update DltResourceHints class to streamline timestamp validation for active_record_timestamp and boundary_timestamp. Adjust tests accordingly. (#3378) * feat: `snowflake` clustering key modifications (#3365) * add support for snowflake clustering key modifications * add cluster column order test case * update snowflake cluster hint docs * switch to reading snowflake cluster hints from table schema * docs: lifecycle of `@dlt.hub.transformation` and `dlt.Relation` (#3329) * Lifecycle of a dlt transformation * Added test to match lifecycle docs * (fix) 3351 fixes default type var (#3373) * tests minimal typing extensions in alpine docker * keeps typevar default but does not use it in the code for backwart compat --------- Co-authored-by: djudjuu <[email protected]> Co-authored-by: Anton Burnashev <[email protected]> Co-authored-by: Violetta Mishechkina <[email protected]> Co-authored-by: Tim Hable <[email protected]> Co-authored-by: Tim Hable <[email protected]> Co-authored-by: rudolfix <[email protected]> Co-authored-by: Andreas Motl <[email protected]> Co-authored-by: anuunchin <[email protected]> Co-authored-by: Thierry Jean <[email protected]> Co-authored-by: jayant <[email protected]> Co-authored-by: Menna <[email protected]> Co-authored-by: Jorrit Sandbrink <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently, the
snowflakedestination only interpretsclusterhints on table creation. Any subsequent changes inclusterhints are ignored. This PR changes that, such thatclusterhints are also interpreted on table alterations.These three cases are supported:
(1) add clustering: without clustering -> with clustering
(2) change clustering: change set of cluster columns or column order
(3) drop clustering: with clustering -> without clustering
Note that there is no active hint change detection. Any changes in
clusterhints are only processed when the current table alteration mechanism is triggered, which is only when a new column is added.Related Issues