1272 Support ClickHouse GCS S3 compatibility mode in filesystem destination #1423
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 9 commits
May 20, 2024 12:34
Signed-off-by: Marcel Coetzee <[email protected]>
This reverts commit cb80c6b.
Signed-off-by: Marcel Coetzee <[email protected]>
…use" This reverts commit f24eb1d.
…orted Signed-off-by: Marcel Coetzee <[email protected]>
…-destination-work-with-gcs-in-s3-compatibility-mode
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Pipboyguy
added
bug
added 3 commits
May 28, 2024 23:19
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
…gcs-in-s3-compatibility-mode
rudolfix
requested changes
May 29, 2024
Collaborator
rudolfix
left a comment
rudolfix
left a comment
There was a problem hiding this comment.
code: LGTM!
@Pipboyguy do we have the right credentials in our CI so this test is really connecting to clickhouse?
@sh-rp you should review this
sh-rp
reviewed
May 30, 2024
sh-rp
reviewed
May 30, 2024
Collaborator
Yeah I put those in yesterday. The question you were asking was wether it is really connecting to GCS, right? |
Signed-off-by: Marcel Coetzee <[email protected]>
Contributor
Author
@rudolfix We are indeed connecting to GCS with HMAC credentials and test passes |
rudolfix
deleted the
1272-make-filesystem-destination-work-with-gcs-in-s3-compatibility-mode
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Updates the filesystem destination to support using the S3 implementation with GCS buckets that have S3 interoperability mode (HMAC keys) enabled.
Key changes:
convert_storage_to_http_schemeutility function to properly handle GCS endpointsRelated Issues
GcpCredentials#1181