-
Notifications
You must be signed in to change notification settings - Fork 418
Accept :memory: mode for credentials parameter in duckdb factory #1297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
8f51b49 to
02fb63c
Compare
rudolfix
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK it looks like we have problem with documentation, not the code. We clearly say how to pass in memory duckdb to be used:
The destination accepts a duckdb connection instance via credentials, so you can also open a database connection yourself and pass it to dlt to use. :memory: databases are supported.
import duckdb
db = duckdb.connect()
p = dlt.pipeline(
pipeline_name='chess',
destination=dlt.destinations.duckdb(db),
dataset_name='chess_data',
full_refresh=False,
)but then say the following:
A few special connection strings are supported:
- :pipeline: creates the database in the working directory of the pipeline with the name
quack.duckdb. - :memory: creates an in-memory database. This may be useful for testing.
what we should IMO do
- drop :memory: form docs
- instead of accepting :memory: let's raise an exception that says what to do instead
if we create the duckdb instance in memory, it will be lost after the data loading, right? even if we somehow avoid closing it, passing this as external db is IMO way more intuitive
24c829d to
e391de4
Compare
sh-rp
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one small suggested change, but it's good!
and one request: please don't mark comments as resolved, the commenter should do this once they are satisfied that you have made the necesary change
e2ecb2d to
affbea3
Compare
rudolfix
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code looks good but documentation was only partially updated and IMO not reviewed before push.
thing that helps me is to read my diff in the PR before asking for a review. because now we need another round and the ticket is delayed again...
2886073 to
e1449e3
Compare
Co-authored-by: VioletM <[email protected]>
6f956b8 to
d7a7909
Compare
rudolfix
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
This PR addresses #1216 and allows
duckdb(:memory:)requestTODO