Skip to content

Conversation

@gruuya
Copy link
Contributor

@gruuya gruuya commented Nov 8, 2022

Hi folks 👋🏻

We're using connector-x to build a remote table querying capability in Seafowl (an analytical, cache friendly, time travel-able database we're building on top of Datafusion).

Given that on one hand we rely on tokio async runtime, while on the other hand some of the DB drivers block the thread when executing queries, we're forced to run the queries in a closure using spawn_blocking.

In turn, instead of creating a single SourceConn upon instantiating the remote table, we're forced to instantiate a new SourceConn for each query execution, since otherwise we can't pass the ref to the get_arrow function inside the closure due to lifetimes incompatibilities (borrowed data escapes outside of associated function).

Long story short, to mitigate this we'd really only need SourceConn to be cloneable (unless this is for some reason a bad idea?), as then we could get away with keeping a single one in remote table struct and clone it or it's fields when needed, hence this PR.

Additionally, I bumped Arrow and Datafusion to the latest available versions (26 and 14 respectively).

@alswang18
Copy link
Contributor

@wangxiaoying I think this should be fine. We are only making a string container clonable as far as I can tell for this PR. Would love to know your thoughts.

@wangxiaoying
Copy link
Contributor

wangxiaoying commented Nov 9, 2022

Hi @gruuya , thanks for the PR! To pass the CI, I made a minor change to align arrow version in rust with cpp and python crates. I'm going to close this PR and redirect to #399 (since I do not have access to the upstream).

@gruuya
Copy link
Contributor Author

gruuya commented Nov 9, 2022

Great, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants