Skip to content

Conversation

@EricFecteau
Copy link
Collaborator

Arrow 0.53 (current used by connectorX) does not play well with any crates needing Arrow 0.55 (the latest version of Arrow). For example, datafusion has updated to Arrow 0.55 and is no longer working with Arrow 0.53. For example, a crate that imports the latest version of connectorx and of datafusion can not compile. See issue here: apache/arrow-rs#7410

[dependencies]
connectorx = { version = "0.4", features = ["src_postgres", "dst_arrow"] }
datafusion = "47"

Arrow 0.54 seems to play well with 0.55 and 0.53, so that is why I bumped it up to this version only. I fear bumping it up to 0.55 will cause issues with other crate features I can not test (e.g. older version of datafusion).

@EricFecteau
Copy link
Collaborator Author

Had to bump datafusion also because same error as #771 (but opposite direction). Bumped it 0.46, the one that has the arrow 54 version. All arrows in this crate now match and should be maximally compatible with other crates for now.

@EricFecteau
Copy link
Collaborator Author

Also confirmed that this branch solves the issue of compatibility with other crates:

[dependencies]
connectorx = { git = "https://github.com/sfu-db/connector-x.git", branch = "update_arrow", features = ["src_postgres", "dst_arrow"] }
datafusion = "47"

@EricFecteau EricFecteau requested a review from wangxiaoying May 6, 2025 10:23
@wangxiaoying wangxiaoying merged commit 5f71ed0 into main May 6, 2025
2 checks passed
@wangxiaoying
Copy link
Contributor

Thanks for the PR @EricFecteau , it looks good to me.

@EricFecteau EricFecteau deleted the update_arrow branch May 6, 2025 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants