Difference from SedonaDB and ecosystem state

[Ant59]

How does this crate differ from the https://github.com/apache/sedona-db project which already implements geospatial functions with DataFusion?

It would be wonderful to streamline efforts at enabling geospatial processing in the ecosystem. As far as I can tell, here's the remaining efforts needed:

• https://github.com/apache/arrow-rs needs to read native geometry and geography from Parquet (completed in [EPIC] [Parquet] Implement Geometry and Geography type support in Parquet apache/arrow-rs#8373)
• https://github.com/apache/iceberg-rust needs to be able to read and write native geometry and geography types in Iceberg v3 tables to catch up to the Java ecosystem (Support geo types apache/iceberg-rust#1884)
• https://github.com/apache/datafusion needs to be able to handle native geometry and geography types and geospatial functions (this repo and/or the aforementioned SedonaDB)
• https://github.com/apache/datafusion-comet needs to align to Spark 4.1's native handling of geometry and geography types, and ultimately provide native geospatial functions for distributed processing

I may have missed some other pieces that need to come together for a frictionless geospatial experience in the DataFusion ecosystem, but those are the outstanding work I know about.

There seems to be some efforts in the Sedona project to use DataFusion via SedonaDB to accelerate Spark queries, which seems a little hacky and backdoor-ish given the context of the wider DataFusion ecosystem and existence of Comet already: apache/sedona#2593

[paleolimbot]

How does this crate differ from the https://github.com/apache/sedona-db project which already implements geospatial functions with DataFusion?

GeoDataFusion came first! SedonaDB focused on the plumbing (e.g., optimizer rules, joins, statistics propagation) more than the components, although at this point we have most of the same functions. We also focused on making the WKB representation work everywhere (whereas GeoDataFusion has support for the native encodings that are potentially faster for some functions).

needs to be able to handle native geometry and geography types and geospatial functions

I think from DataFusion's perspective it already does this (in the sense that it allows arbitrary field metadata and arbitrary components to be plugged in). I would love true user-defined extension types in DataFusion so that I can upstream some of the pieces of SedonaDB (e.g., our table printer) but it has been difficult to find a committer interested.

needs to align to Spark 4.1's native handling of geometry and geography type

This would also be amazing, although I think it's unlikely to happen in the near future given the multitude of other things Comet has to keep up with. Personally I think that a more DataFusion-native distributed framework will give better results than trying to align with Spark (which did not do a very good job with the corner cases of its Geometry type in my opinion).

needs to be able to read and write native geometry and geography types in Iceberg v3

This would be amazing. It's on our TODO list to help with this but it's a long TODO list 🙂

There seems to be some efforts in the Sedona project to use DataFusion via SedonaDB to accelerate Spark queries

I think the general gist of this is more like...we're doing a lot of work in SedonaDB to add functions and file format support, so maybe we can find a way to call SedonaDB functions from Spark to avoid duplicating work when implementing new ones.

[kylebarron]

I don't have too much to add to Dewey's answer.

This project is more focused, ideally, on downstream projects embedding geodatafusion in their DataFusion distribution, whereas SedonaDB seems to be more focused on end-users than other DataFusion developers.

That said, I've had less time lately to spend on this project, so it's been progressing more slowly. I could review a PR if you had a specific feature you wanted to add.