Consolidating libraries

[gadomski]

For better or for worse, I've sort-of-accidentally made an alternate stac-geoparquet reader and writer in rustac. I've got a short section on the differences between the two libraries, but tl;dr: I think rustac has a little bit of a cleaner interface (bias: I wrote it), might be a bit faster, and could be used in javascript via WASM down the road.

It's becoming clear to me that we need to reduce ecosystem confusion (that I've created) around the two repositories. Options:

1. Keep stac-geoparquet and rustac more-or-less as is but clearly document why you might use stac-geoparquet or rustac
2. Keep this repo as is, add a "rustac driver", and point people here
3. Remove code from this repo, make it spec-only, and point people to rustac. We'd need to port or otherwise preserve the delta lake and pgstac portions of this repo

I think I'm leaning towards (2) (keeping this repo but incorporating rustac into it, w/ a bit of an API refactor in this repo) but I'd love to hear others' thoughts.

[TomAugspurger]

Thanks, I was wondering about this over the weekend too :)

FWIW, I think that multiple implementations of readers and writers for the stac-geoparquet spec is a good thing. But consolidating the Python implementations makes sense to me, and if folks feel like a python wrapper around rustac is the way to go then I'm supportive.

Of your options, I'm having trouble between 2 (rustac driver for stac-geoparquet) and 3 (deprecate the stac-geoparquet package in favor of rustac). If the issue is around long-term maintenance, both get us to approximately the same place. If we're mainly worried about user-confusion, how much does option 2 help? I guess it makes answering that question simple: stac-geoparquet is just a wrapper around rustac?

cc @kylebarron, who implemented the bulk of the pyarrow-based implementation.

If we go with 2, I'll probably want to do it slowly, with a deprecation warnings. I'd also want to deprecate / remove the initial geopandas-based implementation, but I guess that's independent (other than having three implementations floating around being plain silly).

[kylebarron]

We essentially have 3 implementations, if you include the original Python pandas implementation, in addition to the python Arrow implementation.

I think aside from the language, we need to have more discussion on the technical differences in implementation. E.g. in particular, how does schema resolution work in rustac?

[gadomski]

I think that multiple implementations of readers and writers for the stac-geoparquet spec is a good thing

Agreed.

If we're mainly worried about user-confusion, how much does option 2 help?

The phrase python -m pip install stac-geoparquet is pretty hard to beat in terms of UX. I think we give that up only if there's a really good reason, and I don't see one yet.

E.g. in particular, how does schema resolution work in rustac?

If I'm understanding what you mean by "schema resolution", right now it's full-scan: https://github.com/stac-utils/rustac/blob/c8f20c3202e4e8c37e81d7c47644a272719d863b/crates/core/src/geoarrow/mod.rs#L140. No reason why we can't make that more configurable, but that's time+effort.

more discussion on the technical differences in implementation

Without having done too much deep research, here's some notes on the options:

• stac-geoparquet is much more mature in terms of usage and history ... we discard those learned lessons at our peril. In particular, as @kylebarron points out, stac-geoparquet has many more knobs available for conversion, including providing a pre-built schema. There's no reason why we couldn't port these knobs to rustac, but that would take time.

• stac-geoparquet (as it currently exists) has more dependencies, including deltalake. rustac has one optional dependency, arro3. Here's the comparison of one-library-only virtual environments (uv pip install 'rustac[arrow]' and uv pip install stac-geoparquet in their respective venvs):

    $ du -h -d 0 rustac stac-geoparquet
    60M	rustac
    341M	stac-geoparquet
  

• stac-geoparquet has code specific to the Planetary Computer (Remove Planetary Comptuer specific code #16) that will take effort to remove.

• In previous testing, I've found rustac to ~30% faster for translations: https://www.gadom.ski/2024-09-FOSS4G-NA-stac-rs/#/4/3. It's probably worth re-running those tests.

• rustac could be extended to Javascript via WASM at some point in the future: Compile and run some functionality in WebAssembly rustac#218

To me, the tl;dr: is that stac-geoparquet (as it currnetly exists) is better for large scale, bulk loading scenarios, while rustac is more suited for small-to-medium scale (e.g. as a format translator). I still don't know what the correct path forward is, though...