Pyo3 bindings to tskit-rust

[currocam]

Hello,
Thanks a lot for putting this together! I’m finding it extremely useful for my PhD work. At some point, I wanted to pass Rust tables into a Python runtime, and I couldn’t figure out how, so I searched in the issues here looking for references.

I didn’t find any so I made (mostly for my future self) a minimal project for exposing rust-tskit into a Python interface. Perhaps someone else finds it useful. Sorry if I missed it and it’s already covered in the book (or there’s already a better reference! Please let me know).

https://github.com/currocam/tskit-rust-pyo3-starter

[molpopgen]

Thanks @currocam -- I have done tricks like this myself. While nothing involving pyo3/python will end up in this repo (the maintenance hassle is too high), there is room for a second repo that handles these abstractions. I think you can modify your code to be zero copy -- at the C level, a tree sequence can "borrow" a table collection rather than take ownership of it. (This is something that a standalone repo would definitely want to do!) Once you go down this road, though, you have the issue where the rust lifetimes need to be maintained via the Python side but the rust lifetime info has been lost due to working through the raw pointers!

[molpopgen]

If the current rust-side TableCollection had the following function, then zero-copy transfer becomes "trivial":

pub fn into_raw(self) -> *mut bindings::tsk_table_collection_t {
todo!()
}

Your "read pointer" function needs to return the *mut *mut pointer instead of dereferencing it. Then, it can be rebound to the pointer generated from rust. (You also need to free the pointer from non-rust tskit, etc.).

The challenge with this is that the safety requirements become many and nuanced. In addition to a bunch of proximal concerns, you also require that the tskit-python has been compiled with a tskit-c that is ABI compatible with that used for tskit-rust. There is no way to guarantee that other than manual inspection / having knowledge of the ecosystem. Fortunately, tables and tree sequences are very stable right now.

ABI stability includes precise layout stability!

These concerns are why I have always preferred dump/load to get data from rust to Python. Many of the same concerns are involved, but there is a lot less code to maintain!

[molpopgen]

See #830 for steps towards better support for this kind of thing.