Should OpenPIV use pandas.DataFrame or xarray.DataArray or Dataset internally ?

[alexlib]

We use rather simple numpy arrays for x,y,u,v,s2n,flags, along the processing, and at the end we write these as columns in the ASCII files with some optional headers.
In the multiprocessing we also have several iterations and then we have temporary arrays and then refined arrays, filtered and flagged arrays, interpolated arrays, etc.

I think it makes more sense to use some modern structures, e.g. dataclass or pandas.DataFrame, xarray.DataArray and similar ones that will open several options for us:

• easier data maintenance, e.g. temporal storage,
• adding new columns with headers, keeping units, experimental metadata as attributes to those arrays
• saving to a large set of file formats while keeping the structure of reading/writing intact - managed by the pandas or xarray or the libraries their use
• allow better memory management by adding, replacing columns, renaming only units, etc.

This can become a part of the OpenPIV v1.0 milestone

[ErichZimmer]

I agree with using pandas or xarray for a modern data structure. These come with a nice side effect of storing the structures in HDF4/5 and NetCDF.

[ErichZimmer]

@alexlib I plan on refactoring OpenPIV to use xarray. The structure would be similar to that of the c++ version where the folder tree would be something along the lines of this:

/openpiv 
  __init__.py
  /core
    __init__.py
    preprocess.py
    process.py
    tools.py
    windef.py
    ...
  /piv_engine <--- contains classes wrapped around xarray 
    __init__.py
   piv.py
   vector.py
  /legacy <--- legacy interface for compatibility reasons 

The classes in the PIV engine submodule would hold the raw results from PIV along with a class that would perform the usual recursive PIV algorithms. I am thinking of making the parameters for PIV a class too since we could implement a function to read and write YAML files. This should allow for a user to use the c++, python, or GPU version of OpenPIV once this feature is completed. Note, this is a proof of concept for now as I am doing paper sketches until I get access to a computer. I would be implementing a similar solution for the c++ version of OpenPIV once I complete the port for the meson build system. I have no plans for the Matlab version of OpenPIV.

[alexlib]

Great plan
please take a look also at ffpiv

[ErichZimmer]

Here is a quick diagram I drew together. It will take a long time to finish this wireframe along with a data flow diagram, but I think it would be worth it since it will get everyone on page of what I am tyring to accomplish (a unified interface that can be implemented in c++17 or python 3.8+).

[ErichZimmer]

I would also like to note that using xarray would make the TomoPIV module significantly easier to manage since such a large volume of data is created in a relatively short time frame.

[alexlib]

let's do it.
i am a bit afraid of additional complexity. we have to keep the code human readable and easy to install and pick up

[ErichZimmer]

Here is something I threw together relatively quickly. I did not do any benchmarking yet, however. You can go ahead and play around with this prototype to see how you like it. For the rest of the original sketch-up I made, there were significant changes made to simplify most of the complicated stuff. I reduced everything down to just PIVParams and a PIVEngine function.

https://github.com/ErichZimmer/openpiv_prototypes/blob/master/xarray/VectorField.ipynb

[alexlib]

Here is something I threw together relatively quickly. I did not do any benchmarking yet, however. You can go ahead and play around with this prototype to see how you like it. For the rest of the original sketch-up I made, there were significant changes made to simplify most of the complicated stuff. I reduced everything down to just PIVParams and a PIVEngine function.

https://github.com/ErichZimmer/openpiv_prototypes/blob/master/xarray/VectorField.ipynb

Looks interesting. It probably makes sense to also combine it with https://github.com/alexlib/pivpy - I'll be happy to use the same notation, dimensions, attributes so we can fluently create also post-processing routines.

However, I'm not sure how this helps openpiv-python itself. I agree that keeping stuff in RAM and then storing it in multiple ASCII files and then use pivpy.io to read those into xarray dataset. However, the integration with openpiv-python should also consider where it can save some time on computations. Probably if we already install xarray we can think also to work with some on-disk memory format (e.g. HDF5 through PyTables) or similar - zarr.

The future of openpiv-python is bright, but we need also to get into the optical flow methods, see the openpiv-users forum, and openpiv-python-gpu and your great work on C++ - the idea is to have a single type of data output and fluent change of GPU to CPU or multi-window interrogation with C++

[ErichZimmer]

How should we handle PTV data? When using xarray or h5py, the data must be on a regular grid for all vector fields, if I remember correctly. That would mean that PTV data would be on an extremely dense coordinate system spanning the width and height of the measured volume. Of course, we could implement an advanced divergence-free PTV grid interpolating scheme to counteract this issue, but I think this is something we should keep in mind. Note: datasets only hold one copy of the coordinate system instead of one for each time instance.

Another issue that may occur is holding time-resolved 2D and 3D vector data. If we use float64 (double precision/4 bytes per element), the amount of data that could be held in memory somewhat limited. For instance, my laptop with 2GB of available RAM would only be able to roughly hold 350 vector fields of 112 x 112 x 56 vectors before my computer starts to freak out. Reading from disk through h5py would allow for something along the lines of 180,000 vector fields (~1TB) on my laptop, which is much better. As such, I think a mix of xarray and H5Py would be nice.

On a sidenote, I noticed that writing to a plaintext file is quite slow, as mentioned above. Writing and reading from a data structure on hard disk should be incomparably faster. For example, if our windef algorithm outputted a VectorField instance, we could save the whole thing to disk in something around the lines of 5ms for the whole dataset where a text file would take 0.1ms for each file. I would simply have to work on chunking the PIV computations to fit on RAM so we can save each chunk to the HDF5 file. NumPy also has a solution via .npy and .nps files which are compressed to save disk space. I have not benchmarked these file formats, though.

[alexlib]

Hi @ErichZimmer - I do not think that it has to be this way. Multiple projects use xarray for trajectories, e.g. https://github.com/alexlib/lt_toolbox?tab=readme-ov-file#getting-started
https://traja.readthedocs.io/en/latest/reading.html
https://opendrift.github.io/trajan/user_guide/data-layout.html

this is the convention
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#trajectory-data

this is a nice example
https://stackoverflow.com/questions/69757359/xarray-groupby-with-ragged-array

[ErichZimmer]

@alexlib I thought about this problem for a while today (lost power due to bad weather and some tornadoes), and I think it is best that we create two data structures for vectors; one for gridded data and one for trajectories. This would allow us to exploit the nature of both vector fields for analysis purposes. For instance, vector trajectories could be fitted onto splines and evaluated onto a common grid using the FlowFit3 algorithm. I believe this is what most commercial PIV software providers do and for a good reason. Implementing the vector trajectories based on the links you provided seems relatively straightforward and opens a path towards advanced volumetric measurement techniques (something I always wanted to try and play with).

[alexlib]

Let's meet online and discuss. I feel less comfortable to create two types of structures for each experiment, cause we might need only one for the analysis, while wasting space and time of calculating both. I agree we need to prepare the structures of both and think mainly on how one could transfer one into another efficiently and what are the use cases. I remember many features being implemented only once for a single use case - these could be efficient but not generalizable (see our existing code space for instance with some ideas than not materialized).
The foundations should be good and thoughtful. I agree we need to stand on the shoulders of existing database packages and ideas like pandas, xarray, zarr, dash, etc.
Let's see what CFD people are doing in such cases. We could also think of what is our outcome - visualization? graphs? animations? heavy lifting post processing? storage?

[ErichZimmer]

Yes, let's do a zoom meeting. I'm available on Tuesday for a meeting. I'll also take a closer look at how OpenFOAM and other CFD software manage their data. On the outcome side of things, I'm not sure about the graphics. I use paraview for data visualization, volume rendering, and animations. This is probably overkill in most situations, though. We should definitely focus on post processing and data storage techniques too. Many users are not using powerful workstations to do their analysis (look at publications using PIVlab). We should try to optimize for both consumer computers and high-performance clusters. Perhaps, even FPGA accelerations for those who don't have a GPU (FPGAs are a lot faster than our c++ software). Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Alex Liberzon ***@***.***> Sent: Saturday, April 5, 2025 11:04:07 PM To: OpenPIV/openpiv-python ***@***.***> Cc: Erich Zimmer ***@***.***>; Mention ***@***.***> Subject: Re: [OpenPIV/openpiv-python] Should OpenPIV use pandas.DataFrame or xarray.DataArray or Dataset internally ? (Issue #274) Let's meet online and discuss. I feel less comfortable to create two types of structures for each experiment, cause we might need only one for the analysis, while wasting space and time of calculating both. I agree we need to prepare the structures of both and think mainly on how one could transfer one into another efficiently and what are the use cases. I remember many features being implemented only once for a single use case - these could be efficient but not generalizable (see our existing code space for instance with some ideas than not materialized). The foundations should be good and thoughtful. I agree we need to stand on the shoulders of existing database packages and ideas like pandas, xarray, zarr, dash, etc. Let's see what CFD people are doing in such cases. We could also think of what is our outcome - visualization? graphs? animations? heavy lifting post processing? storage? — Reply to this email directly, view it on GitHub<#274 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQSCNN7HIWBLSE7XUPSQ2HL2YCRTPAVCNFSM6AAAAABVQXI4D2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBRGIYDOMZWGI>. You are receiving this because you were mentioned.Message ID: ***@***.***> [alexlib]alexlib left a comment (OpenPIV/openpiv-python#274)<#274 (comment)> Let's meet online and discuss. I feel less comfortable to create two types of structures for each experiment, cause we might need only one for the analysis, while wasting space and time of calculating both. I agree we need to prepare the structures of both and think mainly on how one could transfer one into another efficiently and what are the use cases. I remember many features being implemented only once for a single use case - these could be efficient but not generalizable (see our existing code space for instance with some ideas than not materialized). The foundations should be good and thoughtful. I agree we need to stand on the shoulders of existing database packages and ideas like pandas, xarray, zarr, dash, etc. Let's see what CFD people are doing in such cases. We could also think of what is our outcome - visualization? graphs? animations? heavy lifting post processing? storage? — Reply to this email directly, view it on GitHub<#274 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQSCNN7HIWBLSE7XUPSQ2HL2YCRTPAVCNFSM6AAAAABVQXI4D2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBRGIYDOMZWGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>