Support for sparse dataframes

[jamestwebber]

GitHub Issues for Apache Arrow

I opened a Pandas issue but they closed it and referred me here. I was hoping to use parquet files as a way to share pandas.SparseDataFrame objects, but it appears that the current to_parquet method fails with columns of different lengths:

import pandas as pd # v0.22.0
import scipy.sparse # v1.0.1

rpd = pd.SparseDataFrame(scipy.sparse.random(1000, 1000), 
                         columns=list(map(str, range(1000))),
                         default_fill_value=0.0)
rpd.to_parquet('rpd.pq')

---------------------------------------------------------------------------
ArrowIOError                              Traceback (most recent call last)
<ipython-input-65-1aeaae9e36a0> in <module>()
      4                          columns=list(map(str, range(1000))),
      5                          default_fill_value=0.0)
----> 6 rpd.to_parquet('rpd.pq')

...

ArrowIOError: Column 8 had 4 while previous column had 8

Poking around, Pandas is just passing things straight into pyarrow, so I guess there's no support for sparse matrices at the moment? This seems like it'd be a nice use-case because the columns can be heavily compressed, but the current implementation needs to a dense version, which necessitates a round-trip via giant memory consumption.

Are there plans to improve this support, or is this not a good use for the format?

[wesm]

The best way would be to convert the DataFrame to dense then write. It's possible with some effort we could blow up one column at a time, but since pandas is considering deprecating or splitting pandas.sparse into a separate package I'm not sure how much we'll want to do. patches welcome of course. I created https://issues.apache.org/jira/browse/ARROW-2818 as one follow up to improve UX

[cornhundred]

I'm getting a different error using updated versions of pandas and scipy

import pandas as pd # v0.24.2
import scipy.sparse # v1.1.0

rpd = pd.SparseDataFrame(scipy.sparse.random(1000, 1000), 
                         columns=list(map(str, range(1000))),
                         default_fill_value=0.0)
rpd.to_parquet('rpd.pq', engine='pyarrow')

ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column 0 with type Sparse[float64, 0.0]')

[wesm]

Can you report to pandas?

[PGijsbers]

Pandas now supports sparse data through sparse data structures, I see no indication that they are still going to separate this into a distinct package. Would you re-evaluate this feature?

[choucavalier]

Supporting this feature! Would save a lot of storage if we could leverage sparse columns (some columns being really sparse (many NaNs) and some other columns being dense.