Can we use `df.select_columns_by_name` to subselect columns coming from `__dataframe__` protocol

[jcrist]

With #3114 (and some recent work in ibis) we can now pass ibis tables directly to altair.Chart and things just work.

import ibis
import altair as alt

t = ibis.examples.penguins.fetch()

chart = (
    alt.Chart(t, width=600)
    .mark_circle(size=50)
    .encode(
        x=alt.X("bill_length_mm").scale(zero=False),
        y=alt.Y("bill_depth_mm").scale(zero=False),
        color="species"
    )
    .interactive()
)

However, currently it appears the entire table is loaded into memory, even if only a few columns are needed for the plot. For example, the above penguins dataset has 8 columns, but only 3 of them are used to generate the plot. Can altair make use of this information to subselect columns before conversion when using the __dataframe__ protocol?

With some recent work in ibis, the following can all happen without loading data into memory:

df = t.__dataframe__()

# subselect columns
df = df.select_columns_by_name(["bill_length_mm", "bill_depth_mm", "species"])

# view dtypes, as needed for altair's type inference
df.get_column_by_name("bill_length_mm").dtype

# convert to pyarrow here. Only this step will actually execute the query
t = pa.interchange.from_dataframe(df)

Especially for wide input tables, having altair handle subselecting columns automatically may be useful for improving performance.

[binste]

Makes a lot of sense! The issue with subsetting columns is that it's very difficult to evaluate which columns are used as almost any chart property can refer to columns using Vega Expression References. See #2428 and #2586. This is why this feature was never implemented in Altair for any kind of dataframe.

However, @jonmmease managed to implement this functionality in VegaFusion where the Vega-Lite spec (generated by Altair) is compiled to Vega and transforms and references evaluated. I'd assume the column subsetting currently happens after the conversion to an arrow table so it would probably require some changes in VegaFusion to do it on the Ibis table itself. I'm sure @jonmmease can explain this better.

I really like Ibis and a tighter integration with Altair and VegaFusion is something I'm very excited about. Also the possibility of pushing calculations into databases through Ibis. @jonmmease @jcrist Happy to support in making this happen!

[jonmmease]

Thanks for recounting the history on that @binste

Yeah, we've basically decided to not try to implement column subsetting in Altair core, and leave that to VegaFusion.

In the case of the Ibis+VegaFusion integration we're talking about over in vega/vegafusion#355, what this would look like is that the select method would be called with the minimum set of columns required as part of building up the query, and so by the time collect is called, the columns will be trimmed down to only those that are used in the chart's transforms.

[jcrist]

Thanks for the replies y'all, that makes sense. Closing.