[DISCUSSION] 2025 Q1-Q2 Roadmap

[alamb]

We have had past roadmap discussions:

• [DISCUSSION] 2024 Q4 / 2025 Q1 Roadmap #13274

Let's use this ticket to coordinate / discuss our priorities for the next few months

[alamb]

I personally plan to focus on

• 🚀 performance: Help complete the advanced parquet predicate pushdown with @XiangpengHao Enable parquet filter pushdown by default #3463
• 🔨 Improve build time: Build time regression #14256 (thanks to @logan-keede who is helping break)
• 📖 : trying to keep the documentation high quality [Epic] Better / Improved Documentation, Tutorials and Examples #7013 (thanks @@ugoa for the recent help ❤ )
• 📈 : keep doing pre-release testing [DISCUSS] More extensive pre-release testing #13661
• Easier query remote data systems [DISCUSSION] Make it easier and faster to query remote files (S3, iceberg, etc)  #13456
• Helping Comet and other systems aiming to provide spark compatible runtimes: [DISCUSSION] Add separate crate to cover spark builtin functions #5600. I think this is a common and recurring use case

Stuff I am personally unlikely to spend much time on (aka looking for help)

• Adding new functions to the core (I understand the value, but they can be added outside and just adding more functions to the core adds maintenance burden on maintainers). So I am hesitant to accept new functions without additional new maintenance bandwidth
• Revamping coercion (as much as it needs it). I think it is in good hands with @jayzhan211 and @shehabgamin, eg. Signature::Coercible with user defined implicit casting #14440

[Omega359]

My personal list

• Find/fix cause of with_column extremely poor performance
• Add SessionConfig to ScalarFunctionArgs. This will lead to allowing timestamp functions to use DF timezone (various tickets)
• union_by_name (Implement UNION ALL BY NAME #14508, Add some DataFrame method(s) to combine two inputs where the schema can be different #12650) Already being done in feat: Implement UNION ALL BY NAME #14538 !
• add manual trigger for extended tests in pull requests #14331

[comphead]

I'll try to chase

• Datafusion binary size has been getting bigger #13816
• [EPIC] Redesign DataFusion main page #14389

[alamb]

I filed a ticket to discuss the next few months:

• [DISCUSSION] DataFusion Road Map: Q3-Q4 2025 #15878