Expressions becomes enum, ComputeFn becomes ScalarFn

[gatesn]

This issue encapsulates a few changes, but I believe the end state makes sense!

• We should redefine ComputeFn to mean ScalarFn. This let's us make some nice assumptions and keep the trait tightly defined.
• Add an AggFn to cover our current aggregation functions, e.g. sum, count, min, max.
• Make a Vortex Expression and enum with a ScalarFn variant.
• Any expression that can be implemented as a ScalarFn should be. Note that functions that require short-circuiting, e.g. AND, OR, should have their own entry in the enum, rather than delegate to the scalar function.
• While ScalarFn::Options doesn't need to be serializable (we should support invoking compute functions with opaque runtime options data), a ScalarFn wrapped in an Expression does need to be serializable. This is because...
• ...Expression is serializable as a well-defined protobuf union. This is one way we can create expressions over the C API.
• FilterFn and TakeFn should be added to the ArrayOperations trait (returns Result<Option<Array>> since they're optionally implemented) since they're not scalar functions.
• Vortex Session holds a registry of compute functions that are used when deserializing the protobuf to load the actual implementation.

[joseph-isaacs]

I think the scalar compute fn changes will reduce the amount of code we have to write.

I have a few qs/comments.

What is the relation between agg and scalar exprs, can one contain the other?

Do you think window functions/exprs will follow?

Also there will likely be an overhead to this implementation technique we cannot monomophorize specific scalar function impl since it goes via dynamic dispatch. Do you think we would keep common compute fns (e.g. compare) as is?

[gatesn]

What is the relation between agg and scalar exprs, can one contain the other?

There's no AggExpr for now, just an AggFn. So its input is an actual Array.

In the future, we may have one. In which case, AggExpr could contain Expr (implied ScalarExpr): SUM(a + 1)

But, ScalarExpr cannot contain AggExpr as this would involve a shuffle / non-linear compute and should be the job of a compute engine: SUM(a) < b

Do you think window functions/exprs will follow?

No, again windows tend to be in the realm of compute engines. Complexities such as very large (larger than memory), or overlapping windows, means I would (for now) class this outside the realm of linear compute.

Slightly related, figure I should write this down, we could expand AggFn to add a reduce evaluator (in addition to just evaluate(Array) -> Scalar).

I think the signature of such a reduction fn should be: reduce(input: Array, groups: Mask) -> Array

Where the output array has length == Mask::true_count().

Each true bit in the groups mask signifies the start of a reduction group.

This doesn't cover window functions, because the groups are non-overlapping. But it does allow more complex linear aggregations.

Interestingly, the filter function just becomes a reducer over the "first" function.

Also there will likely be an overhead to this implementation technique we cannot monomophorize specific scalar function impl since it goes via dynamic dispatch. Do you think we would keep common compute fns (e.g. compare) as is?

Could you expand? Our compute functions already go via dynamic dispatch with regards to compute kernels.