variant_schema: Initial implementation

[sdf-jkl]

Which issue does this PR close?

• closes variant_schema: Initial implementation #5.

Rationale for this change

Tried to implement schema_of_variant and schema_of_variant_agg into a single udf - variant_schema

What changes are included in this PR?

Adding a new ScalarUdf variant_schema to extract aggregate schema from a scalar Variant or VariantArray

Are these changes tested?

• Tested scalar values with most types and arrays with conflicting and valid schemas.
• Added some sqllogictests

[sdf-jkl]

A quick run after submitting the PR

[sdf-jkl]

Due to Variant not being a first-class type in Arrow I had to use enums to represent extracted types.

[sdf-jkl]

Should probably remove unreachable!

[sdf-jkl]

@friendlymatthew I feel like this is such a mess.

I should split this into multiple PRs handling just the initial implementations of variant_schema and variant_schema_agg functions. Later add extras on top of them.

The order should be variant_schema -> variant_schema_agg -> type widening, etc.

[friendlymatthew]

@friendlymatthew I feel like this is such a mess.

I should split this into multiple PRs handling just the initial implementations of variant_schema and variant_schema_agg functions. Later add extras on top of them.

The order should be variant_schema -> variant_schema_agg -> type widening, etc.

Hi, can you explain why we need type widening?

[sdf-jkl]

From databricks docs for schema_of_variant_agg:

When two fields with the same name have a different type across records, Databricks uses the least common type. When no such type exists, the type is derived as a VARIANT. For example, INT and DOUBLE become DOUBLE, while TIMESTAMP and STRING become VARIANT.

We also want to do the same thing
for VariantList in the scalar version variant_schema.

[friendlymatthew]

From databricks docs for schema_of_variant_agg:

When two fields with the same name have a different type across records, Databricks uses the least common type. When no such type exists, the type is derived as a VARIANT. For example, INT and DOUBLE become DOUBLE, while TIMESTAMP and STRING become VARIANT.

We also want to do the same thing for VariantList in the scalar version variant_schema.

Makes sense to me. Let me know how you want to proceed. I'm fine with breaking this up into smaller PRs, pushing the scalar version first, then working on variant_agg and type widening later

[sdf-jkl]

I'll do that. Otherwise it's too much code to review.