feat: add anchor for non-ordinal plan reference

[drin]

This adds an "anchor" to PlanRel that can be referenced from a ReferenceRel. This anchor and reference relationship provides a non-ordinal method for identifying and accessing a "subtree" or sub-graph. To maintain consistency with type_anchor and function_anchor, the new subtree_anchor attribute is uint32.

This PR leaves in the original subtree_ordinal attribute since it seems a (mildly) more performant method for referencing a subtree, but also since it is still relevant in the typical case. The new anchor attribute is an improvement for cases where multiple plans are merged and at least one already contains a ReferenceRel.

It is expected that only one of subtree_ordinal or subtree_reference will be used, however I don't see a good reason to enforce the use of only one, so I did not group the attributes in a oneof constraint.

If either of the above points ((1) keeping subtree_ordinal and (2) not enforcing a oneof constraint) are changed, then this PR would likely become a breaking change.

Also, this PR addresses #725 , but uses the name subtree_reference and subtree_anchor (instead of plan_anchor) to be consistent with the pre-existing subtree_ordinal attribute. I don't know that relations in a substrait plan are strictly trees, so I think this could be a good time to change "subtree" to "planrel" or "subgraph" or something similar.

BREAKING CHANGE: The field subtree_ordinal is made optional with explicit presence due to being moved into a new oneof field, "ref_type".

[CLAassistant]

All committers have signed the CLA.

[tokoko]

This PR leaves in the original subtree_ordinal attribute since it seems a (mildly) more performant method for referencing a subtree

This is technically true, but on the other hand we are already doing the same thing with function extensions when the number of functions used in a typical plan most likely will be a lot bigger than the number of subtrees. I don't think it makes sense to keep the existing behavior only due to performance considerations.

It is expected that only one of subtree_ordinal or subtree_reference will be used, however I don't see a good reason to enforce the use of only one, so I did not group the attributes in a oneof constraint.

I think leaving both options especially w/o oneof will lead to a lot more confusion and possible divergence in implementations. My choice would be to deprecate subtree_ordinal and eventually get rid of it, but oneof is also a viable option.

If either of the above points ((1) keeping subtree_ordinal and (2) not enforcing a oneof constraint) are changed, then this PR would likely become a breaking change.

Pretty sure putting an existing field in a oneof is a backwards-compatible change.

[drin]

I should have time to finish this up in the next few days

[jacques-n]

let's put these in on a one of (subtree_reference and subtree_ordinal) and deprecate subtree ordinal.

[drin]

I named the oneof ref_type since that seemed descriptive enough without being too verbose.

[vbarua]

@jacques-n is there a reason to both deprecate the existing field AND put both of the deprecaetd and new field in a oneof. It feels noisy to create a oneof if the eventual plan is to remove one of the fields, and hence the need for the oneof.

[vbarua]

Talking through this in the Substrait sync with @jacques-n, we agreed that keeping this out of a oneof make sense.

For migration purposes, its actually useful to allow both to be set for a while. Effectively, we can add support for this in producers and consumers in a decoupled fashion, and once everything supports both, remove the old field.

As part of this, we would need to document that the new relation anchor should be preferred when both are set.

[drin]

rebased on main

[drin]

hm, according to a stackoverflow post, moving an optional field into a oneof is not breaking. This PR essentially makes subtree_ordinal an optional field instead of an implicit field by moving it into a oneof, but that seems backwards-compatible. However, the check for breaking changes still flags it so I decided to add a BREAKING CHANGES: footer to the PR description.

[EpsilonPrime]

By making the anchor on the plan object does that mean you can no longer use a ReferenceRel to point to a subtree of the existing plan? For instance, TPC-H 15 processes the same subtree twice and often ends up with different results (when using float). By using a reference you could process the subtree once and point to it avoiding differences.

[drin]

By making the anchor on the plan object...

I think you may be misunderstanding where the anchor is? I don't think of the PlanRel as the Plan object

...does that mean you can no longer use a ReferenceRel to point to a subtree of the existing plan?

The change is essentially to change a relative ID (ordinal) to an absolute ID (anchor). I think in either case you need to know what you're pointing to and you can point to a subtree of the existing plan.

[vbarua]

Let me see if I understand the intent of this change.

Function / Type References

For things like functions, we define Plan-level anchors

substrait/proto/substrait/plan.proto

Lines 34 to 35 in d430e52

	// a list of extensions this plan may depend on
	repeated substrait.extensions.SimpleExtensionDeclaration extensions = 2;

substrait/proto/substrait/extensions/extensions.proto

Lines 57 to 59 in d430e52

	message ExtensionFunction {
	// references the extension_uri_anchor defined for a specific extension URI.
	uint32 extension_uri_reference = 1;

which are then used within the plan

substrait/proto/substrait/algebra.proto

Lines 1178 to 1183 in d430e52

	// A scalar function call.
	message ScalarFunction {
	// Points to a function_anchor defined in this plan, which must refer
	// to a scalar function in the associated YAML file. Required; avoid
	// using anchor/reference zero.
	uint32 function_reference = 1;

This mechanism is also used for types.

Rel References

For RelReference we don't currently use an anchor, but instead an ordinal reference

substrait/proto/substrait/algebra.proto

Lines 1756 to 1760 in d430e52

	// This rel is used to create references,
	// in case we refer to a RelRoot field names will be ignored
	message ReferenceRel {
	int32 subtree_ordinal = 1;
	}

into an list of relations in the Plan

substrait/proto/substrait/plan.proto

Lines 37 to 38 in d430e52

	// one or more relation trees that are associated with this plan.
	repeated PlanRel relations = 3;

Your Proposal

Bring references to relations into line with functions and types by attaching an anchor to the PlanRel message.

Notes

Bringing this inline with how we do references generally make sense to me. That being said:

The new anchor attribute is an improvement for cases where multiple plans are merged and at least one already contains a ReferenceRel.

How does this make merging plans easier. I assume that as with ordinal-based references you would still have to rewrite anchors when they collide between plans?

It is expected that only one of subtree_ordinal or subtree_reference will be used, however I don't see a good reason to enforce the use of only one

If we're going to do this, I think it would be better to have only 1 way to do it and we should plan to get rid of the ordinal-based resolution.

[vbarua]

If we're going to do this, we should deprecate the old field, with the eventual plan of removing subtree_ordinal entirely.

And if we're going to do that, I don't know how much sense it makes to put them both in a oneof. With the way you've worded the anchor on the PlanRel, 0 isn't a valid rel anchor, so while these field coexist, if you have a ReferenceRel with subtree_reference set to 0 you know you have to use the subtree_ordinal.

[vbarua]

I brought this PR up in the Substrait sync this week. I've updated some comments with what we talked about.

[vbarua]

Talking through this in the Substrait sync with @jacques-n, we agreed that keeping this out of a oneof make sense.

For migration purposes, its actually useful to allow both to be set for a while. Effectively, we can add support for this in producers and consumers in a decoupled fashion, and once everything supports both, remove the old field.

As part of this, we would need to document that the new relation anchor should be preferred when both are set.

[tokoko]

@vbarua I'm not opposed to planrel_reference, but one slight caveat regarding naming: this field will most likely act as a reference to the list of relations in ExtendedExpression message as well (see #794). In my PR I decided to introduce new ExtendedExpressionRel message to hold Rels. If we go that route, calling the field planrel_reference doesn't make as much sense anymore. Alternatively we can reuse PlanRel message for ExtendedExpression relations, which is doable but slightly awkward, because relations in PlanRel for extended expressions will only ever be of type Rel, never RelRoot.

[drin]

I think more context about the interaction with ExtendedExpressionRel would be helpful.

I commented on #794 (moved my comment from here).

[vbarua]

@tokoko left a comment on your proposal in #794 with regards to including relations in ExtendedExpressions.

[drin]

I accommodated feedback from @vbarua and rebased on main

[drin]

@tokoko @vbarua as long as planrel_reference is a new field in ReferenceRel anyways, what if we actually nest it in a oneof, that way we can be explicit about other things the reference may point to (e.g. extendedexpr_reference) in a way that is a bit future proof (although a bit more verbose). Or, we can change it to be anchor_reference and have an enum that specifies what the expected type of message being pointed to is?

[vbarua]

This change looks reasonable to me as is.

We still need reviews from other PMCs for this: @jacques-n @westonpace @EpsilonPrime @cpcloud

When we merge this, we should make sure the commit message is accurate. This is no longer a breaking change.