[SPARK-52451][CONNECT][SQL] Make WriteOperation in SparkConnectPlanner side effect free

[heyihong]

What changes were proposed in this pull request?

This PR refactors the Spark Connect execution flow to make WriteOperation handling side-effect free by separating the transformation and execution phases. The key changes include:

1. Unified execution flow: Consolidated ROOT and COMMAND operations through SparkConnectPlanExecution.handlePlan() instead of separate handlers
2. Pure transformation phase: Introduced transformCommand() that converts WriteOperation to LogicalPlan without side effects. It leverages the new DataFrameWriter methods (saveCommand(), saveAsTableCommand(), insertIntoCommand()), which return logical plans instead of executing immediately.
3. DataFrameWriter refactoring: The refactor adds new DataFrameWriter methods—saveCommand(), saveAsTableCommand(), and insertIntoCommand()—that return logical plans, and it introduces a new SaveAsV1TableCommand.

Why are the changes needed?

The current implementation has several issues:

1. Side effects in transformation: The handleWriteOperation method both transforms and executes write operations, making it difficult to reason about the transformation logic independently.

2. Code duplication: Separate handling paths for ROOT and COMMAND operations in ExecuteThreadRunner create unnecessary complexity and potential inconsistencies.

Does this PR introduce any user-facing change?

No. This is a purely internal refactoring that maintains the same external behavior and API. All existing Spark Connect client code will continue to work without any changes.

How was this patch tested?

build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite"

Was this patch authored or co-authored using generative AI tooling?

Cursor 1.3.5

[heyihong]

@HyukjinKwon @cloud-fan @hvanhovell

[cloud-fan]

Does this really work with runCommand?

[heyihong]

Yes, CompoundBody is actually executed during the analysis phase. But the tracker doesn't seem to get updated correctly. So I made a small fix in this pr

[cloud-fan]

I'm a bit worried about using CompoundBody outside of SQL script execution, and the fix in https://github.com/apache/spark/pull/51727/files#diff-a3fb0a56a7d2d08dc87434eff5b43aba0b006b59ed2f25e29bfb6fb4f81ec0c4R164 is a bit suspicious.

Can we create a new command SaveAsTableCommand to do these operations?

[cloud-fan]

The idea LGTM, we can simplify it futher in the future by using a simple logical plan for each DataFrameWriter API. Then Spark Connect can just use that logical plan, instead of calling DataFrameWriter to generate the logical plan.