Skip to content

[Data] - Evaluate Expressions sequentially and upsert columns to block#56193

Merged
alexeykudinkin merged 1 commit intoray-project:masterfrom
goutamvenkat-anyscale:goutam/eval_expr_update_block
Sep 3, 2025
Merged

[Data] - Evaluate Expressions sequentially and upsert columns to block#56193
alexeykudinkin merged 1 commit intoray-project:masterfrom
goutamvenkat-anyscale:goutam/eval_expr_update_block

Conversation

@goutamvenkat-anyscale
Copy link
Contributor

@goutamvenkat-anyscale goutamvenkat-anyscale commented Sep 3, 2025

Why are these changes needed?

Previously, the columns were updated after all expressions were evaluated, but this required additional handling in the planner and optimization to handle dependency chaining between columns.

In this implementation, each expression will be evaluated and the column will be upserted to the block within the provided Project operator.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@goutamvenkat-anyscale
[Data] - Evaluate Expressions sequentially and upsert columns to block
5a94535 
Signed-off-by: Goutam V <goutam@anyscale.com>
@goutamvenkat-anyscale goutamvenkat-anyscale requested a review from a team as a code owner September 3, 2025 07:29
gemini-code-assist[bot]
gemini-code-assist bot reviewed Sep 3, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new upsert_column method to the BlockAccessor interface and its implementations for Arrow and Pandas blocks. This is then utilized in plan_project_op to evaluate expressions and add or update columns sequentially, which simplifies the planner logic by removing the need for explicit dependency chaining. The implementation is sound, but I have one suggestion to improve the code's conciseness and address a potential performance consideration in the main logic change.

@ray-gardener ray-gardener bot added the data Ray Data-related issues label Sep 3, 2025
@goutamvenkat-anyscale goutamvenkat-anyscale added the go add ONLY when ready to merge, run all tests label Sep 3, 2025
@alexeykudinkin alexeykudinkin merged commit 0bda55a into ray-project:master Sep 3, 2025
5 checks passed
@goutamvenkat-anyscale goutamvenkat-anyscale deleted the goutam/eval_expr_update_block branch September 3, 2025 20:45
sampan-s-nayak pushed a commit to sampan-s-nayak/ray that referenced this pull request Sep 8, 2025
@goutamvenkat-anyscale
[Data] - Evaluate Expressions sequentially and upsert columns to block (
d047950 
ray-project#56193)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Previously, the columns were updated after all expressions were
evaluated, but this required additional handling in the planner and
optimization to handle dependency chaining between columns.

In this implementation, each expression will be evaluated and the column
will be upserted to the block within the provided Project operator.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Goutam V <goutam@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
jugalshah291 pushed a commit to jugalshah291/ray_fork that referenced this pull request Sep 11, 2025
@goutamvenkat-anyscale @jugalshah291
[Data] - Evaluate Expressions sequentially and upsert columns to block (
93902a5 
ray-project#56193)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Previously, the columns were updated after all expressions were
evaluated, but this required additional handling in the planner and
optimization to handle dependency chaining between columns.

In this implementation, each expression will be evaluated and the column
will be upserted to the block within the provided Project operator.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Goutam V <goutam@anyscale.com>
Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>
wyhong3103 pushed a commit to wyhong3103/ray that referenced this pull request Sep 12, 2025
@goutamvenkat-anyscale
[Data] - Evaluate Expressions sequentially and upsert columns to block (
343e304 
ray-project#56193)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Previously, the columns were updated after all expressions were
evaluated, but this required additional handling in the planner and
optimization to handle dependency chaining between columns.

In this implementation, each expression will be evaluated and the column
will be upserted to the block within the provided Project operator.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Goutam V <goutam@anyscale.com>
Signed-off-by: yenhong.wong <yenhong.wong@grabtaxi.com>
dstrodtman pushed a commit to dstrodtman/ray that referenced this pull request Oct 6, 2025
@goutamvenkat-anyscale @dstrodtman
[Data] - Evaluate Expressions sequentially and upsert columns to block (
46f60f5 
ray-project#56193)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Previously, the columns were updated after all expressions were
evaluated, but this required additional handling in the planner and
optimization to handle dependency chaining between columns.

In this implementation, each expression will be evaluated and the column
will be upserted to the block within the provided Project operator.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Goutam V <goutam@anyscale.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
landscapepainter pushed a commit to landscapepainter/ray that referenced this pull request Nov 17, 2025
@goutamvenkat-anyscale @landscapepainter
[Data] - Evaluate Expressions sequentially and upsert columns to block (
e453b12 
ray-project#56193)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Previously, the columns were updated after all expressions were
evaluated, but this required additional handling in the planner and
optimization to handle dependency chaining between columns.

In this implementation, each expression will be evaluated and the column
will be upserted to the block within the provided Project operator.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Goutam V <goutam@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexeykudinkin alexeykudinkin alexeykudinkin approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@goutamvenkat-anyscale @alexeykudinkin