[Data] Fixing `ReorderingBundleQueue` handling of empty output sequences by alexeykudinkin · Pull Request #60470 · ray-project/ray

alexeykudinkin · 2026-01-24T03:58:42Z

Description

This PR revisits ReorderingBundleQueue to move pointer advancements from get_next_inner and finalize into has_next method to guarantee that the queue will not get stuck with any operations sequence.

Currently, ReorderingBundleQueue could still get stuck in case of the sequence captured in test_ordered_queue_getting_stuck.

The queue is guaranteed to traverse through all bundles so long as all keys are finalized (ie tasks finished).

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

gemini-code-assist

Code Review

This pull request aims to fix an issue where _OrderedOutputQueue could get stuck by refactoring its state transition logic. My review identified a critical issue in the new implementation of _OrderedOutputQueue.has_next(). The updated logic can cause a crash by returning True when no output is available, which violates the has_next/get_next contract. I've provided a code suggestion to fix this bug while preserving the intended fix for the 'stuck' issue.

python/ray/data/_internal/execution/operators/map_operator.py

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

python/ray/data/_internal/execution/bundle_queue/reordering.py

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

python/ray/data/_internal/execution/bundle_queue/reordering.py

Tidying up Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

iamjustinhsu

nice!!

iamjustinhsu · 2026-01-28T00:27:03Z

python/ray/data/_internal/execution/bundle_queue/reordering.py

+
+    @override
+    def peek_next(self) -> Optional[RefBundle]:
+        return (


do we also need to rotate the pointer here too?

Good catch!

python/ray/data/tests/unit/test_reordering_bundle_queue.py

iamjustinhsu · 2026-01-28T00:29:06Z

python/ray/data/tests/unit/test_reordering_bundle_queue.py

+    #   - `has_next` would return False, (_inner[_current_key] is empty)
+    #   - `get_next` will never be invoked (b/c `has_next` returns false)
+    #   - `finalize(key=1)` has already been invoked, no pointer advancement will happen


are these comments and the above needed? I think they might be too implementation-detailed specific

This is a test so extra verbosity is fine

Wanted to provide ample comment to explain how it would get stuck

python/ray/data/_internal/execution/bundle_queue/reordering.py

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

python/ray/data/_internal/execution/bundle_queue/reordering.py

iamjustinhsu · 2026-01-28T01:17:22Z

python/ray/data/tests/unit/test_reordering_bundle_queue.py

+        assert queue.peek_next() is bundle2
+        assert queue.get_next() is bundle2


why not always go into this case? (since target_op=="peek" also covers the "get" case)

B/c we want to check that get also does advance

…ces (ray-project#60470) ## Description This PR revisits `ReorderingBundleQueue` to move pointer advancements from `get_next_inner` and `finalize` into `has_next` method to guarantee that the queue will not get stuck with any operations sequence. Currently, `ReorderingBundleQueue` could still get stuck in case of the sequence captured in `test_ordered_queue_getting_stuck`. The queue is guaranteed to traverse through all bundles so long as all keys are finalized (ie tasks finished). ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: jinbum-kim <jinbum9958@gmail.com>

…ces (ray-project#60470) ## Description This PR revisits `ReorderingBundleQueue` to move pointer advancements from `get_next_inner` and `finalize` into `has_next` method to guarantee that the queue will not get stuck with any operations sequence. Currently, `ReorderingBundleQueue` could still get stuck in case of the sequence captured in `test_ordered_queue_getting_stuck`. The queue is guaranteed to traverse through all bundles so long as all keys are finalized (ie tasks finished). ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

…ces (ray-project#60470) ## Description This PR revisits `ReorderingBundleQueue` to move pointer advancements from `get_next_inner` and `finalize` into `has_next` method to guarantee that the queue will not get stuck with any operations sequence. Currently, `ReorderingBundleQueue` could still get stuck in case of the sequence captured in `test_ordered_queue_getting_stuck`. The queue is guaranteed to traverse through all bundles so long as all keys are finalized (ie tasks finished). ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: 400Ping <jiekaichang@apache.org>

…ces (ray-project#60470) ## Description This PR revisits `ReorderingBundleQueue` to move pointer advancements from `get_next_inner` and `finalize` into `has_next` method to guarantee that the queue will not get stuck with any operations sequence. Currently, `ReorderingBundleQueue` could still get stuck in case of the sequence captured in `test_ordered_queue_getting_stuck`. The queue is guaranteed to traverse through all bundles so long as all keys are finalized (ie tasks finished). ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: Adel Nour <ans9868@nyu.edu>

…ces (ray-project#60470) ## Description This PR revisits `ReorderingBundleQueue` to move pointer advancements from `get_next_inner` and `finalize` into `has_next` method to guarantee that the queue will not get stuck with any operations sequence. Currently, `ReorderingBundleQueue` could still get stuck in case of the sequence captured in `test_ordered_queue_getting_stuck`. The queue is guaranteed to traverse through all bundles so long as all keys are finalized (ie tasks finished). ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

alexeykudinkin requested a review from a team as a code owner January 24, 2026 03:58

gemini-code-assist bot reviewed Jan 24, 2026

View reviewed changes

python/ray/data/_internal/execution/operators/map_operator.py Outdated Show resolved Hide resolved

cursor bot reviewed Jan 24, 2026

View reviewed changes

python/ray/data/_internal/execution/operators/map_operator.py Outdated Show resolved Hide resolved

python/ray/data/_internal/execution/operators/map_operator.py Outdated Show resolved Hide resolved

ray-gardener bot added the data Ray Data-related issues label Jan 24, 2026

alexeykudinkin added 2 commits January 27, 2026 13:00

TODO

e5be943

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Fixed ReorderingBundleQueue to make sure it doesn't get stuck

b6024a8

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin force-pushed the ak/reord-que-fix branch from de2db59 to b945852 Compare January 27, 2026 22:59

alexeykudinkin changed the title ~~[Data] Fixed _OrderedOutputQueue to avoid getting stuck~~ [Data] Fixed ReorderingBundleQueue to avoid getting stuck Jan 27, 2026

cursor bot reviewed Jan 27, 2026

View reviewed changes

python/ray/data/_internal/execution/bundle_queue/reordering.py Outdated Show resolved Hide resolved

python/ray/data/_internal/execution/bundle_queue/reordering.py Outdated Show resolved Hide resolved

alexeykudinkin changed the title ~~[Data] Fixed ReorderingBundleQueue to avoid getting stuck~~ [Data] Streamlining ReorderingBundleQueue Jan 27, 2026

alexeykudinkin added 5 commits January 27, 2026 15:20

Updated tests

f4c6cc8

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Revisited ReorderingBundleQueue to implement correct semantic

2b19b36

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Added has_next assertion

296375a

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Updated test

1bab447

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Tidying up

28731ed

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin force-pushed the ak/reord-que-fix branch from 783825c to 28731ed Compare January 27, 2026 23:21

cursor bot reviewed Jan 27, 2026

View reviewed changes

python/ray/data/_internal/execution/bundle_queue/reordering.py Outdated Show resolved Hide resolved

alexeykudinkin added 3 commits January 27, 2026 15:34

Fixed _move_to_next_key;

a726d06

Tidying up Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Tidying up

9a44f89

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Make has_next loop

4e876a3

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin changed the title ~~[Data] Streamlining ReorderingBundleQueue~~ [Data] Fixing ReorderingBundleQueue handling of empty output sequences Jan 28, 2026

Added correctness tests

c557574

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

iamjustinhsu reviewed Jan 28, 2026

View reviewed changes

cursor bot reviewed Jan 28, 2026

View reviewed changes

python/ray/data/_internal/execution/bundle_queue/reordering.py Outdated Show resolved Hide resolved

alexeykudinkin added 3 commits January 27, 2026 17:02

Fixed peek_next

f8e689b

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Updated tests

a26dd27

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

lint

bbfda5f

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin enabled auto-merge (squash) January 28, 2026 01:04

github-actions bot added the go add ONLY when ready to merge, run all tests label Jan 28, 2026

cursor bot reviewed Jan 28, 2026

View reviewed changes

python/ray/data/_internal/execution/bundle_queue/reordering.py Show resolved Hide resolved

iamjustinhsu approved these changes Jan 28, 2026

View reviewed changes

alexeykudinkin merged commit 0c18b72 into master Jan 28, 2026
7 of 8 checks passed

alexeykudinkin deleted the ak/reord-que-fix branch January 28, 2026 01:52

		assert queue.peek_next() is bundle2
		assert queue.get_next() is bundle2

Conversation

alexeykudinkin commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iamjustinhsu left a comment

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iamjustinhsu Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iamjustinhsu Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alexeykudinkin commented Jan 24, 2026 •

edited

Loading