Skip to content

Commit 8fe5cee

Browse files
spkane31bergundy
andauthored
improvement: remove waits before fetching activities (#8144)
## What changed? optimize the batch operation processing in `BatchActivity` and `BatchActivityWithProtobuf` by removing the need to wait for entire pages to complete before fetching the next page. - Implemented proactive page fetching once a worker becomes available - common `processWorkflowsWithProactiveFetching` function to reduce code duplication ## Why? The previous implementation had workers wait for entire pages to complete. This optimization improves resource utilization. The refactoring also eliminates duplicated functions in the `BatchParams` struct and `BatchOperation` protobuf. Addresses issue #8098. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) The changes maintain backward compatibility. ## Potential risks While this change improves performance, it does modify the concurrency model of batch processing: 1. **Timing changes**: The optimization changes when pages are fetched relative to task completion, which could expose edge cases in error handling or heartbeat timing 2. **Memory usage**: Pages may be fetched earlier, potentially increasing peak memory usage if the next page is large 3. **Rate limiting interaction**: The more aggressive task scheduling could interact differently with rate limiting, though the same per-worker limits are maintained 4. **Heartbeat behavior**: heartbeats track the progress of an entire page and are applied after an entire page finishes The changes preserve all existing error handling, retry logic, and rate limiting behavior, but the different execution timing could surface previously hidden race conditions. --------- Co-authored-by: Roey Berman <[email protected]>
1 parent 469526e commit 8fe5cee

File tree

2 files changed

+307
-239
lines changed

2 files changed

+307
-239
lines changed

0 commit comments

Comments
 (0)