Skip to content

Improve slice eval#446

Merged
lshaw8317 merged 27 commits intomainfrom
improve_slice_eval
Aug 8, 2025
Merged

Improve slice eval#446
lshaw8317 merged 27 commits intomainfrom
improve_slice_eval

Conversation

@lshaw8317
Copy link
Collaborator

@lshaw8317 lshaw8317 commented Aug 4, 2025

Necessary bug fixes for integer slicing for reductions and refactoring slices_eval as suggested in #436.
Had to add a couple of modifications to ensure contiguity of numpy arrays in some special cases where an evaluation (and hence copy, to contiguous memory) did not occur (when not using numexpr and when the expression was simply "o0"). Consequently also had to improve the rebasing of operands which the lazyexpr machinery carries out (prior to this PR, the rebasing would not occur when the lazyexpr was just a blosc2 array, i.e. "a" would not be rebased to "o0").
Before merging have to do a benchmark comparing new slicing with non-unit steps for lazy expressions (which only uses numpy at chunk-level) with the old format (which filled the whole span of the slice before taking the steps at the very end). Consequently, the old version might load a very large result into memory, even if the slice was small - consider an array of shape (10**9, ) with slice [::10**8]; the old Blosc2 would fill an array of shape (9.9**9, ) and then slice to give a result of shape (10,). The new version would never have more than the chunksize in memory.

@lshaw8317
Copy link
Collaborator Author

pre-commit.ci autofix

@lshaw8317
Copy link
Collaborator Author

lshaw8317 commented Aug 8, 2025

The proposed change in Blosc2 3.6.2 leads to reduced memory consumption, especially when the slice is small relative to the array size (i.e. the step is very large), relative to the previous implementation in Blosc2 3.6.1. See the benchmark in bench/ndarray/slice-expr-step.py

sliceexpr_LARGE_Blosc3_6_1 sliceexpr_LARGE_Blosc3_6_2_dev0 sliceexpr_SMALL_Blosc3_6_1 sliceexpr_SMALL_Blosc3_6_2_dev0

@lshaw8317 lshaw8317 marked this pull request as ready for review August 8, 2025 09:36
@lshaw8317 lshaw8317 merged commit db28770 into main Aug 8, 2025
13 checks passed
@lshaw8317 lshaw8317 deleted the improve_slice_eval branch August 8, 2025 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants