Python API: Support Deferred mode for read #4530
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pnorbert
force-pushed
the
python-read-deferred2
branch
2 times, most recently
from
a14ef7e to
78742d5
Compare
…() functions. Need to use Stream.read_complete() to make sure data is read into the numpy/etc arrays. Timing test added to examples/hello/deferredRead $ python3 ./deferred_timing_test.py Write 1000 arrays of [100, 1000] shape Time of writing: 1.082126s Incoming nx, ny, nvars = 100, 1000, 1000 Time of reading one by one (Sync mode): 0.306817s Incoming nx, ny, nvars = 100, 1000, 1000 Time of reading at once (Deferred mode): 0.067873s
pnorbert
force-pushed
the
python-read-deferred2
branch
from
78742d5 to
ef9be6d
Compare
anagainaru
reviewed
May 21, 2025
Member
anagainaru
left a comment
anagainaru
left a comment
There was a problem hiding this comment.
Looks good to me, I will run it on Frontier can come back with an Approve
| count=[], | ||
| block_id=None, | ||
| step_selection=None, | ||
| defer_read: bool = False, |
Member
There was a problem hiding this comment.
I'm surprised the format did not complained about this :)
Suggested change
| defer_read: bool = False, | |
| defer_read: bool=False, |
| count=[], | ||
| block_id=None, | ||
| step_selection=None, | ||
| defer_read: bool = False, |
Member
There was a problem hiding this comment.
Suggested change
| defer_read: bool = False, | |
| defer_read: bool=False, |
Contributor
Author
There was a problem hiding this comment.
flake8 does not like your perfectly reasonable suggestions.
python/adios2/stream.py:566: [E252] missing whitespace around parameter equals
anagainaru
approved these changes
May 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
defer_read=booloption toStream.read()andStream.read_in_buffer()functions. Need to useStream.read_complete()to make sure data is read into the numpy/etc arrays. Timing test added toexamples/hello/deferredRead