Skip to content

AsyncIterator API for Result consumption #831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Jan 25, 2022

Conversation

bigmontz
Copy link
Contributor

The current driver implementation has only two options for record consumption, the default async/promise api and the reactive api.

In the async side, there is no simple way to interact over the results before consuming all the data stream and back-pressure the consumption. Meanwhile in the reactive, the back-pressure is implemented but this api usage depends on the knowledge of reactive programming and RxJS. Another issue with our current reactive implementation is not having the way to configure the back-pressure or pause the stream consumption.

The introduction of the Async Iterator is part of a group of improvements to make the driver more idiomatic and memory efficient.

Comment on lines +1519 to +1529
const session = driver.session()
try {
const result = session.run('UNWIND RANGE(0, 10) AS x RETURN x')
const xs = []
for await (const record of result) {
xs.push(record.get('x').toInt())
}
expect(xs).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
} finally {
await session.close()
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usage example

@bigmontz bigmontz added work in progress 5.0 PR which should be merged to the 5.0 version labels Dec 28, 2021
@bigmontz bigmontz force-pushed the 5.0-async-iterator-api branch from 83fff10 to 1c97865 Compare December 30, 2021 12:09
@bigmontz bigmontz changed the base branch from 4.4 to 5.0 January 5, 2022 11:28
@bigmontz bigmontz force-pushed the 5.0-async-iterator-api branch 6 times, most recently from 70bc926 to 73bea11 Compare January 11, 2022 12:02
@bigmontz bigmontz marked this pull request as ready for review January 12, 2022 10:36
@bigmontz bigmontz linked an issue Jan 12, 2022 that may be closed by this pull request
@bigmontz bigmontz force-pushed the 5.0-async-iterator-api branch 4 times, most recently from 6665c57 to 7f008e6 Compare January 19, 2022 12:12
@bigmontz bigmontz force-pushed the 5.0-async-iterator-api branch from 7f008e6 to efc064d Compare January 21, 2022 12:26
Summary was not available with the correct information after the result being consumed. This fix, solve this issue by wrap the summary with the same treatment subscribe and then have on complete.

This change also introduce a cache for capturing the last summary the stream complete, this way the summary function could be called after the results get consumed.

PS: The same fixes were applied to keys()
@bigmontz bigmontz force-pushed the 5.0-async-iterator-api branch from ba89fe3 to 1d87e11 Compare January 24, 2022 11:48
@bigmontz bigmontz force-pushed the 5.0-async-iterator-api branch from 1d87e11 to 1078485 Compare January 24, 2022 11:57
This error happens in newer Nodejs versions.
@bigmontz bigmontz merged commit 5b4f475 into neo4j:5.0 Jan 25, 2022
@bigmontz bigmontz deleted the 5.0-async-iterator-api branch January 25, 2022 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5.0 PR which should be merged to the 5.0 version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Pull through streaming API to prevent back-pressure
1 participant