-
Notifications
You must be signed in to change notification settings - Fork 264
Bulk ingester retry policy #930
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good overall, except for some potentially costly list operations. Bulk ingestion is used in high load contexts where performance is critical, and the number of operations per request (and thus list size) can be large.
...nt/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkOperationRepeatable.java
Outdated
Show resolved
Hide resolved
...nt/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkOperationRepeatable.java
Outdated
Show resolved
Hide resolved
...nt/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkOperationRepeatable.java
Outdated
Show resolved
Hide resolved
java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java
Outdated
Show resolved
Hide resolved
java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java
Outdated
Show resolved
Hide resolved
java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java
Outdated
Show resolved
Hide resolved
java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java
Outdated
Show resolved
Hide resolved
java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java
Outdated
Show resolved
Hide resolved
java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Great stuff!
* WIP: added backoff policy logic to operations * all basic key points implemented * bugfixes * retry - main implementation done * working unit test, bugfixes * more tests * unit tests done * refactoring * more refactoring * checkstyle * refactoring and optimizations
* WIP: added backoff policy logic to operations * all basic key points implemented * bugfixes * retry - main implementation done * working unit test, bugfixes * more tests * unit tests done * refactoring * more refactoring * checkstyle * refactoring and optimizations
* WIP: added backoff policy logic to operations * all basic key points implemented * bugfixes * retry - main implementation done * working unit test, bugfixes * more tests * unit tests done * refactoring * more refactoring * checkstyle * refactoring and optimizations Co-authored-by: Laura Trotta <[email protected]>
* WIP: added backoff policy logic to operations * all basic key points implemented * bugfixes * retry - main implementation done * working unit test, bugfixes * more tests * unit tests done * refactoring * more refactoring * checkstyle * refactoring and optimizations Co-authored-by: Laura Trotta <[email protected]>
* WIP: added backoff policy logic to operations * all basic key points implemented * bugfixes * retry - main implementation done * working unit test, bugfixes * more tests * unit tests done * refactoring * more refactoring * checkstyle * refactoring and optimizations Co-authored-by: Laura Trotta <[email protected]>
Adds retry logic to the bulk ingester, allowing it to to retry operations that failed with error 429 (too many requests), hoping that the error will recover and the request will go through. Users can configure the desired backoff policy using the
backoffPolicy()
method in the bulk ingester builder, which will look like this:This is an example of constant backoff, meaning the single failed operation will be retried 8 times every 50 milliseconds.