Skip to content

Conversation

@willi-mueller
Copy link
Collaborator

@willi-mueller willi-mueller commented Aug 9, 2024

Description

  • Adds a new option to RangePaginator and its subclasses OffsetPaginator and PageNumberPaginator to stop pagination upon receiving a page without data items
  • By default, stops pagination after an empty page
  • Every RangePaginator now requires either total_path or maximum_offset or stop_after_empty_page
  • If total_path is not provided, pagination is controlled by maximum_page and stop_after_empty_page or maximum_offset and stop_after_empty_page respectively.

Related Issues

@netlify
Copy link

netlify bot commented Aug 9, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 83bab15
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66c339cf80854200087f47ef

@willi-mueller willi-mueller changed the title Feat/1637 stop pagination after empty page rest_client: stops pagination after empty page (Feat/1637) Aug 9, 2024
@willi-mueller willi-mueller requested a review from burnash August 9, 2024 10:38
@willi-mueller willi-mueller self-assigned this Aug 9, 2024
@willi-mueller willi-mueller changed the title rest_client: stops pagination after empty page (Feat/1637) RESTClient: stops pagination after empty page (Feat/1637) Aug 9, 2024
@burnash burnash added the enhancement New feature or request label Aug 9, 2024
@burnash burnash linked an issue Aug 9, 2024 that may be closed by this pull request
burnash
burnash previously approved these changes Aug 15, 2024
Copy link
Collaborator

@burnash burnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, @willi-mueller. I've left a couple of comments

Copy link
Collaborator

@burnash burnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, thank you @willi-mueller

@willi-mueller willi-mueller merged commit d448122 into devel Aug 19, 2024
@willi-mueller willi-mueller deleted the feat/1637_stop-pagination-after-empty-page branch August 19, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Update RESTClient range paginators to support stopping after empty pages Additional paginator for offset without a "total_param"

3 participants