Skip to content

[hotfix][Connectors/Kinesis] Pass source configuration to SplitFetcherManager #192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

lukaszkrawiec
Copy link

@lukaszkrawiec lukaszkrawiec commented Mar 4, 2025

Purpose of the change

SplitFetcherManager relies on correct Configuration be passed to its constructor such that it can honor flink options such as

  • ALLOW_UNALIGNED_SOURCE_SPLITS
  • ELEMENT_QUEUE_CAPACITY

This commit forwards the configuration object provided to KinesisStreamsSource to SplitFetcherManager

Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

  • Dependencies have been added or upgraded
  • Public API has been changed (Public API is any class annotated with @Public(Evolving))
  • Serializers have been changed
  • New feature has been introduced
    • If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

Copy link

boring-cyborg bot commented Mar 4, 2025

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

@lukaszkrawiec lukaszkrawiec force-pushed the split-fetcher-configs branch from ce2882b to 4293702 Compare March 4, 2025 06:28
@lukaszkrawiec lukaszkrawiec changed the title [hotfix] make SplitFetcherManager honor job configuration [hotfix][Connectors/Kinesis] Pass source configuration to SplitFetcherManager Mar 4, 2025
Copy link
Contributor

@vahmed-hamdy vahmed-hamdy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice Catch! Thanks for the contribution

Copy link
Contributor

@leekeiabstraction leekeiabstraction left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Lukasz for raising the PR!

I have a question regarding ALLOW_UNALIGNED_SOURCE_SPLITS.

I see that the configuration will affect the correctness of watermarks. Would enabling this cause records which belong to the same key/keygroup to arrive in different order? If so, have we considered excluding the passing of ALLOW_UNALIGNED_SOURCE_SPLITS to prevent users from shooting themselves in the foot?

If my assumption is correct on ordering, I am curious about the use-case where ALLOW_UNALIGNED_SOURCE_SPLITS would make sense for users especially since FIFO is a key characteristic of Kinesis stream. If there is a strong reason to support this, is there a way to warn users?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants