Skip to content

Defer remote file filtering logic until a file is processed #2777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
garyrussell opened this issue Mar 1, 2019 · 0 comments
Closed

Defer remote file filtering logic until a file is processed #2777

garyrussell opened this issue Mar 1, 2019 · 0 comments
Assignees
@garyrussell
Labels
in: sftp type: refactoring
Milestone
5.2.M1

Comments

@garyrussell
Copy link
Contributor

The fix for #2776 will work by reversing the filtered files on exceptions and when the adapter is stopped.

But this will not be a complete solution because unprocessed files can still be filtered if the server loses power.

Moving the filter logic until as late as possible will mitigate the extent of this problem.
@garyrussell garyrussell added this to the 5.2.M1 milestone Mar 1, 2019
@garyrussell garyrussell self-assigned this Mar 1, 2019
@garyrussell garyrussell changed the title Defer remote file filtering logic until file is processed Defer remote file filtering logic until a file is processed Mar 1, 2019
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Mar 1, 2019
@garyrussell
spring-projectsGH-2776: Fix Streaming Remote File MessageSource
e659c06 
Fixes spring-projects#2776
Also see spring-projects#2777

- reset the filter for the current file if the fetch fails
- implement `Lifecycle` and clear the `toBeReceived` queue and corresponding
  filter entries
@garyrussell garyrussell mentioned this issue Mar 1, 2019
Merged
artembilan pushed a commit that referenced this issue Mar 1, 2019
@garyrussell @artembilan
GH-2776: Fix Streaming Remote File MessageSource
a167290 
Fixes #2776
Also see #2777

- reset the filter for the current file if the fetch fails
- implement `Lifecycle` and clear the `toBeReceived` queue and corresponding
  filter entries

* Polishing - PR Comments **cherry-pick to all supported**

* Polishing
artembilan pushed a commit that referenced this issue Mar 1, 2019
@garyrussell @artembilan
GH-2776: Fix Streaming Remote File MessageSource
648760b 
Fixes #2776
Also see #2777

- reset the filter for the current file if the fetch fails
- implement `Lifecycle` and clear the `toBeReceived` queue and corresponding
  filter entries

* Polishing - PR Comments **cherry-pick to all supported**

* Polishing

# Conflicts:
#	spring-integration-file/src/test/java/org/springframework/integration/file/remote/StreamingInboundTests.java
#	spring-integration-ftp/src/test/java/org/springframework/integration/ftp/inbound/FtpStreamingMessageSourceTests.java
#	spring-integration-jdbc/src/test/java/org/springframework/integration/jdbc/StoredProcJavaConfigTests.java
artembilan pushed a commit that referenced this issue Mar 1, 2019
@garyrussell @artembilan
GH-2776: Fix Streaming Remote File MessageSource
34febd4 
Fixes #2776
Also see #2777

- reset the filter for the current file if the fetch fails
- implement `Lifecycle` and clear the `toBeReceived` queue and corresponding
  filter entries

* Polishing - PR Comments **cherry-pick to all supported**

* Polishing

# Conflicts:
#	spring-integration-file/src/test/java/org/springframework/integration/file/remote/StreamingInboundTests.java
#	spring-integration-ftp/src/test/java/org/springframework/integration/ftp/inbound/FtpStreamingMessageSourceTests.java
#	spring-integration-jdbc/src/test/java/org/springframework/integration/jdbc/StoredProcJavaConfigTests.java

# Conflicts:
#	spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java
#	spring-integration-file/src/test/java/org/springframework/integration/file/remote/StreamingInboundTests.java
artembilan pushed a commit that referenced this issue Mar 1, 2019
@garyrussell @artembilan
GH-2776: Fix Streaming Remote File MessageSource
45789a9 
Fixes #2776
Also see #2777

- reset the filter for the current file if the fetch fails
- implement `Lifecycle` and clear the `toBeReceived` queue and corresponding
  filter entries

* Polishing - PR Comments **cherry-pick to all supported**

* Polishing

# Conflicts:
#	spring-integration-file/src/test/java/org/springframework/integration/file/remote/StreamingInboundTests.java
#	spring-integration-ftp/src/test/java/org/springframework/integration/ftp/inbound/FtpStreamingMessageSourceTests.java
#	spring-integration-jdbc/src/test/java/org/springframework/integration/jdbc/StoredProcJavaConfigTests.java

# Conflicts:
#	spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java
#	spring-integration-file/src/test/java/org/springframework/integration/file/remote/StreamingInboundTests.java

# Conflicts:
#	spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java
#	spring-integration-file/src/test/java/org/springframework/integration/file/remote/StreamingInboundTests.java
#	spring-integration-ftp/src/test/java/org/springframework/integration/ftp/inbound/FtpStreamingMessageSourceTests.java
#	spring-integration-sftp/src/test/java/org/springframework/integration/sftp/inbound/SftpStreamingMessageSourceTests.java
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Mar 1, 2019
@garyrussell
spring-projectsGH-2777: Remote File Filter Improvements
03cf2e9 
Resolves spring-projects#2777

If the filter supports it, defer filtering until the last possible moment.
Then, the worst case scenario after a catestrophic failure (e.g. power loss),
would be that at most one file will be incorrectly filtered on restart.
@garyrussell garyrussell mentioned this issue Mar 1, 2019
Closed
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Mar 5, 2019
@garyrussell
spring-projectsGH-2777: Remote File Filter Improvements
f8f72b8 
Resolves spring-projects#2777

If the filter supports it, defer filtering until the last possible moment.
Then, the worst case scenario after a catestrophic failure (e.g. power loss),
would be that at most one file will be incorrectly filtered on restart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@garyrussell garyrussell

Labels
in: sftp type: refactoring
Projects
None yet
Milestone
5.2.M1
Development

Successfully merging a pull request may close this issue.

GH-2777: Remote File Filter Improvements garyrussell/spring-integration
1 participant
@garyrussell