[BUG] Writable Warm - Composite Directory sync() method doesn't take into account block files

[rayshrey]

Describe the bug

Currently in composite directory, in the sync() method we filter out the files that hare present in Remote and then delegate the sync() call to localDirectory with the filtered set of files. But there can arise cases where the file is present in blocks instead of a full file and that results in a NoSuchFileException for the localDirectory sync() call.

Example stacktrace below:

2025-06-04T13:21:03,574][ERROR][o.o.i.r.PeerRecoveryTargetService] [c2af93180a503303bf0aa093eb62a955] Exception while downloading segment files from remote store, will continue with peer to peer segment copy
[test_idxx/1010101T7R9GrsTUi9RjDhMmAcPg][[test_idxx][1]] IndexShardRecoveryException[Exception while copying segment files from remote segment store]; nested: NoSuchFileException[/<path>/var/es/data/nodes/0/indices/1010101T7R9GrsTUi9RjDhMmAcPg/1/index/_o_Lucene90_0.dvd];
	at org.opensearch.index.shard.IndexShard.syncSegmentsFromRemoteSegmentStore(IndexShard.java:5294)
	at org.opensearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:256)
	at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:626)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.file.NoSuchFileException: /<path>/var/es/data/nodes/0/indices/1010101T7R9GrsTUi9RjDhMmAcPg/1/index/_o_Lucene90_0.dvd
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:224)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:309)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:369)
	at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:465)
	at org.apache.lucene.store.FSDirectory.fsync(FSDirectory.java:303)
	at org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:255)
	at org.opensearch.index.store.CompositeDirectory.sync(CompositeDirectory.java:233)
	at org.apache.lucene.store.FilterDirectory.sync(FilterDirectory.java:86)
	at org.apache.lucene.store.FilterDirectory.sync(FilterDirectory.java:86)
	at org.opensearch.index.store.Store.commitSegmentInfos(Store.java:905)
	at org.opensearch.index.shard.IndexShard.syncSegmentsFromRemoteSegmentStore(IndexShard.java:5290)
	... 7 more

Related component

Storage:Remote

To Reproduce

Run ingestion for some prolonged time.

Expected behavior

Composite Directory should be able to handle sync for block files as well.

Additional Details

N/A

[gbbafna]

Currently in composite directory, in the sync() method we filter out the files that hare present in Remote and then delegate the sync() call to localDirectory with the filtered set of files. But there can arise cases where the file is present in blocks instead of a full file and that results in a NoSuchFileException for the localDirectory sync() call.

If a file is present in blocks, it has to be present on remote as well ,right ?

[rayshrey]

If a file is present in blocks, it has to be present on remote as well

Yes file should change to block only after it has been uploaded to remote
But listAll() of the remote directory shows only the latest files present after metadata refresh
So it may happen that a stale file (which hasn't been deleted on local yet) may not be present in remote
In that case if sync call comes, sync() calls to local directory fails as it tries to sync the full file instead of the block.