[META] Merged segment pre-copy

[guojialiang92]

Please describe the end goal of this project

We plan to introduce pre-copy of merged segments in segment copy. It can bring the following benefits:

• Reduce the latency of sharding by more than 20 times.
• The replicas are synchronized with the primary shard more timely, reducing the number of translog recoveries when the replicas are promoted.

Supporting References

The RFC details the background.

Issues

• Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881
• [merged segment warmer] support local merged segment warmer [merged segment warmer] support local merged segment warmer #18255
• Modify merged segment pre-copy classes licenses Modify merged segment pre-copy classes licenses #18737
• Support remote store
• [merged segment warmer] support cleanup redundant pending merged segments [merged segment warmer] support cleanup redundant pending merged segments #18720
• Status statistics of pre-copy for merged segment
• [Feature Request] segment replication and merged segment warmer use a dedicated threadpool [Feature Request] segment replication and merged segment warmer use a dedicated threadpool #18755

Related component

Indexing:Replication

[guojialiang92]

Following @ashking94's suggestion, this META was created to track the overall progress of the merge segment pre-copy feature.
If you have other ideas, you can continue to add them. @ashking94 @shourya035.

[ashking94]

Thanks @guojialiang92, we can track the items for getting the segrep indexes merged segment pre-copy effort here.

[guojialiang92]

Hi, @shourya035 @ashking94
I've submitted a PR ([merged segment warmer] support cleanup redundant pending merged segments). Please take some time to review it. :) Thank you.