Description
Describe the bug
During merge segments call, for high cardinality cases, the actual file size can go beyond 2.2 GB and star tree builder breaks during the same because the offsets of the file are maintained as integers. So at max it can process Integer.maxValue bytes.
Related component
Indexing:Performance
To Reproduce
Merge segments of index where star tree was enabled and indexed with high cardinality dimensions + lots of metrics. It'll end up in error once the offsets array (which tracks the bytes start position for each document) crosses Integer.Max_Value .
Expected behavior
We can remove the offsets tracker completely as the size of each star tree document is fixed length as of now.
In StarTreeDocsFileManager, based on the current file start docId, we can calculate the offset for a particular docId.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.