Documentation feedback: /docs/sources/operations/automatic-stream-sharding.md #17183
Labels
type/docs
Issues related to technical documentation; the Docs Squad uses this label across many repositories
type/feature
Something new we should do
I think there should be a section here about time sharding.
Our use case involves collecting logs from 100,000s of mobile devices. Logs are concatenated client-side and uploaded to object storage. Once the objects are uploaded, an event-driven pipeline parses the logs and pushes them to a Loki cluster running in Kubernetes. Sometimes a single log file can contain log lines spanning multiple days, and the logs are not necessarily concatenated in chronological order.
We struggled with out-of-order log ingestion and considered building a customized pipeline step that would aggregate and sort larger batches of logs. Luckily, after trawling through the configuration parameters, time sharding came to the rescue and elegantly solved the problem.
It would have saved us a lot of time if the time sharding feature were documented alongside other automatic sharding capabilities of Loki.
The text was updated successfully, but these errors were encountered: