Skip to content

Documentation feedback: /docs/sources/operations/automatic-stream-sharding.md #17183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
oofnikj opened this issue Apr 12, 2025 · 0 comments
Open
Labels
type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories type/feature Something new we should do

Comments

@oofnikj
Copy link

oofnikj commented Apr 12, 2025

I think there should be a section here about time sharding.

Our use case involves collecting logs from 100,000s of mobile devices. Logs are concatenated client-side and uploaded to object storage. Once the objects are uploaded, an event-driven pipeline parses the logs and pushes them to a Loki cluster running in Kubernetes. Sometimes a single log file can contain log lines spanning multiple days, and the logs are not necessarily concatenated in chronological order.
We struggled with out-of-order log ingestion and considered building a customized pipeline step that would aggregate and sort larger batches of logs. Luckily, after trawling through the configuration parameters, time sharding came to the rescue and elegantly solved the problem.

It would have saved us a lot of time if the time sharding feature were documented alongside other automatic sharding capabilities of Loki.

@JStickler JStickler added type/feature Something new we should do type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories labels Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories type/feature Something new we should do
Projects
None yet
Development

No branches or pull requests

2 participants