Skip to content

Automatic stream sharding is disabled with empty overrides #17277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
umezawatakeshi opened this issue Apr 17, 2025 · 0 comments
Open

Automatic stream sharding is disabled with empty overrides #17277

umezawatakeshi opened this issue Apr 17, 2025 · 0 comments

Comments

@umezawatakeshi
Copy link

Describe the bug

If an empty mapping is specified for overrides (i.e. overrides: {}) in overrides.yaml, automatic stream sharding is wrongly disabled.

Jsonnet deployment (production/ksonnet/loki) generates such manifest by default.

If -limits.per-user-override-config is not passed to distributors or overrides is null or is not specified, automatic stream sharding is enabled by default as specified in limits_config.shard_streams.enabled.

To Reproduce

  1. Started Loki (I'm using v3.4.2) with -limits.per-user-override-config=overrides.yaml, where overrides.yaml contains overrides: {}
    • Check the distributors' metrics. loki_rate_store_max_stream_rate_bytes, which has meaningful value and is updated by time if automatic stream sharding is enabled, is always zero.
  2. Modify overrides.yaml. overrides: null
    • Check the distributors' metrics. loki_rate_store_max_stream_rate_bytes is now updated.
  3. Modify overrides.yaml. Delete overrides field.
    • Check the distributors' metrics. loki_rate_store_max_stream_rate_bytes is still updated.
  4. restart the distributors without setting -limits.per-user-override-config
    • Check the distributors' metrics. loki_rate_store_max_stream_rate_bytes is updated too.

Expected behavior

Automatic stream sharding is still enabled.

anyShardingEnabled() function, which checks sharding is enabled for any of tenant, does not seem to check the default setting in limits_config if some overrides are written. However, even if some override configs are written for some tenants, other tenants still use the default settings. So, the default setting in limits_config MUST be always checked.

Environment:

  • Loki version: v3.4.2
  • Infrastructure: Kubernetes
  • Deployment tool: jsonnet
  • Deployment mode: microservice (distributed)

Screenshots, Promtail config, or terminal output

(none)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant