Skip to content

Ruler is disabled in the Loki but it is trying to access s3 Bucket #17079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sintopaulose opened this issue Apr 9, 2025 · 1 comment
Open

Comments

@sintopaulose
Copy link

sintopaulose commented Apr 9, 2025

I'm trying to run Loki in simple scalable mode and S3 as backend.

Getting below error logs after pods are up and running.

level=error ts=2025-04-09T09:02:24.841250811Z caller=compactor.go:534 msg="failed to run compaction" err="failed to list tables: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=info ts=2025-04-09T09:02:24.841261436Z caller=compactor.go:592 msg="compactor started"
level=error ts=2025-04-09T09:03:19.980442989Z caller=ruler.go:576 msg="unable to list rules" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=error ts=2025-04-09T09:04:19.980590684Z caller=ruler.go:576 msg="unable to list rules"

Not able to get rid of this error message. Look like loki component is trying to connect to ruler's bucket though the ruler is disabled and It is not able to find the AWS keys

Related settings in loki's values.yaml.


ruler:
enabled: false

**
schemaConfig:
configs:
- from: '2025-01-01'
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage:
type: s3
bucketNames:
chunks: bucketname
ruler: bucketname
admin: bucketname

Observation 1:

If I mount the secret contains AWS keys, this error is going away.
Backend:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: secret_name
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: secret_name
key: AWS_SECRET_ACCESS_KEY
Observation 2:

If I comment out the ruler bucket name and keeping ruler as disabled state getting below error message.

level=error ts=2025-04-08T11:33:33.41502519Z caller=log.go:216 msg="error running loki" err="at least one bucket name must be specified\ngithub.com/grafana/loki/v3/pkg/storage/chunk/client/aws.buckets\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:304\ngithub.com/grafana/loki/v3/pkg/storage/chunk/client/aws.NewS3ObjectClient\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:153\ngithub.com/grafana/loki/v3/pkg/ruler/base.NewLegacyRuleStore\n\t/src/loki/pkg/ruler/base/storage.go:99
\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).initRulerStorage\n\t/src/loki/pkg/loki/modules.go:1254
\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:492
\nmain.main\n\t/src/loki/cmd/loki/main.go:129
\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:492\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
[slab811admin@slab811provnode template]$

Noted there is limitation of ruler.

Image

Is this causing any issue?

is this a known issue or any workaround for this to solve it.

@sintopaulose sintopaulose changed the title Ruler is disabled in the Loki but it is trying to access Bucket when it is S3 configuration. Ruler is disabled in the Loki but it is trying to access s3 Bucket Apr 9, 2025
@JStickler
Copy link
Contributor

Questions have a better chance of being answered if you ask them on the community forums.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants