You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to run Loki in simple scalable mode and S3 as backend.
Getting below error logs after pods are up and running.
level=error ts=2025-04-09T09:02:24.841250811Z caller=compactor.go:534 msg="failed to run compaction" err="failed to list tables: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=info ts=2025-04-09T09:02:24.841261436Z caller=compactor.go:592 msg="compactor started"
level=error ts=2025-04-09T09:03:19.980442989Z caller=ruler.go:576 msg="unable to list rules" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=error ts=2025-04-09T09:04:19.980590684Z caller=ruler.go:576 msg="unable to list rules"
Not able to get rid of this error message. Look like loki component is trying to connect to ruler's bucket though the ruler is disabled and It is not able to find the AWS keys
If I mount the secret contains AWS keys, this error is going away.
Backend:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: secret_name
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: secret_name
key: AWS_SECRET_ACCESS_KEY Observation 2:
If I comment out the ruler bucket name and keeping ruler as disabled state getting below error message.
level=error ts=2025-04-08T11:33:33.41502519Z caller=log.go:216 msg="error running loki" err="at least one bucket name must be specified\ngithub.com/grafana/loki/v3/pkg/storage/chunk/client/aws.buckets\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:304\ngithub.com/grafana/loki/v3/pkg/storage/chunk/client/aws.NewS3ObjectClient\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:153\ngithub.com/grafana/loki/v3/pkg/ruler/base.NewLegacyRuleStore\n\t/src/loki/pkg/ruler/base/storage.go:99
\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).initRulerStorage\n\t/src/loki/pkg/loki/modules.go:1254
\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:492
\nmain.main\n\t/src/loki/cmd/loki/main.go:129
\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:492\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
[slab811admin@slab811provnode template]$
Noted there is limitation of ruler.
Is this causing any issue?
is this a known issue or any workaround for this to solve it.
The text was updated successfully, but these errors were encountered:
sintopaulose
changed the title
Ruler is disabled in the Loki but it is trying to access Bucket when it is S3 configuration.
Ruler is disabled in the Loki but it is trying to access s3 Bucket
Apr 9, 2025
I'm trying to run Loki in simple scalable mode and S3 as backend.
Getting below error logs after pods are up and running.
level=error ts=2025-04-09T09:02:24.841250811Z caller=compactor.go:534 msg="failed to run compaction" err="failed to list tables: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=info ts=2025-04-09T09:02:24.841261436Z caller=compactor.go:592 msg="compactor started"
level=error ts=2025-04-09T09:03:19.980442989Z caller=ruler.go:576 msg="unable to list rules" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=error ts=2025-04-09T09:04:19.980590684Z caller=ruler.go:576 msg="unable to list rules"
Not able to get rid of this error message. Look like loki component is trying to connect to ruler's bucket though the ruler is disabled and It is not able to find the AWS keys
Related settings in loki's values.yaml.
ruler:
enabled: false
**
schemaConfig:
configs:
- from: '2025-01-01'
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage:
type: s3
bucketNames:
chunks: bucketname
ruler: bucketname
admin: bucketname
Observation 1:
If I mount the secret contains AWS keys, this error is going away.
Backend:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: secret_name
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: secret_name
key: AWS_SECRET_ACCESS_KEY
Observation 2:
If I comment out the ruler bucket name and keeping ruler as disabled state getting below error message.
level=error ts=2025-04-08T11:33:33.41502519Z caller=log.go:216 msg="error running loki" err="at least one bucket name must be specified\ngithub.com/grafana/loki/v3/pkg/storage/chunk/client/aws.buckets\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:304\ngithub.com/grafana/loki/v3/pkg/storage/chunk/client/aws.NewS3ObjectClient\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:153\ngithub.com/grafana/loki/v3/pkg/ruler/base.NewLegacyRuleStore\n\t/src/loki/pkg/ruler/base/storage.go:99
\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).initRulerStorage\n\t/src/loki/pkg/loki/modules.go:1254
\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:492
\nmain.main\n\t/src/loki/cmd/loki/main.go:129
\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:492\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
[slab811admin@slab811provnode template]$
Noted there is limitation of ruler.
Is this causing any issue?
is this a known issue or any workaround for this to solve it.
The text was updated successfully, but these errors were encountered: