Error syncing tables to S3 after upgrading to 2.5.0 from 2.4.2 - "invalid key: "

[nc-tamu]

After upgrading from 2.4.2 to 2.5.0, the reader instance of Loki can no longer sync index tables with S3 storage.
Problem occours when trying to search for anything via Grafana interface:

On the log for the read instance the following can be seen:

level=info ts=2022-04-13T15:04:46.623064275Z caller=table_manager.go:176 msg="downloading all files for table loki_index_19095"
level=error ts=2022-04-13T15:04:46.672656292Z caller=cached_client.go:64 msg="failed to build cache" err="invalid key: "
level=error ts=2022-04-13T15:04:46.672702196Z caller=index_set.go:99 msg="failed to initialize table loki_index_19095, cleaning it up" err="invalid key: "
level=error ts=2022-04-13T15:04:46.67271967Z caller=table.go:304 table-name=loki_index_19095 msg="failed to init user index set fake" err="invalid key: "
level=info ts=2022-04-13T15:04:46.672730426Z caller=table.go:298 table-name=loki_index_19095 user-id=fake msg="downloaded index set at query time" duration=0.049333251
level=error ts=2022-04-13T15:04:46.672762948Z caller=chunk_store.go:526 org_id=fake msg="error querying storage" err="invalid key: "
level=error ts=2022-04-13T15:04:46.672792374Z caller=chunk_store.go:526 org_id=fake msg="error querying storage" err="invalid key: "
level=info ts=2022-04-13T15:04:46.67286043Z caller=metrics.go:122 component=querier org_id=fake latency=fast query="{env=\"tfe\"}" query_type=limited range_type=range length=5m0s step=200ms duration=52.296479ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B queue_time=59.751µs subqueries=1
level=error ts=2022-04-13T15:04:46.673922719Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = invalid key: \n"
level=warn ts=2022-04-13T15:04:46.673932671Z caller=logging.go:72 traceID=56527f04f4314591 orgID=fake msg="GET /loki/api/v1/query_range?direction=BACKWARD&limit=1000&query=%7Benv%3D%22tfe%22%7D&start=1649861986466000000&end=1649862286466000000&step=0.2 (500) 54.564886ms Response: \"invalid key: \\n\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip, deflate, br; Accept-Language: en-US,en;q=0.9; Sec-Ch-Ua: \" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"; Sec-Ch-Ua-Mobile: ?0; Sec-Ch-Ua-Platform: \"Windows\"; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Grafana/8.3.3; X-Forwarded-For: 10.128.191.128, 10.128.191.128; X-Forwarded-Ssl: on; X-Grafana-Org-Id: 1; X-Original-Forwarded-For:; X-Real-Ip: ; X-Request-Id: 4ac2c36eb721afb205d0b5b96d217ecb; X-Scheme: https; "
level=error ts=2022-04-13T15:04:46.676009813Z caller=table.go:167 table-name=loki_index_19095 org_id=fake msg="index set fake has some problem, cleaning it up" err="invalid key: "
level=error ts=2022-04-13T15:04:46.676173924Z caller=chunk_store.go:526 org_id=fake msg="error querying storage" err="invalid key: "
level=error ts=2022-04-13T15:04:46.676614796Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = invalid key: \n"
level=error ts=2022-04-13T15:04:46.699635889Z caller=cached_client.go:64 msg="failed to build cache" err="invalid key: "
level=error ts=2022-04-13T15:04:46.699686752Z caller=index_set.go:99 msg="failed to initialize table loki_index_19095, cleaning it up" err="invalid key: "
level=error ts=2022-04-13T15:04:46.699704536Z caller=table.go:304 table-name=loki_index_19095 msg="failed to init user index set fake" err="invalid key: "
level=info ts=2022-04-13T15:04:46.699717285Z caller=table.go:298 table-name=loki_index_19095 user-id=fake msg="downloaded index set at query time" duration=0.020270309
level=error ts=2022-04-13T15:04:46.699746892Z caller=chunk_store.go:526 org_id=fake msg="error querying storage" err="invalid key: "
level=error ts=2022-04-13T15:04:46.700191724Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = invalid key: \n"
level=error ts=2022-04-13T15:04:46.70248434Z caller=table.go:167 table-name=loki_index_19095 org_id=fake msg="index set fake has some problem, cleaning it up" err="invalid key: "
level=error ts=2022-04-13T15:04:46.702644394Z caller=chunk_store.go:526 org_id=fake msg="error querying storage" err="invalid key: "
level=error ts=2022-04-13T15:04:46.734802176Z caller=retry.go:73 org_id=fake msg="error processing request" try=3 err="rpc error: code = Code(500) desc = invalid key: \n"
level=error ts=2022-04-13T15:04:46.759067415Z caller=cached_client.go:64 msg="failed to build cache" err="invalid key: "
level=error ts=2022-04-13T15:04:46.759110922Z caller=index_set.go:99 msg="failed to initialize table loki_index_19095, cleaning it up" err="invalid key: "
level=error ts=2022-04-13T15:04:46.759132366Z caller=table.go:304 table-name=loki_index_19095 msg="failed to init user index set fake" err="invalid key: "
level=info ts=2022-04-13T15:04:46.75914639Z caller=table.go:298 table-name=loki_index_19095 user-id=fake msg="downloaded index set at query time" duration=0.019889037
level=error ts=2022-04-13T15:04:46.759180956Z caller=chunk_store.go:526 org_id=fake msg="error querying storage" err="invalid key: "
level=error ts=2022-04-13T15:04:46.759700945Z caller=retry.go:73 org_id=fake msg="error processing request" try=4 err="rpc error: code = Code(500) desc = invalid key: \n"
level=warn ts=2022-04-13T15:04:46.759806671Z caller=logging.go:72 traceID=102585fd79655ea0 orgID=fake msg="GET /loki/api/v1/label?start=1649861986518000000&end=1649862286518000000 (500) 102.569608ms Response: \"invalid key: \\n\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip, deflate, br; Accept-Language: en-US,en;q=0.9; Sec-Ch-Ua: \" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"; Sec-Ch-Ua-Mobile: ?0; Sec-Ch-Ua-Platform: \"Windows\"; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Grafana/8.3.3; X-Forwarded-For: , 10.128.213.0, 10.128.213.0; X-Forwarded-Ssl: on; X-Grafana-Org-Id: 1; X-Original-Forwarded-For: ; X-Real-Ip: ; X-Request-Id: d47fe27867462d323ca5830a75dd3eda; X-Scheme: https; "
level=info ts=2022-04-13T15:04:51.434721583Z caller=table_manager.go:209 msg="syncing tables"
level=error ts=2022-04-13T15:04:51.471117263Z caller=cached_client.go:64 msg="failed to build cache" err="invalid key: "
level=error ts=2022-04-13T15:04:51.471177807Z caller=table_manager.go:107 msg="error syncing local boltdb files with storage" err="failed to sync index set fake for table loki_index_19095: invalid key: "

The following stacktrace can be observed after restarting the container:

level=error ts=2022-04-13T15:19:31.18143427Z caller=log.go:100 msg="error running loki" err="invalid key: 
error creating index client
github.com/grafana/loki/pkg/storage/chunk/storage.NewStore
	/src/loki/pkg/storage/chunk/storage/factory.go:219
github.com/grafana/loki/pkg/loki.(*Loki).initStore
	/src/loki/pkg/loki/modules.go:394
github.com/grafana/dskit/modules.(*Manager).initModule
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:106
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:78
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:339
main.main
	/src/loki/cmd/loki/main.go:108
runtime.main
	/usr/local/go/src/runtime/proc.go:255
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1581
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:78
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:339
main.main
	/src/loki/cmd/loki/main.go:108
runtime.main
	/usr/local/go/src/runtime/proc.go:255
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1581

Downgrading back to 2.4.2 resolves this issue and objects can be seen downloaded from the S3 storage.

Config used for Loki:

# Config file for loki

auth_enabled: false

server:
  log_level: info
  http_listen_port: 3100
  http_listen_address: 0.0.0.0
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857608
  grpc_server_max_concurrent_streams: 1000

distributor:
  ring:
    kvstore:
      store: memberlist

memberlist:
  join_members:
    - loki-memberlist

common:
  #path_prefix: /data/loki
  replication_factor: 1
  storage:
    filesystem:
      chunks_directory: /data/loki/chunks
      rules_directory: /data/loki/rules

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  wal:
    enabled: true
    dir: /data/loki/wal
    flush_on_shutdown: true

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  aws:
    s3: https://${S3_ACCESS_KEY}:${S3_SECRET_KEY}@${S3_HOST}:443/${S3_BUCKET}
    s3forcepathstyle: true
    insecure: false
    http_config:
      insecure_skip_verify: true
  boltdb_shipper:
    active_index_directory: /data/loki/index
    cache_location: /data/loki/index_caching
    cache_ttl: 480h
    resync_interval: 5s
    shared_store: s3

compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  shared_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  compactor_ring:
    kvstore:
      store: memberlist

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h
  chunk_tables_provisioning:
    inactive_read_throughput: 10
    inactive_write_throughput: 10
    provisioned_read_throughput: 50
    provisioned_write_throughput: 20
  index_tables_provisioning:
    inactive_read_throughput: 10
    inactive_write_throughput: 10
    provisioned_read_throughput: 50
    provisioned_write_throughput: 20

Container args:

- -config.file=/etc/loki/config/config.yaml
- -target=read
- -config.expand-env=true

Any idea what is going on here?

[slim-bean]

Very strange, the only change was updating the version from 2.4.2 to 2.5.0?

[slim-bean]

The error comes from here: https://github.com/grafana/loki/blob/8f024954539b5eb706ab3ef967c4b1696ad04fd6/pkg/storage/stores/shipper/storage/cached_client.go#L149-L148

And in your errors it shows up as an empty string

The objects being iterated over come from a bucket list operation and this implies it's getting back an object which has a blank key which is strange.

What object storage are you using?

[slim-bean]

@sandeepsukhani I wonder if instead of erroring we should log the error and continue? WDYT?

[nc-tamu]

@slim-bean
Yes, the only thing I did was change the version number for the statefulsets for both the reader part and the writer part of Loki that is used. The write works, the reader breaks.

I am using HCP which is S3 compatible.

And in your errors it shows up as an empty string

Could this error be related to files being deleted in the bucket that Loki itself didn't do? Looking at the different index folders I see files that were deleted but no compactor file can be found:

So Loki tries to retrieve information from the bucket that does not exist anymore and Loki does not know how to handle that?

And for loki_index_19095 which the error is about the content looks like this:

Loki is constantly trying to read specific index folders all the time besides the newest one
level=info ts=2022-04-19T10:00:59.145694689Z caller=table_manager.go:207 msg="syncing tables"
level=debug ts=2022-04-19T10:00:59.145803845Z caller=table.go:366 msg="syncing files for table loki_index_19095"
level=debug ts=2022-04-19T10:00:59.179227235Z caller=table.go:373 msg="updates for table loki_index_19095. toDownload: [], toDelete: []"
level=debug ts=2022-04-19T10:00:59.179283274Z caller=table.go:366 msg="syncing files for table loki_index_19101"
level=debug ts=2022-04-19T10:00:59.207457413Z caller=table.go:373 msg="updates for table loki_index_19101. toDownload: [], toDelete: []"
level=debug ts=2022-04-19T10:00:59.20753369Z caller=table.go:366 msg="syncing files for table loki_index_19093"
level=debug ts=2022-04-19T10:00:59.211561502Z caller=table.go:373 msg="updates for table loki_index_19093. toDownload: [], toDelete: []"
level=debug ts=2022-04-19T10:00:59.211620732Z caller=table.go:366 msg="syncing files for table loki_index_19094"
level=debug ts=2022-04-19T10:00:59.217778033Z caller=table.go:373 msg="updates for table loki_index_19094. toDownload: [], toDelete: []"

For example, loki_index_19093 does not even exist:

[slim-bean]

It's not expected for an index folder to be empty unless there were no logs for the entire day..

[slim-bean]

i do suspect their is either an edge case here where this particular s3 implementation returns a list operation slightly different from others.

i think we should not error and fail though in this scenario and instead move on to the next index item, ill work with the team to make that change.

not sure about the missing folder. when you say "deleted that loki didnt do" do you mean you know another person or process is deleting files outside of loki?

[nc-tamu]

A file was deleted by a user when first setting up loki, seems like it conflicts even when everything is cleared.

I tried starting over with 2.5.0 to see if it persists and decided to target a sub directory within the bucket: bucket.host/loki
So now the storage config looks like this:

storage_config:
  aws:
    s3: https://${S3_ACCESS_KEY}:${S3_SECRET_KEY}@${S3_BUCKET}.${S3_HOST}.:443/loki

I started the writer and reader and the writer puts up data into the folder no problem:

However, the reader fails again with the same type of error, though with a different message. Instead of invalid key it returns 'NoSuchKey':

level=info ts=2022-04-22T09:25:22.907334665Z caller=compactor.go:356 msg="compactor startup delay completed"
level=info ts=2022-04-22T09:25:22.907419303Z caller=compactor.go:407 msg="compactor started"
level=info ts=2022-04-22T09:25:22.907436179Z caller=marker.go:178 msg="mark processor started" workers=150 delay=2h0m0s
level=info ts=2022-04-22T09:25:22.907435874Z caller=compactor.go:364 msg="applying retention with compaction"
level=info ts=2022-04-22T09:25:22.907506558Z caller=expiration.go:60 msg="overall smallest retention period 1647941122.907, default smallest retention period 1647941122.907"
level=info ts=2022-04-22T09:25:22.907678046Z caller=marker.go:203 msg="no marks file found"
level=error ts=2022-04-22T09:25:22.913813527Z caller=cached_client.go:64 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T09:25:22.913865946Z caller=compactor.go:370 msg="failed to run compaction" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "

I couldn't find any info about specifying sub directories in the documentation, so maybe it isn't supported?

And when testing via grafana:

level=info ts=2022-04-22T10:30:48.91822506Z caller=table_manager.go:176 msg="downloading all files for table loki_index_19104"
level=error ts=2022-04-22T10:30:48.95748629Z caller=cached_client.go:64 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.95755486Z caller=index_set.go:99 msg="failed to initialize table loki_index_19104, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.957572766Z caller=table.go:304 table-name=loki_index_19104 msg="failed to init user index set nonprod" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=info ts=2022-04-22T10:30:48.957589617Z caller=table.go:298 table-name=loki_index_19104 user-id=nonprod msg="downloaded index set at query time" duration=0.039070998
level=error ts=2022-04-22T10:30:48.957679414Z caller=chunk_store.go:526 org_id=nonprod msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.959024787Z caller=retry.go:73 org_id=nonprod msg="error processing request" try=0 err="rpc error: code = Code(500) desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: \n"
level=error ts=2022-04-22T10:30:48.96162146Z caller=table.go:167 table-name=loki_index_19104 org_id=nonprod msg="index set nonprod has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.961757132Z caller=chunk_store.go:526 org_id=nonprod msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.962199097Z caller=retry.go:73 org_id=nonprod msg="error processing request" try=1 err="rpc error: code = Code(500) desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: \n"
level=error ts=2022-04-22T10:30:48.969405822Z caller=cached_client.go:64 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.96947899Z caller=index_set.go:99 msg="failed to initialize table loki_index_19104, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.969502986Z caller=table.go:304 table-name=loki_index_19104 msg="failed to init user index set nonprod" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=info ts=2022-04-22T10:30:48.969519734Z caller=table.go:298 table-name=loki_index_19104 user-id=nonprod msg="downloaded index set at query time" duration=0.004768063
level=error ts=2022-04-22T10:30:48.969551774Z caller=chunk_store.go:526 org_id=nonprod msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.970120902Z caller=retry.go:73 org_id=nonprod msg="error processing request" try=2 err="rpc error: code = Code(500) desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: \n"
level=error ts=2022-04-22T10:30:48.973040583Z caller=table.go:167 table-name=loki_index_19104 org_id=nonprod msg="index set nonprod has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.973200476Z caller=chunk_store.go:526 org_id=nonprod msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:48.973643032Z caller=retry.go:73 org_id=nonprod msg="error processing request" try=3 err="rpc error: code = Code(500) desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: \n"
level=error ts=2022-04-22T10:30:49.000178409Z caller=cached_client.go:64 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:49.000238743Z caller=index_set.go:99 msg="failed to initialize table loki_index_19104, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:49.000257182Z caller=table.go:304 table-name=loki_index_19104 msg="failed to init user index set nonprod" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=info ts=2022-04-22T10:30:49.00026906Z caller=table.go:298 table-name=loki_index_19104 user-id=nonprod msg="downloaded index set at query time" duration=0.024211699
level=error ts=2022-04-22T10:30:49.00029344Z caller=chunk_store.go:526 org_id=nonprod msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:30:49.000746393Z caller=retry.go:73 org_id=nonprod msg="error processing request" try=4 err="rpc error: code = Code(500) desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: \n"

level=error ts=2022-04-22T10:34:10.005487822Z caller=index_set.go:99 msg="failed to initialize table loki_index_19104, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: "
level=error ts=2022-04-22T10:34:10.005583419Z caller=log.go:100 msg="error running loki" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: \nerror creating index client\ngithub.com/grafana/loki/pkg/storage/chunk/storage.NewStore\n\t/src/loki/pkg/storage/chunk/storage/factory.go:219\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:394\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:106\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:78\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:339\nmain.main\n\t/src/loki/cmd/loki/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:78\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:339\nmain.main\n\t/src/loki/cmd/loki/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

[nc-tamu]

Also the storage I use (HCP) does not support aws signature_version2.
EDIT: just found in the source code that v4 is used, so that part works

[ChrisSteinbach]

This is an issue we also have that is stopping us from updating. Same problem, same storage (HCP).

[GPF-IAG]

Yep, same problem for me using Loki2.5.0 and HCP.

[JensErat]

I seem to be observing the same issue with Ceph as backend, for a newly setup loki installation though (didn't work yet, but be a different issue). At least the stack trace is the same, though.

[slim-bean]

Sorry for the trouble here folks.

@sandeepsukhani could you take a look at this, #5909 (comment) I think that not all object stores do a list query exactly the same and we should probably change this to not error.

We've been doing a lot of changes in main, once we get things a little more stable in the next couple weeks we'll see if we can get an RC for a 2.5.1 which fixes this for folks to try 👍