failed to sync index set fake for table to s3 with prefix containing more than 2 delimiters

[WesselAtWork]

Describe the bug
I am setting path_prefix: index/ and prefix: v13/ this will result in a combined path of index/v13/table_num/name.tsdb.gz

This works fine.
I am able to operate and work with loki fine.

However, when loki tries to upload index/v13/table_num/fake it cannot do this because it errors out.

In the code you are explicitly checking for prefixes to only contain 2 delimited names.

loki/pkg/storage/stores/shipper/indexshipper/storage/cached_client.go

Lines 97 to 100 in e038d0f

	ss := strings.Split(prefix, delimiter)
	if len(ss) > 2 {
	return nil, nil, fmt.Errorf("invalid prefix %s", prefix)
	}

I couldn't find anything related to why exactly this needs to be a maximum of 2 delimited names.
The normal indexes seem to work fine, so why is the fake object throwing an error like this?

This can be worked around by simply setting to prefix: v13_ or some-such, but I like that the s3 prefixes are nicely formatted, when I need to go investigate an issue, and the indexes are nicely organised when I need to update to a new schema version.

It could also be that this subcomponent should not be enabled, if this is the case, can you point me to the relevant config block that would enable this subcomponent?

Found this from back in 2022, might be related.
#5909

I think this might be effecting retention. Not sure

To Reproduce
Set storage config to:

- from: "2024-01-01"
  index:
    path_prefix: index/
    period: 24h
    prefix: v13/
  object_store: s3
  schema: v13
  store: tsdb

Expected behavior
No errors are produced for "normal" config
Alternatively: the documentation updated to warn you that this is bad config

Environment:

• Infrastructure: Kubernetes (1.31)
• Deployment tool: helm (loki-6.29.0)
• Version: 3.4.2

Screenshots, Promtail config, or terminal output

level=info ts=2025-04-15T10:23:16.777879655Z caller=table_manager.go:239 index-store=tsdb-2024-01-01 msg="syncing tables"
level=error ts=2025-04-15T10:23:16.777921027Z caller=index_set.go:308 table-name=v13/20193 user-id=fake msg="sync failed, retrying it" err="invalid prefix v13/20193/fake"
level=error ts=2025-04-15T10:23:16.777930979Z caller=index_set.go:308 table-name=v13/20193 user-id=fake msg="sync failed, retrying it" err="invalid prefix v13/20193/fake"
level=error ts=2025-04-15T10:23:16.777942525Z caller=table_manager.go:137 index-store=tsdb-2024-01-01 msg="error syncing local index files with storage" err="failed to sync table 'v13/20193': failed to sync index set fake for table v13/20193: invalid prefix v13/20193/fake"
level=info ts=2025-04-15T10:23:16.77794855Z caller=table_manager.go:300 index-store=tsdb-2024-01-01 msg="query readiness setup completed" duration=1.311µs distinct_users_len=0 distinct_users=
level=info ts=2025-04-15T10:23:22.798664519Z caller=compactor.go:527 msg="compactor startup delay completed"

panic: runtime error: slice bounds out of range [-2:]

goroutine 859 [running]:
github.com/grafana/loki/v3/pkg/compactor/retention.ExtractIntervalFromTableName({0x400581f08b, 0x3})
    /src/loki/pkg/compactor/retention/util.go:50 +0x98
github.com/grafana/loki/v3/pkg/compactor.SortTablesByRange({0x4006386f00, 0x5, 0x8})
	/src/loki/pkg/compactor/compactor.go:891 +0x90
github.com/grafana/loki/v3/pkg/compactor.(*Compactor).RunCompaction(0x4000c29408, {0x40371f0, 0x400421d3b0}, 0x0)
	/src/loki/pkg/compactor/compactor.go:742 +0x278
github.com/grafana/loki/v3/pkg/compactor.(*Compactor).runCompactions(0x4000c29408, {0x40371f0, 0x400421d3b0})	
	/src/loki/pkg/compactor/compactor.go:533 +0x44
	
created by github.com/grafana/loki/v3/pkg/compactor.(*Compactor).loop in goroutine 488
	/src/loki/pkg/compactor/compactor.go:495 +0x8fc

[WesselAtWork]

Inspecting my logs, it might also be causing a knock on panic: runtime error: slice bounds out of range [-2:] but I can't understand how.
Loki can still function because the pods restart faster then the queries can be effected.

Edit: Updated the log context to include more details

[WesselAtWork]

Ok took a closer look at the function in question

loki/pkg/storage/stores/shipper/indexshipper/storage/cached_client.go

Lines 114 to 116 in e038d0f

	// user objects in a table were requested
	tableName := ss[0]
	userID := ss[1]

It requires there to be 2 identifieriers to set the tableName and the userID correctly.

this could be changed to:

 tableName := strings.Join(ss[:len(ss)-1], delimiter)  // or s[:strings.LastIndex(s, delimiter)]
 userID := ss[len(ss)-1] 

which should rebuild to:

tableName: "my/weird/table/name"
userID: = "id" 

but this of course assumes that the table name is used in such a way that assumes the delimiter is not present.

This all to say this is probably unintended behaviour.

Still seems to only effect this specific component, with general functionality being fine and the other tsdb indexes ending up in the right places.

[WesselAtWork]

I think this is unintended.

Updated my config to:

- from: "2024-01-01"
  index:
    path_prefix: index/v13/
    period: 24h
  object_store: s3
  schema: v13
  store: tsdb

seems to work fine, the indexes get migrated from table/db.gz into /table/fake/db.gz fine.