Loki Chunks Cache :: LRU Crawler efficiency

[jpadmin]

Describe the bug
I have installed Loki using helm chart version : 6.29.0 with the default settings and could see that it comes with loki chunks cache - memcached with LRU crawler. I can see that it works perfectly with higher cache hit ratio, but the problem is the LRU Crawler is not working and eviction is happening via direct_reclaims, ie through mandatory release due to the memory pressure.

Recall these variables from here.

direct_reclaims - Number of times worker threads had to directly pull LRU tails to find memory for a new item.
crawler_reclaimed - Number of items freed by the LRU Crawler.
evictions - Number of valid items removed from cache to free memory for new items
reclaimed - Number of times an entry was stored using memory from an expired entry

So with the helm chart default values, the LRU Crawler is not efficient, but we have chunksCache.defaultValidity to any non zero values say 1h(higher values in production) would help so that objects in memcache would be inserted with an expiry and then LRU Crawler can take care of it .

chunksCache:
  defaultValidity: 1h

To Reproduce
Steps to reproduce the behavior:

1. Deploy loki with chunksCache using default values
2. Watch the memory utilization of chunksCache
3. Watch the metrics - evictions, reclaimed, crawler_reclaimed and direct_reclaims

Expected behavior
LRU Crawler works efficiently and manages the items

Actual behavior
Cache grows till to the value of 8192M as per chunksCache.allocatedMemory with absolutely no changes in the items by LRU crawler despite of the crawler runs. Once it reaches the limit, it continues to evict items, but not then we might not have same cost for the new items as that of the initial items.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

STAT evictions 3570
STAT reclaimed 0
STAT crawler_reclaimed 0
STAT direct_reclaims 3570