How to share KV Cache across servers

[alejandrods]

I have four Mac Studios (512 GB each). My plan is to deploy the same model on each machine and configure a load balancer using Nginx to distribute the load across the servers. The KV cache is computed per server, but is there any way to share a common KV cache among all the servers instead of each one having its own?

I also tried deploying the models using RPC servers, but it penalizes the performance.

Thank you!