diff --git a/docs/fern/pages/reference/components/kvbm-configuration.mdx b/docs/fern/pages/reference/components/kvbm-configuration.mdx index 9942ab59cac2..f710f7d7bf5a 100644 --- a/docs/fern/pages/reference/components/kvbm-configuration.mdx +++ b/docs/fern/pages/reference/components/kvbm-configuration.mdx @@ -7,14 +7,18 @@ subtitle: Field reference for the KV Block Manager environment variables and the The KV Block Manager (KVBM) is configured through two layers: -- **`DYN_KVBM_*` runtime environment variables** — the primary, user-facing controls for cache tier sizing, offload behavior, object storage, distributed (leader/worker) mode, and metrics. Set these in the worker container `env`. This is what most deployments use. +- **`DYN_KVBM_*` runtime environment variables** — the primary, user-facing controls for cache tier sizing, offload behavior, distributed (leader/worker) mode, and metrics. Set these in the worker container `env`. This is what most deployments use. - **The `KVBM_*` / TOML figment system** — a lower-level, layered configuration for the KVBM runtime (Tokio/Rayon threads, messenger transport, NixL backends, offload policies, event publishing). Used for advanced tuning and for vLLM's `kv_connector_extra_config`. Names and defaults on this page come from [`environment_names.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/config/environment_names.rs) and the [`kvbm-config`](https://github.com/ai-dynamo/dynamo/tree/main/lib/kvbm-config) crate. ## Cache tiers -KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3) → object storage (G4). Enable a tier by giving it a size; each tier accepts either a size in GB or an explicit block count (the block-count override wins). +KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3). Enable a tier by giving it a size; each tier accepts either a size in GB or an explicit block count (the block-count override wins). + + + Object storage (G4) has no runtime environment variables yet. The tier appears in the offload simulator and in in-flight work, but nothing in the runtime reads an object-storage variable today. + Set the CPU cache larger than the engine's GPU KV cache — a good rule of thumb is `DYN_KVBM_CPU_CACHE_GB >= 100` and `DYN_KVBM_DISK_CACHE_GB >= DYN_KVBM_CPU_CACHE_GB`. If the CPU cache is smaller than the device cache, KVBM churns (offloading blocks after every forward pass) and you may see performance *degradation*. Consult your engine's KV-cache configuration for the minimum useful value. @@ -80,38 +84,6 @@ KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3) → object stor Fall back to zero-filling when preallocating disk cache files is not supported. -## Object storage (G4) - -The G4 tier stores blocks in an S3-compatible object store. Enable it with `DYN_KVBM_OBJECT_ENABLED=1` and supply the bucket and credentials. - - - Enable object storage. Set to `1` to enable. - - - - Bucket name for the object-storage cache. Supports a `{worker_id}` template for per-worker buckets, for example `kv-cache-{worker_id}`. - - - - Endpoint URL for the object store. Set for S3-compatible services such as MinIO. - - - - Region for the object store. - - - - Access key for object-store authentication. - - - - Secret key for object-store authentication. - - - - Number of blocks to store in object storage. - - ## Distributed (leader/worker) In multi-worker deployments, one KVBM leader coordinates the worker cache tiers over ZMQ. @@ -120,7 +92,7 @@ In multi-worker deployments, one KVBM leader coordinates the worker cache tiers Timeout in seconds for KVBM leader and worker initialization. Raise it for large multi-node deployments that take longer to come up. - + ZMQ host for the KVBM leader. @@ -128,7 +100,7 @@ In multi-worker deployments, one KVBM leader coordinates the worker cache tiers ZMQ publish port for the KVBM leader. The TRT-LLM integration derives its port from this value. - + ZMQ acknowledgment port for the KVBM leader. @@ -160,7 +132,7 @@ Enable individual NixL transfer backends with the `DYN_KVBM_NIXL_BACKEND_ - + Port for the KVBM metrics endpoint.