From 9b02d4e29a20c23c5799c3b20028733c9ce148be Mon Sep 17 00:00:00 2001 From: ayaangazali Date: Sat, 29 Aug 2026 12:33:07 -0700 Subject: [PATCH 1/2] docs(kvbm): drop the object-storage tier that has no implementation The KVBM configuration reference documents seven DYN_KVBM_OBJECT_* variables and states that the G4 tier stores blocks in an S3-compatible object store, down to a {worker_id} bucket template and credential handling. Nothing in the runtime reads any of them. Their only appearances are the constants in environment_names.rs and this page. Someone following the page would supply S3 credentials and expect KV blocks to offload, and get silence: there is no reader to warn, so the tier just never does anything. Remove the section and the two intro claims that promise the tier, and note in its place that object storage has no environment surface yet, so the page does not simply lose the concept. Signed-off-by: ayaangazali --- .../components/kvbm-configuration.mdx | 40 +++---------------- 1 file changed, 6 insertions(+), 34 deletions(-) diff --git a/docs/fern/pages/reference/components/kvbm-configuration.mdx b/docs/fern/pages/reference/components/kvbm-configuration.mdx index 9942ab59cac2..a7c44d62b45f 100644 --- a/docs/fern/pages/reference/components/kvbm-configuration.mdx +++ b/docs/fern/pages/reference/components/kvbm-configuration.mdx @@ -7,14 +7,18 @@ subtitle: Field reference for the KV Block Manager environment variables and the The KV Block Manager (KVBM) is configured through two layers: -- **`DYN_KVBM_*` runtime environment variables** — the primary, user-facing controls for cache tier sizing, offload behavior, object storage, distributed (leader/worker) mode, and metrics. Set these in the worker container `env`. This is what most deployments use. +- **`DYN_KVBM_*` runtime environment variables** — the primary, user-facing controls for cache tier sizing, offload behavior, distributed (leader/worker) mode, and metrics. Set these in the worker container `env`. This is what most deployments use. - **The `KVBM_*` / TOML figment system** — a lower-level, layered configuration for the KVBM runtime (Tokio/Rayon threads, messenger transport, NixL backends, offload policies, event publishing). Used for advanced tuning and for vLLM's `kv_connector_extra_config`. Names and defaults on this page come from [`environment_names.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/config/environment_names.rs) and the [`kvbm-config`](https://github.com/ai-dynamo/dynamo/tree/main/lib/kvbm-config) crate. ## Cache tiers -KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3) → object storage (G4). Enable a tier by giving it a size; each tier accepts either a size in GB or an explicit block count (the block-count override wins). +KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3). Enable a tier by giving it a size; each tier accepts either a size in GB or an explicit block count (the block-count override wins). + + + Object storage (G4) has no runtime environment variables yet. The tier appears in the offload simulator and in in-flight work, but nothing in the runtime reads an object-storage variable today. + Set the CPU cache larger than the engine's GPU KV cache — a good rule of thumb is `DYN_KVBM_CPU_CACHE_GB >= 100` and `DYN_KVBM_DISK_CACHE_GB >= DYN_KVBM_CPU_CACHE_GB`. If the CPU cache is smaller than the device cache, KVBM churns (offloading blocks after every forward pass) and you may see performance *degradation*. Consult your engine's KV-cache configuration for the minimum useful value. @@ -80,38 +84,6 @@ KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3) → object stor Fall back to zero-filling when preallocating disk cache files is not supported. -## Object storage (G4) - -The G4 tier stores blocks in an S3-compatible object store. Enable it with `DYN_KVBM_OBJECT_ENABLED=1` and supply the bucket and credentials. - - - Enable object storage. Set to `1` to enable. - - - - Bucket name for the object-storage cache. Supports a `{worker_id}` template for per-worker buckets, for example `kv-cache-{worker_id}`. - - - - Endpoint URL for the object store. Set for S3-compatible services such as MinIO. - - - - Region for the object store. - - - - Access key for object-store authentication. - - - - Secret key for object-store authentication. - - - - Number of blocks to store in object storage. - - ## Distributed (leader/worker) In multi-worker deployments, one KVBM leader coordinates the worker cache tiers over ZMQ. From c210d31380894d98d1c8b1aba080a0ee61575938 Mon Sep 17 00:00:00 2001 From: ayaangazali Date: Sun, 30 Aug 2026 13:09:32 -0700 Subject: [PATCH 2/2] docs(kvbm): correct three leader and metrics defaults The reference lists these as unset when the code supplies a real default: DYN_KVBM_LEADER_ZMQ_HOST 127.0.0.1 (distributed/utils.rs DEFAULT_LEADER_ZMQ_HOST) DYN_KVBM_LEADER_ZMQ_ACK_PORT 56002 (distributed/utils.rs DEFAULT_LEADER_ZMQ_ACK_PORT) DYN_KVBM_METRICS_PORT 6880 (vllm/connector/leader.rs fallback) ACK_PORT was also typed as a string. It is a port, parsed as u16 and range checked like its PUB_PORT sibling, which the page already documents as an integer defaulting to 56001. Folded into this PR rather than opened separately: it is the same page and the same class of drift, and a third open PR against one reference file would just conflict with the other two. Signed-off-by: ayaangazali --- docs/fern/pages/reference/components/kvbm-configuration.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/fern/pages/reference/components/kvbm-configuration.mdx b/docs/fern/pages/reference/components/kvbm-configuration.mdx index a7c44d62b45f..f710f7d7bf5a 100644 --- a/docs/fern/pages/reference/components/kvbm-configuration.mdx +++ b/docs/fern/pages/reference/components/kvbm-configuration.mdx @@ -92,7 +92,7 @@ In multi-worker deployments, one KVBM leader coordinates the worker cache tiers Timeout in seconds for KVBM leader and worker initialization. Raise it for large multi-node deployments that take longer to come up. - + ZMQ host for the KVBM leader. @@ -100,7 +100,7 @@ In multi-worker deployments, one KVBM leader coordinates the worker cache tiers ZMQ publish port for the KVBM leader. The TRT-LLM integration derives its port from this value. - + ZMQ acknowledgment port for the KVBM leader. @@ -132,7 +132,7 @@ Enable individual NixL transfer backends with the `DYN_KVBM_NIXL_BACKEND_ - + Port for the KVBM metrics endpoint.