A small, from-scratch way to learn NVIDIA Dynamo's core architecture in under 750 lines of production Python: discovery, heartbeats, KV-aware routing, real vLLM inference, and frontend-orchestrated prefill/decode. Tests and docs are excluded from the count; production uses vLLM only.
| Area | nano-dynamo simplification |
|---|---|
| Runtime and discovery | In-memory FastAPI Registry plus heartbeat polling instead of Dynamo's distributed runtime and Kubernetes/etcd-backed discovery |
| Request transport | Direct HTTP between components instead of Dynamo's broader TCP/HTTP/NATS request plane |
| Routing state | Predicts cached prefixes from its own routing history instead of consuming backend KV events and maintaining Dynamo's richer global cache index |
| Routing policy | Longest-prefix affinity, then in-flight load and round-robin, instead of Dynamo's tunable combined cache-overlap/load scoring |
| Inference backends | vLLM only; Dynamo supports vLLM, SGLang, TensorRT-LLM, and extensible backends |
| P/D orchestration | Synchronous vLLM/NIXL prefill-to-decode handoff without conditional disaggregation, topology constraints, or backend-specific orchestration |
| Operations | Manual processes with no Operator, planner, autoscaling, draining, topology-aware placement, or production lifecycle management |
| Observability | Application logs and inspectable in-memory state, without Dynamo's Prometheus metrics, OpenTelemetry tracing, dashboards, and active health checks |
| API surface | Minimal chat request and plain-text stream rather than Dynamo's broader OpenAI-compatible and multimodal serving surface |
These are deliberate teaching tradeoffs, not claims that the omitted systems are unnecessary. nano-dynamo keeps the request path small enough to read end to end; use NVIDIA Dynamo for production-scale deployment, resilience, observability, and optimization.
sequenceDiagram
participant C as Client
participant F as Frontend
participant R as Registry
participant W as Aggregated Worker
C->>F: Chat completion
F->>R: List model workers
R-->>F: Live aggregated workers
F->>W: Generate(prompt, predicted cache hit)
W->>W: Prefill + decode
W-->>F: Token stream
F-->>C: Token stream
sequenceDiagram
participant F as Frontend KV Router
participant R as Registry
participant W as Live Workers
F->>R: List workers for model
R-->>F: Worker IDs + load
F->>F: Chain-hash prompt blocks
F->>F: Find longest cached prefix
F->>F: Break ties by load, then round-robin
F-->>W: Select worker + cache_hit_blocks
sequenceDiagram
participant C as Client
participant F as Frontend
participant P as Prefill Worker
participant D as Decode Worker
C->>F: Chat completion
F->>P: Prefill(prompt, predicted cache hit)
P-->>F: disaggregated_params
F->>D: Generate(prompt, disaggregated_params)
P-->>D: KV cache via NIXL
D-->>F: Token stream
F-->>C: Token stream
Prefix affinity selects aggregated workers or prefill workers; decode selection is load-based. The Frontend carries metadata, not KV-cache bytes.
Python 3.11+ is required. On a controller-only host, install the base package; on a GPU worker, install the vLLM extra and NIXL:
git clone https://github.com/elizabetht/nano-dynamo
cd nano-dynamo
python -m venv .venv
source .venv/bin/activate
pip install -e . # controller
pip install -e ".[vllm]" nixl # GPU workerRun each command in its own shell:
# Registry
python -m nano_dynamo.registry.main
# Aggregated vLLM worker
WORKER_TYPE=aggregated \
WORKER_MODEL_NAME=Qwen/Qwen3-0.6B \
python -m nano_dynamo.worker.main
# Frontend
python -m nano_dynamo.frontend.mainUse the same model and compatible vLLM settings on both GPU workers. Replace these example addresses with your own topology:
| Example | Replace with |
|---|---|
192.168.1.75 |
Controller address where the Registry listens |
192.168.1.76 |
Prefill worker's HTTP/service address |
192.168.1.77 |
Decode worker's HTTP/service address |
192.168.100.10 |
Prefill GPU's routable NIXL/interconnect address |
192.168.100.11 |
Decode GPU's routable NIXL/interconnect address |
The 192.168.1.x network carries Registry and worker HTTP traffic; the
192.168.100.x network carries the cross-GPU KV path. They may be the same
network in your lab, but NIXL must not advertise localhost across hosts.
# Prefill GPU
WORKER_TYPE=prefill \
WORKER_MODEL_NAME=Qwen/Qwen3-0.6B \
WORKER_GPU_MEMORY_UTILIZATION=0.85 \
WORKER_ENDPOINT_URL=http://192.168.1.76:8001 \
REGISTRY_URL=http://192.168.1.75:8000 \
VLLM_NIXL_SIDE_CHANNEL_HOST=192.168.100.10 \
python -m nano_dynamo.worker.main
# Decode GPU
WORKER_TYPE=decode \
WORKER_MODEL_NAME=Qwen/Qwen3-0.6B \
WORKER_GPU_MEMORY_UTILIZATION=0.85 \
WORKER_ENDPOINT_URL=http://192.168.1.77:8001 \
REGISTRY_URL=http://192.168.1.75:8000 \
VLLM_NIXL_SIDE_CHANNEL_HOST=192.168.100.11 \
python -m nano_dynamo.worker.mainThe P/D control path is tested; end-to-end two-host verification is pending.
curl -N http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Say hello in one sentence."}]}'pip install -e ".[dev]"
pytest -q