Skip to content

About

No description, website, or topics provided.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Latest commit

 

History

71 Commits

Folders and files

Repository files navigation

nano-dynamo

A small, from-scratch way to learn NVIDIA Dynamo's core architecture in under 750 lines of production Python: discovery, heartbeats, KV-aware routing, real vLLM inference, and frontend-orchestrated prefill/decode. Tests and docs are excluded from the count; production uses vLLM only.

Where nano-dynamo simplifies Dynamo

Area nano-dynamo simplification
Runtime and discovery In-memory FastAPI Registry plus heartbeat polling instead of Dynamo's distributed runtime and Kubernetes/etcd-backed discovery
Request transport Direct HTTP between components instead of Dynamo's broader TCP/HTTP/NATS request plane
Routing state Predicts cached prefixes from its own routing history instead of consuming backend KV events and maintaining Dynamo's richer global cache index
Routing policy Longest-prefix affinity, then in-flight load and round-robin, instead of Dynamo's tunable combined cache-overlap/load scoring
Inference backends vLLM only; Dynamo supports vLLM, SGLang, TensorRT-LLM, and extensible backends
P/D orchestration Synchronous vLLM/NIXL prefill-to-decode handoff without conditional disaggregation, topology constraints, or backend-specific orchestration
Operations Manual processes with no Operator, planner, autoscaling, draining, topology-aware placement, or production lifecycle management
Observability Application logs and inspectable in-memory state, without Dynamo's Prometheus metrics, OpenTelemetry tracing, dashboards, and active health checks
API surface Minimal chat request and plain-text stream rather than Dynamo's broader OpenAI-compatible and multimodal serving surface

These are deliberate teaching tradeoffs, not claims that the omitted systems are unnecessary. nano-dynamo keeps the request path small enough to read end to end; use NVIDIA Dynamo for production-scale deployment, resilience, observability, and optimization.

Request sequences

Aggregated mode

sequenceDiagram
    participant C as Client
    participant F as Frontend
    participant R as Registry
    participant W as Aggregated Worker
    C->>F: Chat completion
    F->>R: List model workers
    R-->>F: Live aggregated workers
    F->>W: Generate(prompt, predicted cache hit)
    W->>W: Prefill + decode
    W-->>F: Token stream
    F-->>C: Token stream
Loading

Prefix-aware cache routing

sequenceDiagram
    participant F as Frontend KV Router
    participant R as Registry
    participant W as Live Workers
    F->>R: List workers for model
    R-->>F: Worker IDs + load
    F->>F: Chain-hash prompt blocks
    F->>F: Find longest cached prefix
    F->>F: Break ties by load, then round-robin
    F-->>W: Select worker + cache_hit_blocks
Loading

Disaggregated P/D

sequenceDiagram
    participant C as Client
    participant F as Frontend
    participant P as Prefill Worker
    participant D as Decode Worker
    C->>F: Chat completion
    F->>P: Prefill(prompt, predicted cache hit)
    P-->>F: disaggregated_params
    F->>D: Generate(prompt, disaggregated_params)
    P-->>D: KV cache via NIXL
    D-->>F: Token stream
    F-->>C: Token stream
Loading

Prefix affinity selects aggregated workers or prefill workers; decode selection is load-based. The Frontend carries metadata, not KV-cache bytes.

Install

Python 3.11+ is required. On a controller-only host, install the base package; on a GPU worker, install the vLLM extra and NIXL:

git clone https://github.com/elizabetht/nano-dynamo
cd nano-dynamo
python -m venv .venv
source .venv/bin/activate
pip install -e .                 # controller
pip install -e ".[vllm]" nixl   # GPU worker

Aggregated serving

Run each command in its own shell:

# Registry
python -m nano_dynamo.registry.main

# Aggregated vLLM worker
WORKER_TYPE=aggregated \
WORKER_MODEL_NAME=Qwen/Qwen3-0.6B \
python -m nano_dynamo.worker.main

# Frontend
python -m nano_dynamo.frontend.main

Disaggregated serving

Use the same model and compatible vLLM settings on both GPU workers. Replace these example addresses with your own topology:

Example Replace with
192.168.1.75 Controller address where the Registry listens
192.168.1.76 Prefill worker's HTTP/service address
192.168.1.77 Decode worker's HTTP/service address
192.168.100.10 Prefill GPU's routable NIXL/interconnect address
192.168.100.11 Decode GPU's routable NIXL/interconnect address

The 192.168.1.x network carries Registry and worker HTTP traffic; the 192.168.100.x network carries the cross-GPU KV path. They may be the same network in your lab, but NIXL must not advertise localhost across hosts.

# Prefill GPU
WORKER_TYPE=prefill \
WORKER_MODEL_NAME=Qwen/Qwen3-0.6B \
WORKER_GPU_MEMORY_UTILIZATION=0.85 \
WORKER_ENDPOINT_URL=http://192.168.1.76:8001 \
REGISTRY_URL=http://192.168.1.75:8000 \
VLLM_NIXL_SIDE_CHANNEL_HOST=192.168.100.10 \
python -m nano_dynamo.worker.main

# Decode GPU
WORKER_TYPE=decode \
WORKER_MODEL_NAME=Qwen/Qwen3-0.6B \
WORKER_GPU_MEMORY_UTILIZATION=0.85 \
WORKER_ENDPOINT_URL=http://192.168.1.77:8001 \
REGISTRY_URL=http://192.168.1.75:8000 \
VLLM_NIXL_SIDE_CHANNEL_HOST=192.168.100.11 \
python -m nano_dynamo.worker.main

The P/D control path is tested; end-to-end two-host verification is pending.

Send a request

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Say hello in one sentence."}]}'

Test (does not initialize vLLM)

pip install -e ".[dev]"
pytest -q

Design and evidence

P/D design · verified NIXL contract · Frontend · Worker

About

No description, website, or topics provided.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages