Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/deployment-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Sampler replicas scale between `inference_min_replicas` and `inference_max_repli

We have certain configs with speculative decoding using public DFlash checkpoints, which we finds increases rollout tps by 2-3x depending on workload. Note that this throughput gain is dependent on accept length which can differ based on your training domain, # of steps in RL, and sampling params.

The following 32K LoRA recipes enable speculative decoding by default:
The following LoRA recipes enable speculative decoding by default:

| Recipe | Method | Draft checkpoint |
| --- | --- | --- |
Expand All @@ -76,6 +76,7 @@ The following 32K LoRA recipes enable speculative decoding by default:
| `qwen35-9b-instruct-lora-32k` | DFlash | `modal-labs/Qwen3.5-9B-DFlash` |
| `qwen38-27b-lora-32k` | DFlash | `incoai/Qwen3.8-27B-DFlash2` |
| `glm53-flash-lora-32k` | DFlash | `modal-labs/GLM-5.3-Flash-DFlash` |
| `nemotron35-lightning-30b-a3b-lora-64k` | Native MTP (`NEXTN`) | Reuses the target checkpoint |

Deploy a recipe by passing its Python path to `spindle deploy`, for example:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,22 @@ class Config(Parent):
trainer_memory_mib = 262144
trainer_max_clients_per_instance = 4
inference_max_replicas = 2
inference_target_concurrency = 128
inference_target_concurrency = 89
overrides = {
"miles_cfg.expert_model_parallel_size": 2,
"miles_cfg.max_lora_slots": 4,
"miles_cfg.cli_options.log_probs_chunk_size": 4096,
"sglang_cfg.max_loras_per_batch": 4,
"sglang_cfg.max_loaded_loras": 32,
"sglang_cfg.max_mamba_cache_size": 640,
"sglang_cfg.max_mamba_cache_size": 448,
"sglang_cfg.mem_fraction_static": 0.9,
"sglang_cfg.prefill_decode_interval": 16,
"sglang_cfg.prefill_decode_interval_min_bs": 32,
"sglang_cfg.lora_use_virtual_experts": True,
"sglang_cfg.speculative_algorithm": "NEXTN",
"sglang_cfg.speculative_num_steps": 3,
"sglang_cfg.speculative_eagle_topk": 1,
"sglang_cfg.speculative_num_draft_tokens": 4,
}


Expand Down
Loading