Skip to content

Commit 614d356

Browse files
committed
Add H200 sglang disagg configs from srtslurm
- Add dsr1-fp8-h200-dynamo-sglang config to nvidia-master.yaml - Include 1k1k configs: aggregated, low-latency (1P9D), high-throughput TEP/DEP (1P6D) - Include 8k1k configs: aggregated, TEP variants (1P7D, 1P6D, 1P3D, 2P3D), DEP (1P1D) - Add perf-changelog entry for new configuration - Document recipe registration process in AGENT.md
1 parent 4c6567f commit 614d356

3 files changed

Lines changed: 255 additions & 1 deletion

File tree

‎.github/configs/nvidia-master.yaml‎

Lines changed: 157 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1880,4 +1880,160 @@ gptoss-fp4-gb200-dynamo-trt:
18801880
- "DECODE_MAX_NUM_TOKENS=20000"
18811881
- "DECODE_MAX_BATCH_SIZE=512"
18821882
- "DECODE_GPU_MEM_FRACTION=0.9"
1883-
1883+
1884+
dsr1-fp8-h200-dynamo-sglang:
1885+
image: lmsysorg/sglang:v0.5.8-cu130-runtime
1886+
model: deepseek-ai/DeepSeek-R1-0528
1887+
model-prefix: dsr1
1888+
runner: h200-multinode-slurm
1889+
precision: fp8
1890+
framework: dynamo-sglang
1891+
multinode: true
1892+
disagg: true
1893+
seq-len-configs:
1894+
- isl: 1024
1895+
osl: 1024
1896+
search-space:
1897+
# Aggregated mode (single node TEP)
1898+
- conc-list: [1, 4, 16, 32, 64, 128, 256, 512]
1899+
prefill:
1900+
num-worker: 1
1901+
tp: 8
1902+
ep: 1
1903+
dp-attn: false
1904+
additional-settings:
1905+
- "CONFIG_FILE=recipes/h200/1k1k/bs128-agg-tp.yaml"
1906+
decode:
1907+
num-worker: 0
1908+
tp: 8
1909+
ep: 1
1910+
dp-attn: false
1911+
# Low latency (1 prefill, 9 decode, TEP)
1912+
- conc-list: [1, 4, 8, 16, 32, 64, 128, 256]
1913+
prefill:
1914+
num-worker: 1
1915+
tp: 8
1916+
ep: 1
1917+
dp-attn: false
1918+
additional-settings:
1919+
- "CONFIG_FILE=recipes/h200/1k1k/low-latency-1p9d.yaml"
1920+
decode:
1921+
num-worker: 9
1922+
tp: 8
1923+
ep: 1
1924+
dp-attn: false
1925+
# High throughput TEP (1 prefill, 6 decode)
1926+
- conc-list: [512, 1024, 2048]
1927+
prefill:
1928+
num-worker: 1
1929+
tp: 8
1930+
ep: 1
1931+
dp-attn: false
1932+
additional-settings:
1933+
- "CONFIG_FILE=recipes/h200/1k1k/bs256-1p6d-tp.yaml"
1934+
decode:
1935+
num-worker: 6
1936+
tp: 8
1937+
ep: 1
1938+
dp-attn: false
1939+
# High throughput DEP (1 prefill, 6 decode, dp-attention)
1940+
- conc-list: [128, 256, 512, 1024, 2048]
1941+
prefill:
1942+
num-worker: 1
1943+
tp: 8
1944+
ep: 8
1945+
dp-attn: true
1946+
additional-settings:
1947+
- "CONFIG_FILE=recipes/h200/1k1k/bs256-1p6d-dep.yaml"
1948+
decode:
1949+
num-worker: 6
1950+
tp: 8
1951+
ep: 8
1952+
dp-attn: true
1953+
- isl: 8192
1954+
osl: 1024
1955+
search-space:
1956+
# Aggregated mode (single node TEP)
1957+
- conc-list: [1, 4, 16, 32, 64, 128, 256]
1958+
prefill:
1959+
num-worker: 1
1960+
tp: 8
1961+
ep: 1
1962+
dp-attn: false
1963+
additional-settings:
1964+
- "CONFIG_FILE=recipes/h200/8k1k/bs128-agg-tp.yaml"
1965+
decode:
1966+
num-worker: 0
1967+
tp: 8
1968+
ep: 1
1969+
dp-attn: false
1970+
# Low latency TEP (1 prefill, 7 decode)
1971+
- conc-list: [1, 4, 8]
1972+
prefill:
1973+
num-worker: 1
1974+
tp: 8
1975+
ep: 1
1976+
dp-attn: false
1977+
additional-settings:
1978+
- "CONFIG_FILE=recipes/h200/8k1k/bs4-1p7d.yaml"
1979+
decode:
1980+
num-worker: 7
1981+
tp: 8
1982+
ep: 1
1983+
dp-attn: false
1984+
# TEP (1 prefill, 6 decode)
1985+
- conc-list: [4, 8, 16]
1986+
prefill:
1987+
num-worker: 1
1988+
tp: 8
1989+
ep: 1
1990+
dp-attn: false
1991+
additional-settings:
1992+
- "CONFIG_FILE=recipes/h200/8k1k/bs8-1p6d.yaml"
1993+
decode:
1994+
num-worker: 6
1995+
tp: 8
1996+
ep: 1
1997+
dp-attn: false
1998+
# TEP (1 prefill, 3 decode)
1999+
- conc-list: [8, 16, 32]
2000+
prefill:
2001+
num-worker: 1
2002+
tp: 8
2003+
ep: 1
2004+
dp-attn: false
2005+
additional-settings:
2006+
- "CONFIG_FILE=recipes/h200/8k1k/bs16-1p3d.yaml"
2007+
decode:
2008+
num-worker: 3
2009+
tp: 8
2010+
ep: 1
2011+
dp-attn: false
2012+
# TEP (2 prefill, 3 decode)
2013+
- conc-list: [32, 64, 128]
2014+
prefill:
2015+
num-worker: 2
2016+
tp: 8
2017+
ep: 1
2018+
dp-attn: false
2019+
additional-settings:
2020+
- "CONFIG_FILE=recipes/h200/8k1k/bs64-2p3d.yaml"
2021+
decode:
2022+
num-worker: 3
2023+
tp: 8
2024+
ep: 1
2025+
dp-attn: false
2026+
# High throughput DEP (1 prefill, 1 decode, dp-attention)
2027+
- conc-list: [64, 128, 256]
2028+
prefill:
2029+
num-worker: 1
2030+
tp: 8
2031+
ep: 1
2032+
dp-attn: false
2033+
additional-settings:
2034+
- "CONFIG_FILE=recipes/h200/8k1k/bs128-1p1d-dep.yaml"
2035+
decode:
2036+
num-worker: 1
2037+
tp: 8
2038+
ep: 8
2039+
dp-attn: true

‎AGENT.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,92 @@ When working with benchmark configurations, use these valid values:
179179
2. Create launcher script in `runners/` directory
180180
3. Update relevant master config with new runner type
181181

182+
### Registering Recipes from srtslurm
183+
184+
For disaggregated multi-node configurations (dynamo-sglang, dynamo-trt), recipes are stored in the external [srtslurm](https://github.com/ishandhanani/srt-slurm) repository. To stage these recipes in InferenceMAX:
185+
186+
**1. Locate source recipes in srtslurm:**
187+
```bash
188+
# Example: H200 sglang disagg recipes
189+
ls /path/to/srtslurm/recipes/h200/
190+
# 1k1k/ 8k1k/
191+
```
192+
193+
**2. Analyze recipe structure:**
194+
Each recipe YAML contains:
195+
- `name`: Recipe identifier
196+
- `model`: Model path/container info
197+
- `resources`: GPU type, prefill/decode node/worker counts
198+
- `backend.sglang_config`: Prefill and decode configuration (tp-size, dp-size, ep-size, dp-attention, etc.)
199+
- `benchmark`: ISL/OSL and concurrency settings
200+
201+
**3. Add config to nvidia-master.yaml:**
202+
```yaml
203+
dsr1-fp8-h200-dynamo-sglang:
204+
image: lmsysorg/sglang:v0.5.8-cu130-runtime
205+
model: deepseek-ai/DeepSeek-R1-0528
206+
model-prefix: dsr1
207+
runner: h200-multinode-slurm
208+
precision: fp8
209+
framework: dynamo-sglang
210+
multinode: true
211+
disagg: true
212+
seq-len-configs:
213+
- isl: 1024
214+
osl: 1024
215+
search-space:
216+
- conc-list: [1, 4, 16, 32, 64, 128, 256, 512]
217+
prefill:
218+
num-worker: 1
219+
tp: 8
220+
ep: 1
221+
dp-attn: false
222+
additional-settings:
223+
- "CONFIG_FILE=recipes/h200/1k1k/bs128-agg-tp.yaml"
224+
decode:
225+
num-worker: 0
226+
tp: 8
227+
ep: 1
228+
dp-attn: false
229+
```
230+
231+
**4. Key mapping from srtslurm to nvidia-master.yaml:**
232+
233+
| srtslurm field | nvidia-master.yaml field |
234+
|----------------|-------------------------|
235+
| `resources.prefill_workers` | `prefill.num-worker` |
236+
| `resources.decode_workers` | `decode.num-worker` |
237+
| `sglang_config.prefill.tp-size` | `prefill.tp` |
238+
| `sglang_config.prefill.ep-size` | `prefill.ep` |
239+
| `sglang_config.prefill.enable-dp-attention` | `prefill.dp-attn` |
240+
| `benchmark.concurrencies` (parsed) | `conc-list` |
241+
| Recipe file path | `additional-settings: CONFIG_FILE=...` |
242+
243+
**5. Common patterns:**
244+
- **Aggregated (AGG)**: Single node, `num-worker: 1` for prefill, `num-worker: 0` for decode
245+
- **TEP (Tensor-Expert Parallel)**: `dp-attn: false`, `ep: 1`
246+
- **DEP (Data-Expert Parallel)**: `dp-attn: true`, `ep: 8` (typically)
247+
- **Low latency**: More decode workers (e.g., 9), lower concurrencies
248+
- **High throughput**: Fewer decode workers, higher concurrencies
249+
250+
**6. Add perf-changelog entry:**
251+
```yaml
252+
- config-keys:
253+
- dsr1-fp8-h200-dynamo-sglang
254+
description:
255+
- "Add DSR1 FP8 H200 Dynamo SGLang disaggregated multinode configuration"
256+
- "Image: lmsysorg/sglang:v0.5.8-cu130-runtime"
257+
- "Recipes sourced from srtslurm repo (recipes/h200/)"
258+
pr-link: https://github.com/InferenceMAX/InferenceMAX/pull/XXX
259+
```
260+
261+
**7. Validate configuration:**
262+
```bash
263+
python utils/matrix_logic/generate_sweep_configs.py full-sweep \
264+
--master-config .github/configs/nvidia-master.yaml \
265+
--framework dynamo-sglang
266+
```
267+
182268
### Updating Docker Images
183269

184270
When upgrading Docker images in benchmark scripts and master configs .yaml:

‎perf-changelog.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,3 +269,15 @@
269269
- "Includes MTP and STP configurations for 1k1k and 8k1k sequence lengths"
270270
- "Concurrency levels: 4, 8, 16, 32, 64, 128, 256, 512"
271271
pr-link: https://github.com/InferenceMAX/InferenceMAX/pull/570
272+
273+
- config-keys:
274+
- dsr1-fp8-h200-dynamo-sglang
275+
description:
276+
- "Add DSR1 FP8 H200 Dynamo SGLang disaggregated multinode configuration"
277+
- "Image: lmsysorg/sglang:v0.5.8-cu130-runtime"
278+
- "Runner: h200-multinode-slurm with multinode and disagg enabled"
279+
- "Recipes sourced from srtslurm repo (recipes/h200/)"
280+
- "1k1k configs: aggregated, low-latency (1P9D), high-throughput TEP (1P6D), DEP (1P6D)"
281+
- "8k1k configs: aggregated, TEP configs (1P7D, 1P6D, 1P3D, 2P3D), DEP (1P1D)"
282+
- "Concurrency levels range from 1 to 2048 depending on configuration"
283+
pr-link: https://github.com/InferenceMAX/InferenceMAX/pull/TBD

0 commit comments

Comments
 (0)