Repository navigation
Conversation
…ntity Without ais_perf_model, the Planner built its cold fpm_regression model with system "unprofiled" and default parallel widths of 1, whatever the deployment. A state snapshot saved from it therefore claimed facts the Planner never observed, and AISimulate could not use it to time a recommendation role. The cold identity now records what the worker reports (model, backend, KV block size, MTP depth) and, as deployment_limits, the GPUs per worker, scheduler limits, KV capacity and context length. Facts the Planner cannot observe are AISimulate's "unknown": hardware, backend version, quantization, attention backend, and the parallel split of a worker with more than one GPU (one GPU fixes every width to 1). The legacy-options migration that supplies estimator defaults keeps native placeholders, since only its estimator controls are used. Requires an AISimulate nightly that accepts "unknown" identity facts and deployment_limits (ai-dynamo/aisim#658 and its stack). Signed-off-by: hongkuanz <hongkuanz@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
❌ Dynamo PR CI failed — run 41153898441 (attempt 1) on
|
| Other | Jobs |
|---|---|
| dynamo-runtime | ❌ 4 ✅ 3 |
| planner | ❌ 2 ✅ 3 |
Failure details
6 jobs failed, all from one error: ForwardPassPerfModelConfig.__init__() rejects the deployment_limits keyword that dynamo/planner/core/perf_model/ais_adapter.py:300 (_build_ais_config) passes to it. Planner unit/integration tests fail directly, and the replay aisimulate CLI e2e tests fail in a subprocess. There are no infrastructure signatures and no cancelled jobs.
❌ planner / CPU Test cpu, arm64: TypeError: unexpected keyword argument 'deployment_limits'
Job: planner / CPU Test cpu, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 124398266810 -R ai-dynamo/dynamo --log-failed
components/src/dynamo/planner/core/state_machine.py:110: in __init__
self._prefill_regression = PlannerEnginePerfModel(
components/src/dynamo/planner/core/perf_model/ais_adapter.py:137: in __init__
self._init_engine_model()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:154: in _init_engine_model
model_key = self._model_key()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:397: in _model_key
json.dumps(self._build_ais_config(), sort_keys=True, separators=(",", ":")),
components/src/dynamo/planner/core/perf_model/ais_adapter.py:300: in _build_ais_config
ForwardPassPerfModelConfig(
E TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
=========== 82 failed, 2747 passed, 24 skipped in 635.62s (0:10:35) ============
82 tests fail with the same TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits', raised from PlannerEnginePerfModel._build_ais_config (ais_adapter.py:300) whenever a planner perf model is built. Affected files: test_engine_adapter.py (44), test_reconfiguration.py (18), test_ais_perf_config.py (5), test_signal_health.py (3), test_pending_startup_scaling.py, test_retune_engine_adapter.py, test_simulation_integration.py (2 each), plus single tests in test_ais_native_estimator.py, test_replay_adapter_fpm.py and test_replay_retune.py; the other 3 failures are the test_unified_cli_dynamo.py aisimulate CLI cases, which hit the same error in a subprocess.
❌ planner / CPU Test cpu, amd64: TypeError: unexpected keyword argument 'deployment_limits'
Job: planner / CPU Test cpu, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 124398266854 -R ai-dynamo/dynamo --log-failed
components/src/dynamo/planner/core/state_machine.py:110: in __init__
self._prefill_regression = PlannerEnginePerfModel(
components/src/dynamo/planner/core/perf_model/ais_adapter.py:137: in __init__
self._init_engine_model()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:154: in _init_engine_model
model_key = self._model_key()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:397: in _model_key
json.dumps(self._build_ais_config(), sort_keys=True, separators=(",", ":")),
components/src/dynamo/planner/core/perf_model/ais_adapter.py:300: in _build_ais_config
ForwardPassPerfModelConfig(
E TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
=========== 82 failed, 2747 passed, 24 skipped in 490.20s (0:08:10) ============
The same 82 failures as the arm64 job, all from the same deployment_limits TypeError in ais_adapter.py:300.
❌ dynamo-runtime / test / sequential cuda13.0, amd64: TypeError: unexpected keyword argument 'deployment_limits'
Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 124399138051 -R ai-dynamo/dynamo --log-failed
components/src/dynamo/planner/core/state_machine.py:110: in __init__
self._prefill_regression = PlannerEnginePerfModel(
components/src/dynamo/planner/core/perf_model/ais_adapter.py:137: in __init__
self._init_engine_model()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:154: in _init_engine_model
model_key = self._model_key()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:397: in _model_key
json.dumps(self._build_ais_config(), sort_keys=True, separators=(",", ":")),
components/src/dynamo/planner/core/perf_model/ais_adapter.py:300: in _build_ais_config
ForwardPassPerfModelConfig(
E TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
== 80 failed, 6287 passed, 158 skipped, 2814 deselected in 1545.09s (0:25:45) ==
80 planner/replay tests fail with the same TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits' from ais_adapter.py:300: the planner test files listed above, plus components/src/dynamo/replay/tests/test_retune_replay.py (1).
❌ dynamo-runtime / test / sequential cuda13.0, arm64: TypeError: unexpected keyword argument 'deployment_limits'
Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 124399138078 -R ai-dynamo/dynamo --log-failed
components/src/dynamo/planner/core/state_machine.py:110: in __init__
self._prefill_regression = PlannerEnginePerfModel(
components/src/dynamo/planner/core/perf_model/ais_adapter.py:137: in __init__
self._init_engine_model()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:154: in _init_engine_model
model_key = self._model_key()
components/src/dynamo/planner/core/perf_model/ais_adapter.py:397: in _model_key
json.dumps(self._build_ais_config(), sort_keys=True, separators=(",", ":")),
components/src/dynamo/planner/core/perf_model/ais_adapter.py:300: in _build_ais_config
ForwardPassPerfModelConfig(
E TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
== 80 failed, 6287 passed, 158 skipped, 2814 deselected in 2037.86s (0:33:57) ==
The same 80 failures as the amd64 sequential job, all from the same deployment_limits TypeError.
❌ dynamo-runtime / test / parallel cuda13.0, amd64: TypeError: unexpected keyword argument 'deployment_limits'
Job: dynamo-runtime / test / parallel cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 124399138058 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_predict_cli_cases[08-synthetic-throughput-planner] - AssertionError: aisimulate predict --stack dynamo ... failed with 1
aisimulate predict failed: TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
FAILED components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[04-planner-preset-off-goodput] - AssertionError: aisimulate recommend --stack dynamo ... failed with 1
File "/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/planner/core/perf_model/ais_adapter.py", line 300, in _build_ais_config
ForwardPassPerfModelConfig(
TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
FAILED components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[06-disagg-router-planner-goodput] - AssertionError: aisimulate recommend --stack dynamo ... failed with 1
= 3 failed, 1378 passed, 33 skipped, 7925 deselected, 7 subtests passed in 1125.61s (0:18:45) =
3 test_unified_cli_dynamo.py cases fail because the aisimulate predict/recommend --stack dynamo subprocess exits with code 1. Its stderr shows the same deployment_limits TypeError from the planner's ais_adapter.py:300.
❌ dynamo-runtime / test / parallel cuda13.0, arm64: TypeError: unexpected keyword argument 'deployment_limits'
Job: dynamo-runtime / test / parallel cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 124399138062 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_predict_cli_cases[08-synthetic-throughput-planner] - AssertionError: aisimulate predict --stack dynamo ... failed with 1
aisimulate predict failed: TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
FAILED components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[04-planner-preset-off-goodput] - AssertionError: aisimulate recommend --stack dynamo ... failed with 1
File "/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/planner/core/perf_model/ais_adapter.py", line 300, in _build_ais_config
ForwardPassPerfModelConfig(
TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'
FAILED components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[06-disagg-router-planner-goodput] - AssertionError: aisimulate recommend --stack dynamo ... failed with 1
= 3 failed, 1378 passed, 33 skipped, 7925 deselected, 7 subtests passed in 1620.44s (0:27:00) =
The same 3 aisimulate CLI cases as the amd64 parallel job, with the same deployment_limits TypeError in the subprocess stderr.
For agents
{
"pr": 17743,
"run_id": 41153898441,
"run_attempt": 1,
"head_sha": "39f0a5dff4f37d30543df94d656cf3ebd04a88dd",
"failures": [
{
"job": "planner / CPU Test cpu, arm64",
"job_id": 124398266810,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'",
"tests": [
"components/src/dynamo/planner/tests/plugins/orchestrator/test_engine_adapter.py (44 tests)",
"components/src/dynamo/planner/tests/unit/test_reconfiguration.py (18 tests)",
"components/src/dynamo/planner/tests/unit/test_ais_perf_config.py (5 tests)",
"components/src/dynamo/planner/tests/unit/test_signal_health.py (3 tests)",
"components/src/dynamo/replay/tests/test_simulation_integration.py (2 tests)",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_predict_cli_cases[08-synthetic-throughput-planner]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[04-planner-preset-off-goodput]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[06-disagg-router-planner-goodput]"
],
"log_cmd": "gh run view --job 124398266810 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "planner / CPU Test cpu, amd64",
"job_id": 124398266854,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'",
"tests": [
"components/src/dynamo/planner/tests/plugins/orchestrator/test_engine_adapter.py (44 tests)",
"components/src/dynamo/planner/tests/unit/test_reconfiguration.py (18 tests)",
"components/src/dynamo/planner/tests/unit/test_ais_perf_config.py (5 tests)",
"components/src/dynamo/planner/tests/unit/test_signal_health.py (3 tests)",
"components/src/dynamo/replay/tests/test_simulation_integration.py (2 tests)",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_predict_cli_cases[08-synthetic-throughput-planner]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[04-planner-preset-off-goodput]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[06-disagg-router-planner-goodput]"
],
"log_cmd": "gh run view --job 124398266854 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "dynamo-runtime / test / sequential cuda13.0, amd64",
"job_id": 124399138051,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'",
"tests": [
"components/src/dynamo/planner/tests/plugins/orchestrator/test_engine_adapter.py (44 tests)",
"components/src/dynamo/planner/tests/unit/test_reconfiguration.py (18 tests)",
"components/src/dynamo/planner/tests/unit/test_ais_perf_config.py (5 tests)",
"components/src/dynamo/planner/tests/unit/test_signal_health.py (3 tests)",
"components/src/dynamo/replay/tests/test_simulation_integration.py (2 tests)",
"components/src/dynamo/replay/tests/test_retune_replay.py"
],
"log_cmd": "gh run view --job 124399138051 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "dynamo-runtime / test / sequential cuda13.0, arm64",
"job_id": 124399138078,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'",
"tests": [
"components/src/dynamo/planner/tests/plugins/orchestrator/test_engine_adapter.py (44 tests)",
"components/src/dynamo/planner/tests/unit/test_reconfiguration.py (18 tests)",
"components/src/dynamo/planner/tests/unit/test_ais_perf_config.py (5 tests)",
"components/src/dynamo/planner/tests/unit/test_signal_health.py (3 tests)",
"components/src/dynamo/replay/tests/test_simulation_integration.py (2 tests)",
"components/src/dynamo/replay/tests/test_retune_replay.py"
],
"log_cmd": "gh run view --job 124399138078 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "dynamo-runtime / test / parallel cuda13.0, amd64",
"job_id": 124399138058,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'",
"tests": [
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_predict_cli_cases[08-synthetic-throughput-planner]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[04-planner-preset-off-goodput]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[06-disagg-router-planner-goodput]"
],
"log_cmd": "gh run view --job 124399138058 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "dynamo-runtime / test / parallel cuda13.0, arm64",
"job_id": 124399138062,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "TypeError: ForwardPassPerfModelConfig.__init__() got an unexpected keyword argument 'deployment_limits'",
"tests": [
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_predict_cli_cases[08-synthetic-throughput-planner]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[04-planner-preset-off-goodput]",
"components/src/dynamo/replay/tests/e2e/test_unified_cli_dynamo.py::test_dynamo_recommend_cli_cases_round_trip[06-disagg-router-planner-goodput]"
],
"log_cmd": "gh run view --job 124399138062 -R ai-dynamo/dynamo --log-failed"
}
]
}Posted automatically by Devin for run 41153898441. Updated on every full-CI run of this PR.
Overview
This PR is a draft until an AISimulate nightly carries the stack ai-dynamo/aisim#669 → ai-dynamo/aisim#670 → ai-dynamo/aisim#671 → ai-dynamo/aisim#658. Once that nightly exists, this PR will bump the AISimulate pin and become ready.
Without
ais_perf_model, the Planner builds a coldfpm_regressionmodel. Its identity usedsystem: "unprofiled"and parallel widths of 1, whatever the deployment. A state snapshot saved from that model therefore claimed facts the Planner never observed, and AISimulate could not use it to time arecommend/predictrole.The AISimulate stack adds three things:
"unknown"identity facts and adeployment_limitsblock;This PR makes the Planner record its cold identity in that format.
Key Changes
components/src/dynamo/planner/core/perf_model/ais_adapter.py,_build_ais_config(cold branch only; a configuredais_perf_modelis unchanged):model:model_name, or"unknown";backend;kv_block_sizeandnextn, as before;fallback_policy: deny, now explicit;deployment_limits, fromEngineCapabilities:num_gpus,max_num_batched_tokens,max_num_seqs,max_kv_tokens,context_length. A missing or non-positive value becomes"unknown"."unknown":systemandbackend_version;attention_backend;tp,pp,attention_dp,moe_tp_sizeandmoe_ep_size, for a worker with more than one GPU. A one-GPU worker records every width as 1.estimator_configis used.Validation
All runs used
python3 -m pytest, withPYTHONPATHset to this worktree'scomponents/srcand to the AISimulate stack head (ai-dynamo/aisimulate#658at9f233acf):components/src/dynamo/planner/tests/unit/test_ais_perf_config.pycomponents/src/dynamo/planner/tests/unitThe new tests check two things:
pre-commit run --files <changed files>anddocs/fern/scripts/docs_lint.pypass.This PR is not validated against the currently pinned AISimulate. It needs the nightly above, which is why it is a draft.
Issues and Ongoing Work
ais_adapter.py; whichever merges second rebases.EngineCapabilities.max_kv_tokens(total_kv_blocks × kv_cache_block_sizefrom the worker's MDC) is the KV capacity of one attention-DP rank, which is how AISimulate readsdeployment_limits.max_kv_tokens.Pull Request Checklist
/pr-checklistskill.🤖 Generated with Claude Code