Repository navigation
fix(multimodal): keep invalid-audio errors as client errors in batch loading - #17937
Conversation
load_audio_batch folded a ValueError from load_audio (empty or undecodable audio) into a generic Exception. The Python bindings map ValueError to InvalidArgument and a generic Exception to Unknown, so bad client audio was reported as a server error. Keep the first ValueError and raise it after the typed HTTP, URL-policy and missing-decoder errors and before the generic aggregate, as the image and video batch loaders already do. Refs DYN-5237. Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
|
👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
factory: Initial validation snapshot. Validation status: complete
The live check that corrupt audio returns HTTP 400 from a vLLM worker was not run: it needs a GPU and the vLLM engine, and the sandbox was CPU-only. Detailed evidence — validation completeLeft out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1, passed), and lint and formatting with Evidence tally: [3/3 validated] Advisory AI assessment: sound. No findings. Validation result: pass — all 3 new test cases fail on Evidence audit: complete [3/3 validated] — the command report comes from recorded runs. Commands and results [3/3 validated]Generated from the commands recorded during this run. Check 1Builds the changed Dynamo source and confirms that Python can import its compiled extension. Result: Passed ( Command: Not shown because the exact command contained private run data. Check 2Checks the changed files with the repository's fast lint and formatting commands. Result: Passed ( Command: bash -c 'python3 -m py_compile components/src/dynamo/common/multimodal/audio_loader.py components/src/dynamo/common/tests/multimodal/test_audio_loader.py && echo py_compile OK && pre-commit run --files components/src/dynamo/common/multimodal/audio_loader.py components/src/dynamo/common/tests/multimodal/test_audio_loader.py'Check 3Runs the relevant Python unit tests without requiring a GPU. Result: Passed ( Command: /opt/dynamo/venv/bin/python -m pytest components/src/dynamo/common/tests/multimodal/test_audio_loader.py components/src/dynamo/common/tests/multimodal/test_video_loader.py components/src/dynamo/common/tests/multimodal/test_image_loader.py -v --tb=short -p no:cacheprovider |
|
No description provided. |
|
factory: @dynamo-ops please run full CI for 83f8bb2 |
|
/ok to test 83f8bb2 |
WalkthroughAudioLoader.load_audio_batch now retains the first ValueError from failed audio loads. After checking for specific HTTP, URL validation, and missing decoder errors, it raises the retained ValueError before the generic aggregated exception. Tests cover decoder failures, empty waveforms, and mixed batch failures. ChangesAudio batch error handling
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk:
|
…ient errors load_audio turned fetch timeouts, connection errors and a missing vLLM into ValueError, so the batch loader reported them as bad requests. Keep their types so only invalid audio is a client error. Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
|
factory: @dynamo-ops please run full CI for ab2ac40 |
|
/ok to test ab2ac40 |
❌ Dynamo PR CI failed — run 42127806048 (attempt 1) on
|
| Other | Jobs |
|---|---|
| dynamo-runtime | ❌ 1 ✅ 6 |
Failure details
1 job failed: all 24 cases in tests/fault_tolerance/test_backend_admission_gate.py fail during shared etcd fixture setup because the server exits before port 2380 is ready.
❌ dynamo-runtime / test / parallel cuda13.0, arm64: shared etcd exits before port 2380 is ready
Job: dynamo-runtime / test / parallel cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 127602538473 -R ai-dynamo/dynamo --log-failed
tests/conftest.py:1254: in runtime_services_session
with SharedEtcdServer(request, tmp_path_factory) as etcd:
tests/conftest.py:1059: in __enter__
self._server.__enter__()
...
tests/utils/managed_process.py:675: in _check_process_alive
raise RuntimeError(
E RuntimeError: Main server process exited with code 1 while waiting for port 2380
...
FAILED tests/fault_tolerance/test_backend_admission_gate.py::test_waiting_limit_holds_requests_until_a_first_response[tcp] - RuntimeError: Main server process exited with code 1 while waiting for port 2380
test_waiting_limit_holds_requests_until_a_first_response[tcp] and the other 23 TCP/NATS admission-gate cases all fail in runtime_services_session: its shared EtcdServer exits with code 1 while waiting for client port 2380. This is a shared test-service startup failure, not 24 independent assertion failures; the job log does not include etcd's own exit reason.
For agents
{
"pr": 17937,
"run_id": 42127806048,
"run_attempt": 1,
"head_sha": "0470aca09c3c99944145f554363f2bfa14fc563d",
"failures": [
{
"job": "dynamo-runtime / test / parallel cuda13.0, arm64",
"job_id": 127602538473,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "Shared etcd exits with code 1 before port 2380 is ready",
"tests": [
"tests/fault_tolerance/test_backend_admission_gate.py::test_waiting_limit_holds_requests_until_a_first_response[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_waiting_limit_holds_requests_until_a_first_response[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_request_limit_holds_requests_until_one_finishes[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_request_limit_holds_requests_until_one_finishes[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[defaults-tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[defaults-nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[zero-queue-limit-tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[zero-queue-limit-nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_queue_limit_without_an_engine_limit_fails_worker_startup[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_queue_limit_without_an_engine_limit_fails_worker_startup[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_legacy_aliases_configure_the_request_and_queue_limits[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_legacy_aliases_configure_the_request_and_queue_limits[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_the_former_waiting_limit_name_configures_the_waiting_limit[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_the_former_waiting_limit_name_configures_the_waiting_limit[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_ranks_admit_and_queue_independently_and_share_the_aggregate[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_ranks_admit_and_queue_independently_and_share_the_aggregate[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_full_rank_refuses_for_itself_and_its_idle_sibling_is_admitted_at_once[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_full_rank_refuses_for_itself_and_its_idle_sibling_is_admitted_at_once[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_workers_in_one_process_admit_through_their_own_gates[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_workers_in_one_process_admit_through_their_own_gates[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_caller_disconnecting_while_queued_frees_its_place_before_admission[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_caller_disconnecting_while_queued_frees_its_place_before_admission[nats]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_failed_worker_is_still_reported_down[tcp]",
"tests/fault_tolerance/test_backend_admission_gate.py::test_a_failed_worker_is_still_reported_down[nats]"
],
"log_cmd": "gh run view --job 127602538473 -R ai-dynamo/dynamo --log-failed"
}
]
}Posted automatically by Devin for run 42127806048. Updated on every full-CI run of this PR.
furionw
left a comment
There was a problem hiding this comment.
One correctness issue: the new batch ValueError propagation also classifies host resource exhaustion as invalid client input. See the inline comment and before/after reproduction.
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: Added the missing test docstrings in commit |
|
nursery: @dynamo-ops please run full CI for 0470aca |
|
/ok to test 0470aca |
Two upstream commits since d2ceb6a, no conflicts, nothing under a router-owned path and no Rust change: - ai-dynamo#18334 fix(recipes): restore Nemotron 3.5 Super VL benchmark defaults. recipes/.../nemotron-3.5-super-vl-120b-a12b/perf (README, perf.yaml). - ai-dynamo#17937 fix(multimodal): keep invalid-audio errors as client errors in batch loading. components/src/dynamo/common/multimodal/audio_loader.py and its test. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…loading (ai-dynamo#17937) Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com> Signed-off-by: GLAMR <svc-glamr@nvidia.com>
Failure behavior
When a client sends corrupt or empty audio to a vLLM worker, Dynamo reports a server error instead of a client error. The single-item audio loader correctly raises a
ValueErrorsuch asFailed to load audio from https://example.com/bad.wav: ... Decoded waveform is empty.The batch loader that the vLLM worker calls then folds that error into a plainException. The Python engine binding turns aValueErrorinto an invalid-argument error (4xx), but a plainExceptionbecomes an unknown error (5xx). So the client-error verdict is lost. The image and video batch loaders already keep theValueError; only audio drops it. The defect is in Dynamo's Python audio loader.Exceptioninstead ofValueErrorwithout the fix, and raisedValueErrorwith it.Proposed fix
The audio batch loader now keeps the first invalid-audio error and raises it unchanged, as the image and video batch loaders do. Typed HTTP status, URL-policy and missing-decoder errors still take priority over it.
load_audioturns unexpected errors intoValueError. It now lets three errors through with their own type, because they say nothing about the audio itself: an HTTP fetch timeout (HttpTimeoutError), a connection error (HttpConnectionError), and missing vLLM audio components (a privateRuntimeErrorsubclass). In a batch these still produce the generic server error, as they did before, instead of taking the new client-error branch.HttpConfigurationErrorwas already let through the same way.load_audioalso preservesMemoryErrorandOSErrorwithENOMEM,EMFILE, orENFILE. These report host memory or file-descriptor exhaustion, so they must remain server errors rather than become invalid audio.use_audio_in_videocallsload_audiodirectlyValueError)load_audiois called directlyValueError)Validation
This PR adds four unit tests, three of them parametrized (10 cases in total), to
components/src/dynamo/common/tests/multimodal/test_audio_loader.py. They run without a GPU, against a mocked audio fetch, decoder call orload_audio. The four host-resource cases execute the realload_audioandload_audio_batchmethods. They do not load a model or use the network; each is expected to take milliseconds.ValueErrorValueErrorwins over a generic failure in the same batchExceptionValueErrorValueErrortest_load_audio_batch_resource_failure_is_not_client_errorab2ac405c6wraps them inValueError; verified by code inspection and the reviewer’s reproduction0470aca09cAt
ab2ac405c6on CPU,python3 -m pytest components/src/dynamo/common/tests/multimodal/test_audio_loader.py -qpassed (25 tests).pre-commit run --fileson both changed files passed.components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.pyneeds vLLM and was not run here. Full CI (thePRworkflow, including the vLLM, SGLang and TensorRT-LLM runtime test jobs) passed onab2ac405c6. Not run: a live vLLM request with corrupt audio checking for HTTP 400, which needs a GPU.Relates to #16829, which made the Rust frontend return 400 for media that cannot be fetched or decoded.
Closes DYN-5237.
Summary by CodeRabbit
ValueError, even when another item in the batch fails first.