Skip to content

fix(multimodal): keep invalid-audio errors as client errors in batch loading - #17937

Merged
rmccorm4 merged 4 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5237-preserve-invalid-audio-errors-batch-loading-e1b3fd00a177
Nov 6, 2026
Merged

rmccorm4 merged 4 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5237-preserve-invalid-audio-errors-batch-loading-e1b3fd00a177

Conversation

@glamr-agent

@glamr-agent glamr-agent commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Failure behavior

When a client sends corrupt or empty audio to a vLLM worker, Dynamo reports a server error instead of a client error. The single-item audio loader correctly raises a ValueError such as Failed to load audio from https://example.com/bad.wav: ... Decoded waveform is empty. The batch loader that the vLLM worker calls then folds that error into a plain Exception. The Python engine binding turns a ValueError into an invalid-argument error (4xx), but a plain Exception becomes an unknown error (5xx). So the client-error verdict is lost. The image and video batch loaders already keep the ValueError; only audio drops it. The defect is in Dynamo's Python audio loader.

Reproduction Without the fix With the fix Note
Exact failure on the reported setup Not run — Needs a vLLM deployment on a GPU that sends corrupt audio and checks for HTTP 400; the coding run had a CPU-only sandbox.
Exact failure on a similar setup Not run — Needs a live Dynamo deployment with the vLLM engine on a GPU; the coding run had a CPU-only sandbox.
Exact error in a test ❌ Exact error ✅ Passed New unit tests on CPU: undecodable and empty audio in a two-item batch raised a plain Exception instead of ValueError without the fix, and raised ValueError with it.

Proposed fix

The audio batch loader now keeps the first invalid-audio error and raises it unchanged, as the image and video batch loaders do. Typed HTTP status, URL-policy and missing-decoder errors still take priority over it.

load_audio turns unexpected errors into ValueError. It now lets three errors through with their own type, because they say nothing about the audio itself: an HTTP fetch timeout (HttpTimeoutError), a connection error (HttpConnectionError), and missing vLLM audio components (a private RuntimeError subclass). In a batch these still produce the generic server error, as they did before, instead of taking the new client-error branch. HttpConfigurationError was already let through the same way.

load_audio also preserves MemoryError and OSError with ENOMEM, EMFILE, or ENFILE. These report host memory or file-descriptor exhaustion, so they must remain server errors rather than become invalid audio.

Case Before After
Corrupt or empty audio in a request Server error (5xx) Client error (4xx)
Invalid audio plus a NIXL read failure in one batch Server error Client error
Audio fetch timeout, connection error, or vLLM audio components missing, in a batch Server error Unchanged
The same failures when use_audio_in_video calls load_audio directly Client error (ValueError) Server error (original type)
Blocked URL or upstream HTTP status error Typed error kept Unchanged
Missing audio decoder Typed error kept Unchanged
Host memory or file-descriptor exhaustion in a batch Server error Unchanged
The same host-resource failures when load_audio is called directly Client error (ValueError) Server error (original type)
Only resource or configuration failures Server error Unchanged

Core change: about 40 lines (+40/−2). The logic is in 1 file:

  • audio_loader.py: the batch loader keeps the first invalid-audio error and raises it after the typed errors and before the generic aggregate; load_audio keeps the type of fetch-timeout, connection, missing-vLLM and host-resource errors.

The rest of the +156/−3 diff is tests.

Validation

This PR adds four unit tests, three of them parametrized (10 cases in total), to components/src/dynamo/common/tests/multimodal/test_audio_loader.py. They run without a GPU, against a mocked audio fetch, decoder call or load_audio. The four host-resource cases execute the real load_audio and load_audio_batch methods. They do not load a model or use the network; each is expected to take milliseconds.

Behavior checked Checked by Without the fix With the fix
Undecodable or empty audio in a batch stays a ValueError Two new unit test cases; also the reproduction above ❌ Exact error ✅ Passed
A ValueError wins over a generic failure in the same batch New unit test ❌ Failed: raised a plain Exception ✅ Passed
A fetch timeout, connection error or missing vLLM in a batch is not a ValueError Three new unit test cases ❌ Failed: raised ValueError ✅ Passed
Host-resource failures remain server errors in a batch Four cases in test_load_audio_batch_resource_failure_is_not_client_error The intermediate head ab2ac405c6 wraps them in ValueError; verified by code inspection and the reviewer’s reproduction Passed in the Dynamo sequential CPU job on amd64 at 0470aca09c

At ab2ac405c6 on CPU, python3 -m pytest components/src/dynamo/common/tests/multimodal/test_audio_loader.py -q passed (25 tests). pre-commit run --files on both changed files passed. components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.py needs vLLM and was not run here. Full CI (the PR workflow, including the vLLM, SGLang and TensorRT-LLM runtime test jobs) passed on ab2ac405c6. Not run: a live vLLM request with corrupt audio checking for HTTP 400, which needs a GPU.

Relates to #16829, which made the Rust frontend return 400 for media that cannot be fetched or decoded.

Closes DYN-5237.

Summary by CodeRabbit

  • Bug Fixes
    • Batch audio loading now reports invalid audio inputs as a ValueError, even when another item in the batch fails first.

load_audio_batch folded a ValueError from load_audio (empty or undecodable
audio) into a generic Exception. The Python bindings map ValueError to
InvalidArgument and a generic Exception to Unknown, so bad client audio was
reported as a server error. Keep the first ValueError and raise it after the
typed HTTP, URL-policy and missing-decoder errors and before the generic
aggregate, as the image and video batch loaders already do.

Refs DYN-5237.

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent requested review from a team as code owners November 3, 2026 09:20
@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 3, 2026 09:20 — with GitHub Actions Inactive
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 3, 2026 09:20 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added the external-contribution Pull request is from an external contributor label Nov 3, 2026
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: Initial validation snapshot.

Validation status: complete

Check Result Scope or reason
Runs the relevant Python unit tests without requiring a GPU. Passed (exit 0) Audio, video and image loader tests: 148 passed. With only audio_loader.py reset to main, the 3 new cases fail with a plain Exception (3 failed, 8 passed, 11 deselected); with the fix, 11 passed. Tested at 83f8bb2725.

The live check that corrupt audio returns HTTP 400 from a vLLM worker was not run: it needs a GPU and the vLLM engine, and the sandbox was CPU-only.

Detailed evidence — validation complete

Left out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1, passed), and lint and formatting with pre-commit (Check 2, passed).

Evidence tally: [3/3 validated]

Advisory AI assessment: sound. No findings.

Validation result: pass — all 3 new test cases fail on main with the reported plain Exception and pass with the fix, and the audio, video and image loader tests (148) and pre-commit all pass. The live HTTP 400 check needs a GPU and vLLM, which this CPU-only sandbox lacks.

Evidence audit: complete [3/3 validated] — the command report comes from recorded runs.

Commands and results [3/3 validated]

Generated from the commands recorded during this run.

Check 1

Builds the changed Dynamo source and confirms that Python can import its compiled extension.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 2

Checks the changed files with the repository's fast lint and formatting commands.

Result: Passed (exit 0)

Command:

bash -c 'python3 -m py_compile components/src/dynamo/common/multimodal/audio_loader.py components/src/dynamo/common/tests/multimodal/test_audio_loader.py && echo py_compile OK && pre-commit run --files components/src/dynamo/common/multimodal/audio_loader.py components/src/dynamo/common/tests/multimodal/test_audio_loader.py'

Check 3

Runs the relevant Python unit tests without requiring a GPU.

Result: Passed (exit 0)

Command:

/opt/dynamo/venv/bin/python -m pytest components/src/dynamo/common/tests/multimodal/test_audio_loader.py components/src/dynamo/common/tests/multimodal/test_video_loader.py components/src/dynamo/common/tests/multimodal/test_image_loader.py -v --tb=short -p no:cacheprovider

@glamr-agent

glamr-agent commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

No description provided.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: @dynamo-ops please run full CI for 83f8bb2

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 83f8bb2

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Walkthrough

AudioLoader.load_audio_batch now retains the first ValueError from failed audio loads. After checking for specific HTTP, URL validation, and missing decoder errors, it raises the retained ValueError before the generic aggregated exception. Tests cover decoder failures, empty waveforms, and mixed batch failures.

Changes

Audio batch error handling

Layer / File(s) Summary
Retain and raise batch ValueError
components/src/dynamo/common/multimodal/audio_loader.py, components/src/dynamo/common/tests/multimodal/test_audio_loader.py
The batch loader records the first ValueError and raises it before the generic aggregated exception. Tests cover decoder failures, empty waveforms, and a ValueError after a RuntimeError.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Moderate · up to 83f8b

Transport and backend failures can be reported as invalid audio. Preserve their server-error classification before merging.

Pre-merge checks: Passed 3 · Failed 1 · Inconclusive 1

Failed Failed checks: 1 warning · 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Inconclusive The description covers the purpose, changes, validation, and related issues. However, it claims load_audio error-handling changes and tests that are not in the supplied change summary. The validatio… Reconcile the description and validation results with the current PR diff. Remove unsupported claims or confirm that the changes and tests are present. Use the template headings and add the pull request checklist.
Passed Passed checks: 3 passed

Check name Status Explanation
Title check Passed The title names the change: the audio batch loader keeps invalid-audio errors as ValueError, so the client sees a 4xx instead of a 5xx. The raw summary and objectives describe this same change in audi…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the change: the audio batch loader keeps invalid-audio errors as ValueError, so the client sees a 4xx instead of a 5xx. The raw summary and objectives describe this same change in audio_loader.py and its tests. The title uses the conventional fix(scope) form, is one sentence, and names the primary change. It is 76 characters, which is slightly long but acceptable.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Full details: Description check

Explanation

The description covers the purpose, changes, validation, and related issues. However, it claims load_audio error-handling changes and tests that are not in the supplied change summary. The validation also refers to different commits, so its applicability to this diff is unclear.

Resolution

Reconcile the description and validation results with the current PR diff. Remove unsupported claims or confirm that the changes and tests are present. Use the template headings and add the pull request checklist.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2ec0b11 and 83f8bb2.

Files selected for processing: 2

  • components/src/dynamo/common/multimodal/audio_loader.py
  • components/src/dynamo/common/tests/multimodal/test_audio_loader.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

…ient errors

load_audio turned fetch timeouts, connection errors and a missing vLLM into ValueError, so the batch loader reported them as bad requests. Keep their types so only invalid audio is a client error.

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: @dynamo-ops please run full CI for ab2ac40

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test ab2ac40

@devin-ai-integration

devin-ai-integration Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 42127806048 (attempt 1) on 0470aca09c

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other Jobs
dynamo-runtime ❌ 1 ✅ 6

Failure details

1 job failed: all 24 cases in tests/fault_tolerance/test_backend_admission_gate.py fail during shared etcd fixture setup because the server exits before port 2380 is ready.

❌ dynamo-runtime / test / parallel cuda13.0, arm64: shared etcd exits before port 2380 is ready

Job: dynamo-runtime / test / parallel cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 127602538473 -R ai-dynamo/dynamo --log-failed

tests/conftest.py:1254: in runtime_services_session
    with SharedEtcdServer(request, tmp_path_factory) as etcd:
tests/conftest.py:1059: in __enter__
    self._server.__enter__()
...
tests/utils/managed_process.py:675: in _check_process_alive
    raise RuntimeError(
E   RuntimeError: Main server process exited with code 1 while waiting for port 2380
...
FAILED tests/fault_tolerance/test_backend_admission_gate.py::test_waiting_limit_holds_requests_until_a_first_response[tcp] - RuntimeError: Main server process exited with code 1 while waiting for port 2380

test_waiting_limit_holds_requests_until_a_first_response[tcp] and the other 23 TCP/NATS admission-gate cases all fail in runtime_services_session: its shared EtcdServer exits with code 1 while waiting for client port 2380. This is a shared test-service startup failure, not 24 independent assertion failures; the job log does not include etcd's own exit reason.

For agents
{
  "pr": 17937,
  "run_id": 42127806048,
  "run_attempt": 1,
  "head_sha": "0470aca09c3c99944145f554363f2bfa14fc563d",
  "failures": [
    {
      "job": "dynamo-runtime / test / parallel cuda13.0, arm64",
      "job_id": 127602538473,
      "failed_step": "Run CPU-only tests (parallelized)",
      "signature": "Shared etcd exits with code 1 before port 2380 is ready",
      "tests": [
        "tests/fault_tolerance/test_backend_admission_gate.py::test_waiting_limit_holds_requests_until_a_first_response[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_waiting_limit_holds_requests_until_a_first_response[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_request_limit_holds_requests_until_one_finishes[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_request_limit_holds_requests_until_one_finishes[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[defaults-tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[defaults-nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[zero-queue-limit-tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_unset_engine_limits_pass_every_request_to_the_engine[zero-queue-limit-nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_queue_limit_without_an_engine_limit_fails_worker_startup[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_queue_limit_without_an_engine_limit_fails_worker_startup[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_legacy_aliases_configure_the_request_and_queue_limits[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_legacy_aliases_configure_the_request_and_queue_limits[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_the_former_waiting_limit_name_configures_the_waiting_limit[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_the_former_waiting_limit_name_configures_the_waiting_limit[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_ranks_admit_and_queue_independently_and_share_the_aggregate[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_ranks_admit_and_queue_independently_and_share_the_aggregate[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_full_rank_refuses_for_itself_and_its_idle_sibling_is_admitted_at_once[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_full_rank_refuses_for_itself_and_its_idle_sibling_is_admitted_at_once[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_workers_in_one_process_admit_through_their_own_gates[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_workers_in_one_process_admit_through_their_own_gates[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_caller_disconnecting_while_queued_frees_its_place_before_admission[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_caller_disconnecting_while_queued_frees_its_place_before_admission[nats]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_failed_worker_is_still_reported_down[tcp]",
        "tests/fault_tolerance/test_backend_admission_gate.py::test_a_failed_worker_is_still_reported_down[nats]"
      ],
      "log_cmd": "gh run view --job 127602538473 -R ai-dynamo/dynamo --log-failed"
    }
  ]
}

Posted automatically by Devin for run 42127806048. Updated on every full-CI run of this PR.

@furionw furionw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One correctness issue: the new batch ValueError propagation also classifies host resource exhaustion as invalid client input. See the inline comment and before/after reproduction.

@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 5, 2026 23:11 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: Added the missing test docstrings in commit 0470aca09c3c99944145f554363f2bfa14fc563d to address CodeRabbit’s summary. _fetch now explains its simulated decoder result, and test_load_audio_batch_value_error_outranks_generic_failure explains why invalid audio wins over a generic failure. The transport and missing-vLLM concern in that summary was fixed in ab2ac405c64cb05b4c3e454d7140a9471fab4d1a; this new commit also preserves host resource failures as server errors.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 0470aca

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 0470aca

@furionw furionw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@rmccorm4
rmccorm4 enabled auto-merge (squash) November 6, 2026 17:52
@rmccorm4
rmccorm4 merged commit 2d4996e into ai-dynamo:main Nov 6, 2026
193 of 195 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 6, 2026
Two upstream commits since d2ceb6a, no conflicts, nothing under a
router-owned path and no Rust change:

- ai-dynamo#18334 fix(recipes): restore Nemotron 3.5 Super VL benchmark defaults.
  recipes/.../nemotron-3.5-super-vl-120b-a12b/perf (README, perf.yaml).
- ai-dynamo#17937 fix(multimodal): keep invalid-audio errors as client errors in
  batch loading. components/src/dynamo/common/multimodal/audio_loader.py
  and its test.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devivasudevan pushed a commit to devivasudevan/dynamo that referenced this pull request Nov 7, 2026
…loading (ai-dynamo#17937)

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: GLAMR <svc-glamr@nvidia.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 0470aca0 Deployed Nov 5, 2026 by glamr-agent via ok-to-test #47067
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix multimodal size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants