Skip to content

fix(recipes): restore Nemotron 3.5 Super VL benchmark defaults - #18334

Merged
krishung5 merged 1 commit into
ai-dynamo:mainfrom
milesial:alexandrem/nemotron-benchmark-parameters
Nov 6, 2026
Merged

krishung5 merged 1 commit into
ai-dynamo:mainfrom
milesial:alexandrem/nemotron-benchmark-parameters

Conversation

@milesial

@milesial milesial commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Overview

The Nemotron 3.5 Super VL benchmark Job leaves four shared parameters empty, so users cannot find their values in the recipe. Restore the prior Job defaults and document their units.

Key Changes

  • No core logic changes. Set BENCHMARK_DURATION=3600, WARMUP_GRACE_PERIOD=1800, REQUEST_TIMEOUT_SECONDS=7200, and MAX_CONTEXT_LENGTH=1048576.
  • Document each default with its variable name and unit. Endpoint, concurrency, artifact directory, and cache salt remain required.

Validation

Parsed both YAML documents and compared the restored values with the prior benchmark Job. The embedded shell script passes bash -n. These are recovered Job defaults, not a new reproduction of the published performance results.

Issues and Ongoing Work

Closes #18333.

No duplicate fix found. #14753 addresses AIPerf version pins in other recipes.

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • New Features
    • Performance jobs now use defaults of 3,600 seconds for benchmark duration, 1,800 seconds for warmup, 7,200 seconds for request timeout, and 1,048,576 tokens for maximum context length.
  • Documentation
    • Performance setup guidance identifies ENDPOINT, CONCURRENCY, ARTIFACT_DIR, and CACHE_SALT as the values to configure in perf.yaml; the other settings use job defaults.

Signed-off-by: milesial <milesial@users.noreply.github.com>
@milesial
milesial temporarily deployed to external_collaborator November 6, 2026 18:17 — with GitHub Actions Inactive
@milesial
milesial temporarily deployed to external_collaborator November 6, 2026 18:17 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

👋 Hi milesial! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added the external-contribution Pull request is from an external contributor label Nov 6, 2026
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 600234a

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 6, 2026
@milesial
milesial marked this pull request as ready for review November 6, 2026 18:20
@milesial
milesial requested a review from a team as a code owner November 6, 2026 18:20
@krishung5
krishung5 merged commit aa620b7 into ai-dynamo:main Nov 6, 2026
112 of 113 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 6, 2026
Two upstream commits since d2ceb6a, no conflicts, nothing under a
router-owned path and no Rust change:

- ai-dynamo#18334 fix(recipes): restore Nemotron 3.5 Super VL benchmark defaults.
  recipes/.../nemotron-3.5-super-vl-120b-a12b/perf (README, perf.yaml).
- ai-dynamo#17937 fix(multimodal): keep invalid-audio errors as client errors in
  batch loading. components/src/dynamo/common/multimodal/audio_loader.py
  and its test.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devivasudevan pushed a commit to devivasudevan/dynamo that referenced this pull request Nov 7, 2026
…namo#18334)

Signed-off-by: milesial <milesial@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 600234a7 Deployed Nov 6, 2026 by milesial via ok-to-test #47679
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nemotron 3.5 Super VL benchmark parameters are missing

4 participants