Skip to content

[VideoGenX] add SGLang Wan2.2 on H100 with 4 and 8 GPUs / [VideoGenX] 在 H100 上新增 SGLang Wan2.2(4、8 卡) - #5114

Open
edwingao28 wants to merge 10 commits into
mainfrom
feat/wan22-h100-sglang-video
Open

edwingao28 wants to merge 10 commits into
mainfrom
feat/wan22-h100-sglang-video

Conversation

@edwingao28

@edwingao28 edwingao28 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Wan2.2-T2V-A14B has no H100 curve. The dashboard shows it on B200 and B300 with vLLM-Omni only, so an H100 result has no baseline on either engine. This PR adds the H100 SGLang row on 4 and 8 GPUs. #5113 adds the same row on H200.

Changes:

GPUs Recipe CFG parallel Ulysses Server warmup clip
4 serving-client-g4.yaml 2 2 17 frames (SGLang default)
8 serving-client.yaml 2 4 21 frames (--warmup-num-frames 21)
  • --warmup-num-frames 21 on 8 GPUs. Without it the first 81-frame decode ran out of GPU memory, and the 8-GPU point returned 0 / 20 clips.
    • SGLang shards the Wan VAE decode across its decode ranks only at 4096 or more latent elements per rank (base.py#L118).
    • Its default server warmup clip has 17 frames (warmup_request_builder.py#L56). At 832x480 that is 5 x 60 x 104 latent elements, or 3900 per rank on the 8 decode ranks of CFG 2 x Ulysses 4.
    • The warmup decode therefore ran unsharded and never used the decode group's NCCL communicator.
    • NCCL connected that communicator's channels inside the first 81-frame decode. Six of the eight GPUs had under 0.5 GiB free at that point, and ncclCuMemAlloc failed.
    • A 21-frame clip has 6 latent frames, or 4680 elements per rank. The warmup now runs the sharded decode, so NCCL allocates before the server reports ready.
    • 21 is the next Wan frame count (4k + 1) after 17. An 81-frame warmup also works and leaves less free memory, as the second Results table shows.
    • The 4-GPU recipe keeps the default. Its 17-frame warmup gives 7800 elements per rank.
    • The [VideoGenX] add SGLang Wan2.2 on H200 with 1, 2, 4 and 8 GPUs / [VideoGenX] 在 H200 上新增 SGLang Wan2.2(1、2、4、8 卡) #5113 H200 8-GPU recipe keeps the default. It returned 20 / 20 clips on H200 in run 44067008117.
  • No 1-GPU or 2-GPU point. 80 GB H100s run out of memory on the deployments that the H200 1-GPU and 2-GPU points use. The Results table has both failed runs. Later commits remove the two points and their recipes.
  • Row wan22-bf16-h100-sglang-video on cluster:h100-dsxe runs wan22.yaml release-v1 at C1 on 4 and 8 GPUs.
  • Changelog. One perf-changelog.yaml block lists only the new key.
  • Docs. MODELS.md and docs/video-generation.md add the H100 SGLang row.

Results:

PR sweeps on cluster:h100-dsxe: wan22.yaml release-v1 at C1 with 1 warmup and 20 measured clips per point, SGLang v0.5.23 (297e54fd), every component resident on the GPU. Latency, throughput and energy are the values the VideoGenX preview API returns for each run.

Run Head GPUs CFG parallel x Ulysses Valid clips p50 s per clip Clips per hour kJ per clip Outcome
44198185716 attempt 1 242d44fef 8 2 x 4 20 / 20 28.27 127.14 155.8 passed
44198185716 attempt 1 242d44fef 4 2 x 2 20 / 20 52.37 68.66 145.8 passed
44087759927 attempt 1 0ef28e01c 8 2 x 4 0 / 20 none none none failed in the first 81-frame decode
44087759927 attempt 1 0ef28e01c 4 2 x 2 20 / 20 53.24 67.55 148.2 passed
44043897837 attempt 1 654359e93 2 2 x 1 0 / 20 none none none Failed in the first denoising step
44037536088 attempt 1 88f5855d3 1 1 x 1 0 / 20 none none none Failed while loading
  • Run 44198185716 qualifies. validate-video-results passes, with 20 / 20 valid clips and valid power at both points.
  • The preview draws both points. VideoGenX preview of run 44198185716.
  • 8 GPUs with the flag. The server log of run 44198185716 shows Warmup requests: 1/1 server warmup req (832x480x21f, 2/40 steps). It has 21 decodes of 0.95 to 0.99 s and no out of memory line.
  • 8 GPUs without the flag. In run 44087759927 the first request denoised in 27.3 s. Its decode failed after 2.6 s with (ncclCuMemAlloc) NCCL WARN Cuda failure 2 'out of memory'. SGLang kept the request in progress, so the client waited out its 900 s limit.
  • The 4-GPU point did not change. Both runs render the same 4-GPU recipe, fingerprint 03e996b141a5. Its p50 differs by 0.87 s (1.6 %) between the two runs.
  • p90 exceeds p50 by at most 0.11 s at every passing point.
  • H200 run 44067008117 in [VideoGenX] add SGLang Wan2.2 on H200 with 1, 2, 4 and 8 GPUs / [VideoGenX] 在 H200 上新增 SGLang Wan2.2(1、2、4、8 卡) #5113 measured a p50 of 28.3 s on 8 GPUs and 53.3 s on 4 GPUs with the same two layouts.
  • 1 GPU: the load failed with torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 160.00 MiB. GPU 0 has a total capacity of 79.18 GiB of which 58.12 MiB is free. transformer_2 took 26.62 GB and transformer took 26.62 GB. The text_encoder load then started with 19.47 GB free. SGLang loads the UMT5 encoder in FP32, which is 21.2 GiB for its 5.68 billion parameters.
  • 2 GPUs: both GPUs failed with CUDA out of memory. Tried to allocate 320.00 MiB. GPU 0 has a total capacity of 79.18 GiB of which 162.00 MiB is free. SGLang split the text encoder across the two GPUs, 10.58 GB each. After loading, each GPU had 6.80 GB free. At the failure PyTorch had 68.18 GiB allocated and 5.07 GiB reserved but unallocated.
  • The 1-GPU and 2-GPU runs each stopped at their canary, so no other point ran in them.

Direct reproduction of the 8-GPU failure: Slurm job 1867 on one cluster:h100-dsxe node with 8 H100 80 GB GPUs. It ran outside Actions, so its rows have no run link. It used the image digest, SGLang revision, model snapshot and server flags of the 8-GPU recipe, with NCCL_DEBUG=INFO. Each row is one server start followed by 81-frame, 40-step requests. nvidia-smi sampled GPU memory every 250 ms.

Server warmup frames Valid 81-frame clips Decode time (s) NCCL channel connects in the first 81-frame decode Lowest free memory on any GPU at ready (GiB) Lowest free memory on any GPU at peak (GiB)
17 (default) 0 / 1 failed after 2.75 496 2.15 0.47
21 3 / 3 0.96 to 1.08 0 9.00 6.83
81 3 / 3 0.97 to 1.00 0 3.58 3.20
  • With 17 frames, ncclCuMemAlloc failed on ranks 1 to 6. The decode had logged 496 channel connects and 1099 P2P buffer allocations by then.
  • With 21 frames those connects happen during the warmup. The log has 2976 channel-connect lines before ready, against 2336 with 17 frames.
  • Submit-to-terminal time was 28.59 to 28.61 s for all 6 clips of the two passing rows.
  • The 21-frame and 81-frame servers returned files of the same size for seeds 11, 23 and 37: 2231625, 1426815 and 1708190 bytes.

Testing:

  • infx generate --config-key wan22-bf16-h100-sglang-video renders 2 recipes. Only the 8-GPU recipe carries --warmup-num-frames 21.
  • python -m infx.matrix.changed against merge base 7e2b3622f lists only wan22-bf16-h100-sglang-video, with 2 added points and 0 removed. Their fingerprints, 03e996b141a5 and 3061acbb60c7, are the two points of run 44198185716.
  • The video-generation suite passes, 141 tests.

Limits:

  • There is no 1-GPU or 2-GPU H100 SGLang number. On 80 GB cards those counts need a deployment that differs from the H200 points, such as tensor parallelism or text-encoder offload. This PR adds neither, so each H100 point is the same deployment as its H200 point.
  • GPU memory was sampled only in the direct job. The PR sweep shows the 21-frame warmup line and the clip counts, not per-GPU memory.
  • The memory that the decode group's NCCL buffers take per rank was not measured.
  • The threshold and the 17-frame default are read from SGLang source at 297e54fd. A different SGLang version needs the frame count rechecked.
  • SGLang does not mark a request failed when its decode fails. This PR does not change that, so a failed decode still costs the client its full 900 s limit.

Related: Shares video-generation/ with #5113 (H200 SGLang). Whichever of the two merges second needs a main sync, because both edit the same places in nvidia-master.yaml, MODELS.md, docs/video-generation.md and perf-changelog.yaml. #5111 adds the vLLM-Omni row on the same H100 cluster. Follows #5082 (B300) and #5084 (B200 GPU-count sweep).

PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, a maintainer with write access to the repository has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@edwingao28
edwingao28 force-pushed the feat/wan22-h100-sglang-video branch from f28acb3 to 88f5855 Compare November 17, 2026 07:17
@edwingao28 edwingao28 added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 17, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread video-generation/evaluator/runner.py Resolved
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

@edwingao28 edwingao28 changed the title [VideoGenX] add SGLang Wan2.2 on H100 with 1, 2, 4 and 8 GPUs / [VideoGenX] 在 H100 上新增 SGLang Wan2.2(1、2、4、8 卡) [VideoGenX] add SGLang Wan2.2 on H100 with 2, 4 and 8 GPUs / [VideoGenX] 在 H100 上新增 SGLang Wan2.2(2、4、8 卡) Nov 17, 2026
@edwingao28 edwingao28 added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 17, 2026
@edwingao28 edwingao28 changed the title [VideoGenX] add SGLang Wan2.2 on H100 with 2, 4 and 8 GPUs / [VideoGenX] 在 H100 上新增 SGLang Wan2.2(2、4、8 卡) [VideoGenX] add SGLang Wan2.2 on H100 with 4 and 8 GPUs / [VideoGenX] 在 H100 上新增 SGLang Wan2.2(4、8 卡) Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant