Skip to content

feat: update FastVideo diffusers worker to new typed API - #9591

Merged
rmccorm4 merged 12 commits into
ai-dynamo:mainfrom
vishruthb:vishruthb/fastvideo-examples-update
Jul 30, 2026
Merged

rmccorm4 merged 12 commits into
ai-dynamo:mainfrom
vishruthb:vishruthb/fastvideo-examples-update

Conversation

@vishruthb

@vishruthb vishruthb commented May 15, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

Update the FastVideo example worker in examples/diffusers/ to use FastVideo's new typed API (fastvideo.api), replacing the legacy generate_video() interface. Full native backend promotion is tracked separately in #9535.

This version also wires the optimization flag to FastVideo's real NVFP4 transformer quantization path and pins a FastVideo commit that implements that typed carrier.

Details:

examples/diffusers/worker.py

  • Rewritten to use VideoGenerator.from_config(GeneratorConfig(...)) and generator.generate(GenerationRequest(...)) with structured config objects (EngineConfig, OffloadConfig, CompileConfig, QuantizationConfig, PipelineSelection)
  • Added CLI flags mapping directly to the typed config: --model, --num-gpus, --attention-backend, offload toggles, --torch-compile, --fp4-quantization, --vsa-sparsity
  • --fp4-quantization now requests QuantizationConfig(transformer_quant="NVFP4"), matching FastVideo's typed NVFP4 transformer quantization API
  • Added configurable max video dimension validation (default 4096x4096) to prevent OOM from oversized requests
  • Added cancellation cleanup so completed staging MP4s are unlinked even when the client cancels
  • Returns complete MP4 payloads as data[0].b64_json; url is rejected because this example worker does not host generated files
  • Default model updated to FastVideo/FastWan2.1-T2V-1.3B-Diffusers

Supporting files

  • Dockerfile: pinned FastVideo install to e6022c20b27c94ef546143a7796bb73e1c77cdb4 for typed API and NVFP4 transformer quantization support
  • Dockerfile / local/docker-compose.yml: default TORCH_CUDA_ARCH_LIST now includes 12.0 for RTX 50-series GPUs, alongside Blackwell data center targets
  • local/run_local.sh: updated for new CLI flags, uses shared launch utilities
  • local/docker-compose.yml: updated worker command
  • docs/features/diffusion/fastvideo.md: updated for new API, NVFP4 behavior, RTX 50-series build defaults, and current response-format behavior

Testing done

Static/local verification:

nvidia-smi
python3 -c "import torch; print(torch.cuda.get_device_name(0)); print(torch.cuda.get_device_capability(0))"
python3 -m py_compile examples/diffusers/worker.py
git diff --check origin/main...HEAD

Verified host GPU: NVIDIA GeForce RTX 5090, compute capability (12, 0).

Verified the pinned FastVideo revision exposes the required typed NVFP4 path:

  • get_quantization_config("NVFP4")
  • generator_config_to_fastvideo_args(...) resolves engine.quantization.transformer_quant
  • FastVideoArgs._apply_transformer_quant() pins the quant config onto pipeline_config.dit_config.quant_config

Smoke test from this branch before the latest cleanup/NVFP4 pin update (RTX 5090, CUDA 12.8):

python examples/diffusers/worker.py \
  --model FastVideo/FastWan2.1-T2V-1.3B-Diffusers \
  --discovery-backend file

python -m dynamo.frontend --http-port 8000 --discovery-backend file

curl -s -X POST http://localhost:8000/v1/videos \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
    "prompt": "A cat walking across a table",
    "size": "256x256",
    "response_format": "b64_json",
    "nvext": {"fps": 8, "num_frames": 8, "num_inference_steps": 1}
  }'

Response: "status": "completed", "object": "video", b64_json present.

Known limitations

  • Flash Attention: The worker sets FASTVIDEO_ATTENTION_BACKEND from the --attention-backend CLI flag. When Flash Attention packages are unavailable, FastVideo falls back to Torch SDPA automatically. Explicit FA4/FA3 import path alignment with upstream FastVideo is tracked separately.
  • Aggregated mode only

Where should the reviewer start?

examples/diffusers/worker.py - the core change. Everything else follows from it.

Related Issues:

Closes DIS-2247

@vishruthb
vishruthb requested a review from a team as a code owner May 15, 2026 00:02
@vishruthb
vishruthb requested a review from a team May 15, 2026 00:02
@vishruthb
vishruthb requested review from a team as code owners May 15, 2026 00:02
@copy-pr-bot

copy-pr-bot Bot commented May 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented May 15, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added docs external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation labels May 15, 2026
@coderabbitai

coderabbitai Bot commented May 15, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

This PR refactors the FastVideo worker to support a typed request/response API with expanded configuration for torch.compile, FP4 quantization, and attention backend selection. The core worker implementation is rewritten, documentation is updated to reflect new defaults, and local deployment infrastructure is modernized to use utility script sourcing and updated environment variables.

Changes

FastVideo Worker Typed API & Configuration Update

Layer / File(s) Summary
Worker API schema and configuration constants
examples/diffusers/worker.py (lines 4–114, 123–517)
NvExtVideoCreateRequest fields become optional with default=None; VideoCreateRequest expands to include size, seconds, response_format, output_format, nvext, and input_reference; VideoData and VideoCreateResponse refactored with output_format, conditional url/b64_json, and new status/progress/error/timing fields.
Worker generation implementation and validation
examples/diffusers/worker.py (lines 123–517)
FastVideoBackend.create_video() now constructs GeneratorConfig with optional torch.compile and FP4 quantization, validates/coerces input sizing and frame parameters, rejects image-to-video requests, serializes generation behind asyncio lock, encodes output MP4 to base64 on demand, and yields structured success/failure responses.
Worker CLI arguments and discovery initialization
examples/diffusers/worker.py (lines 563–734)
CLI parser expanded with flags for torch.compile, FP4 quantization, FSDP, CPU offload modes, attention backends, and VSA sparsity; added --discovery-backend option; main() updated to resolve discovery backend from arguments or environment before runtime creation.
Docker build revision pinning and container capabilities
examples/diffusers/Dockerfile (lines 46–53), examples/diffusers/local/docker-compose.yml (lines 17–18)
FastVideo build now pins to explicit Git revision via FASTVIDEO_REF build argument (default 17f07bc313d77adaf1d60aa344d92f8145e54cfb) with explicit checkout and submodule update; added SYS_PTRACE capability to x-backend-base service for all backend containers.
Local deployment script refactoring
examples/diffusers/local/run_local.sh (lines 9–80)
Script now sources launch_utils.sh and gpu_utils.sh; switches default model to FastVideo/FastWan2.1-T2V-1.3B-Diffusers; renames HTTP_PORT to DYN_HTTP_PORT (default 8000); refactors frontend launch to build command with port and extra args; replaces custom cleanup with kill 0-based trap; replaces manual banners with utility functions; replaces wait -n with wait_any_exit helper.
Documentation alignment with new schema and configuration
docs/features/diffusion/fastvideo.md (lines 16–31, 62, 105–122, 212–219, 238–257, 266, 276–279)
Documented defaults and MP4 response format updated; default attention backend changed to VIDEO_SPARSE_ATTN with VSA sparsity guidance; optimization flag references updated from --enable-optimizations to --torch-compile and FP4; example request payload and worker configuration reference tables updated; troubleshooting refreshed to address compile/quantization/attention-backend compatibility.

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The PR title accurately reflects the main change: updating the FastVideo diffusers worker to use a new typed API, which is the primary objective and most substantial code change in this changeset.
Description check ✅ Passed The PR description matches the required template and includes overview, details, review focus, and related issue linkage.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/local/run_local.sh Resolved
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
@vishruthb vishruthb changed the title docs: update FastVideo diffusers example API feat: update FastVideo diffusers worker to new typed API May 16, 2026
@github-actions github-actions Bot added feat and removed docs labels May 16, 2026
@vishruthb
vishruthb force-pushed the vishruthb/fastvideo-examples-update branch 2 times, most recently from 9f6f384 to 0db3ada Compare May 16, 2026 11:59
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/Dockerfile Resolved
Comment thread docs/features/diffusion/fastvideo.md Resolved
Comment thread examples/diffusers/Dockerfile Resolved
@vishruthb
vishruthb force-pushed the vishruthb/fastvideo-examples-update branch from 0db3ada to 5be627c Compare May 20, 2026 04:26
Comment thread docs/features/diffusion/fastvideo.md Resolved
Comment thread examples/diffusers/Dockerfile Resolved
@vishruthb
vishruthb force-pushed the vishruthb/fastvideo-examples-update branch from 5be627c to 566bc49 Compare May 21, 2026 02:24
Comment thread examples/diffusers/Dockerfile Resolved
Comment thread docs/features/diffusion/fastvideo.md Resolved
@jkosek

jkosek commented Jun 17, 2026 •

Copy link
Copy Markdown

@SolitaryThinker @vishruthb I did a quick check using Docker Compose and have following error:

Traceback (most recent call last):
  File "/opt/app/worker.py", line 42, in <module>
    from dynamo.llm import (  # type: ignore[attr-defined]
ImportError: cannot import name 'WorkerType' from 'dynamo.llm' (/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/llm/__init__.py)

After removing those lines:

I was able to run it but got another error on RTX A6000:

INFO 06-17 14:13:40.867 [component_loader.py:412] Loading weights took 13.32 seconds
Traceback (most recent call last):
  File "/opt/app/worker.py", line 884, in <module>
    asyncio.run(main(_args))
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/app/worker.py", line 869, in main
    await backend_worker(runtime, args)
  File "/opt/app/worker.py", line 666, in backend_worker
    await backend.initialize_model()
  File "/opt/app/worker.py", line 246, in initialize_model
    self.generator = await asyncio.to_thread(
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/fastvideo/entrypoints/video_generator.py", line 213, in from_config
    generator = cls.from_fastvideo_args(fastvideo_args, log_queue=log_queue)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/fastvideo/entrypoints/video_generator.py", line 251, in from_fastvideo_args
    return cls(
           ^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/fastvideo/entrypoints/video_generator.py", line 142, in __init__
    self.executor = executor_class(fastvideo_args, log_queue=log_queue)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/fastvideo/worker/executor.py", line 27, in __init__
    self._init_executor()
  File "/opt/dynamo/venv/lib/python3.12/site-packages/fastvideo/worker/multiproc_executor.py", line 112, in _init_executor
    self.workers = WorkerMultiprocProc.wait_for_ready(unready_workers)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/fastvideo/worker/multiproc_executor.py", line 641, in wait_for_ready
    raise e from None
Exception: WorkerMultiprocProc initialization failed due to an exception in a background process. See stack trace for root cause.

Attached the bash script used for reproduction: run_docker.sh

Could you please verify if the examples is functional using local and Docker setup? Thanks.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 new potential issues.

Open in Devin Review

Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
Comment thread examples/diffusers/worker.py Resolved
@rmccorm4
rmccorm4 requested review from a team as code owners July 15, 2026 16:50
@rmccorm4

Copy link
Copy Markdown
Contributor

Hi @vishruthb @SolitaryThinker just following up on Jakub's feedback above: #9591 (comment)

@rmccorm4
rmccorm4 temporarily deployed to external_collaborator July 15, 2026 16:50 — with GitHub Actions Inactive
vishruthb added 10 commits July 18, 2026 19:52
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
The ai-dynamo==1.2.1 wheel installed by the example Dockerfile does not
export dynamo.llm.WorkerType, and its register_llm() has no worker_type
parameter, so importing/passing it crashed the worker at startup.

Also gate VIDEO_SPARSE_ATTN on compute capability >= 9.0 before
VideoGenerator.from_config: fastvideo-kernel 0.2.6 compiles its VSA
kernels for sm90a only and its Triton fallback fails at runtime on
pre-Hopper GPUs (e.g. sm86), previously surfacing as an opaque
background-process error after weights had already loaded.

Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
- Move the torch import to module level; torch is a hard dependency of
  fastvideo, so the try/except ImportError path was unreachable.
- Catch ValueError from request validation specifically in create_video;
  keep a deliberate broad catch for FastVideo internals that yields the
  documented VideoCreateResponse(status="failed") instead of re-raising,
  so /v1/videos clients receive structured errors rather than transport
  failures.
- Default --vsa-sparsity to None so users can opt out of setting
  VSA_sparsity in PipelineSelection.experimental; apply the documented
  0.8 default only when the flag is unset and the backend is
  VIDEO_SPARSE_ATTN.

Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
Document the compute-capability requirements surfaced by the new
startup gate (FastWan2.1+VSA needs >= 9.0, LTX2+TORCH_SDPA as the
pre-Hopper path, NVFP4 needs >= 10.0) and replace the permanently
redirected tree/ Dockerfile link with its blob/ form for the lychee
check.

Signed-off-by: Vishruth Bharath <vishruthbharath@gmail.com>
@vishruthb
vishruthb force-pushed the vishruthb/fastvideo-examples-update branch from eb535d9 to ed1d4dc Compare July 19, 2026 02:53
@vishruthb
vishruthb temporarily deployed to external_collaborator July 19, 2026 02:53 — with GitHub Actions Inactive
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator July 30, 2026 05:08 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 91ddc82

@rmccorm4
rmccorm4 temporarily deployed to external_collaborator July 30, 2026 05:10 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 5b923b9

@rmccorm4
rmccorm4 enabled auto-merge (squash) July 30, 2026 05:11
@rmccorm4
rmccorm4 merged commit 36f6ddb into ai-dynamo:main Jul 30, 2026
87 checks passed

This branch was previously deployed

2 inactive (1 outdated) deployments
external_collaborator — 5b923b9b Deployed Jul 30, 2026 by rmccorm4 via ok-to-test #5723
GITLAB — 4f91b67b Deployed Jun 11, 2026 by copy-pr-bot[bot] via Trigger CI Pipeline #30905
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants