Repository navigation
Serve MiniMax-H3-Image through the generations API - #5563
Draft
cglagovichTT wants to merge 54 commits into
Draft
cglagovichTT wants to merge 54 commits into
cglagovichTT wants to merge 54 commits into
Conversation
OpenAPI design for tt-media-server's per-model generation API: one submit
endpoint per model, POST /v1/generations/{org}/{model}, and a common
/v1/jobs API. Covers Wan2.2 I2V A14B, Qwen-Image-Edit-2511, FLUX.2 klein 4B,
FLUX.3 Image klein and Ideogram 4.
common.yaml holds the jobs API and shared schemas; models/{org}/{model}.yaml
is one complete spec per model deployment, so each can be replaced by the
spec generated from its Pydantic models as it is implemented.
REQUIREMENTS.md records runner behavior the spec leaves out, and follow-ups.
Docs only; no code changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
{job_id} and {index} are not valid in a URI fragment; Scalar warned on
these $refs. Encode them as %7B/%7D (resolves the same).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- minimaxai/minimax-h3-fl2va and minimaxai/minimax-h3-ref2va, plus tenstorrent/*-lightx2v variants that reuse the base request schema on their own routes - lightricks/ltx-2.3-22b-distilled: one served shape (1920x1080, 6 s, 25 fps), shape fields kept for later - common: VideoMedia and AudioMedia inputs, optional audio track on VideoOutput, per-type size limits - REQUIREMENTS: H3 adaptive ratios, input limits, variants; LTX crop to 1080 in tt-metal; follow-ups for LTX image-to-video and H3 size tiers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PR now holds only the API spec and its README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses review: a reference video's soundtrack is audio conditioning, so it counts toward the 3-audio and 15 s audio limits; a video with sound still counts once toward the 12-reference total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per review: the references don't set the shape in today's stack (or SGLang), so Ref2VA takes a ratio like text-to-video does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Media Addresses review: - every model document now declares components.securitySchemes.Bearer (referencing common.yaml), as OpenAPI 3.1 requires for its security requirement - Media no longer says an earlier output can be passed by its url: output urls are internal runner paths the fetch policy would refuse Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the "Elsewhere" comparisons with other providers. Where one also said how tt-media-server behaves today, keep that part as "tt-media-server today". Only descriptions change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implements the per-model generation API designed in docs/api/generation for
Wan2.2 I2V A14B, Qwen-Image-Edit-2511, FLUX.2 klein 4B and Ideogram 4: one
submit endpoint per model, POST /v1/generations/{org}/{model}, with the
model's own Pydantic request, and the common /v1/jobs API (status with
outputs and expires_at, content, cancel, delete). A deployment mounts only
the submit endpoint of the model it serves. Each request model is checked
against its spec file in the tests.
- Runners consume the requests directly. The tt_dit pipelines take the output
size per call, so a deployment serves every size its schema allows; the
pipeline rejects a size its mesh preset doesn't serve. Warmup covers every
size and input combination the pipeline doesn't warm up itself.
- Input images are https URLs fetched at submit through media_downloader,
data URIs, or an earlier output's url. A refused URL is 400, an image over
30 MB is 413, everything else 422.
- Outputs are written to one directory per job with an outputs.json manifest,
under TT_GENERATIONS_DIR, and deleted with the job.
- count is fixed at 1, as the spec has it.
- Wan2.2 I2V A14B moves from /v1/videos to the generations API, and its
multihost SP path is dropped. test_module's Wan I2V clients use the new API.
LTX-2.3 distilled moves from /v1/videos to POST /v1/generations/lightricks/ltx-2.3-distilled and /v1/jobs. - The request keeps what LTX served before: prompt and seed, and width, height, frames, duration and fps, each allowing only the one shape the pipeline is built and traced for (1920x1088, 153 frames at 25 fps). Steps and the ignored negative prompt are gone. - Video outputs report `audio`; LTX's MP4 has an AAC soundtrack. The service writes frames, frames with a soundtrack, or an MP4 the pipeline wrote, remuxed for faststart. - A generation job's directory is its result path from submit on. LTX's runner writes its video into it, so a file a cancelled or failed job leaves behind is deleted with the job. - The LTX runner is one class in generation_runners.py; the old one, LTX's branch of VideoGenerateRequest and its shape helpers are removed. - start_ltx_fast.sh sets TT_GENERATIONS_DIR and no longer SERVED_MODEL_NAME, which the generation API's /v1/models does not use. The runbooks use the new routes.
The five MiniMax-H3 deployments move from /v1/videos to the generation API and /v1/jobs: - POST /v1/generations/minimaxai/minimax-h3 (FL2VA: optional first and last frame) - POST /v1/generations/minimaxai/minimax-h3-fasth3 (text only) - POST /v1/generations/minimaxai/minimax-h3-ref2va (reference images, videos and audio clips) - POST /v1/generations/lightx2v/minimax-h3-turbo - POST /v1/generations/lightx2v/minimax-h3-turbo-ref2va - The requests keep what H3 served before: prompt, aspect_ratio or width and height, duration (4 to 15 s, at 24 fps), seed, and the keyframe and reference limits. Ratios accept "16:9", "16x9" and "16/9"; a blank prompt is rejected. The multipart upload route, synchronous mode and the seed defaulting to 0 are gone, as is the T2VA runner, a strict subset of FL2VA. - Inputs can be videos and audio clips as well as images. The service loads and probes them and checks the limits up front (422 per input); H3 caps inline data URIs per model (413). The device and the multihost ranks get the input files, not decoded pixels, and the runner decodes them. - H3's runner writes the MP4 with its soundtrack into the job's directory. The deep canary runs the shortest video on every rank and exports nothing. - The Turbo runners extend the FL2VA and Ref2VA runners. - The (4, 32) quad keeps the lockstep runner; the request round-trips through it with its input files. - The old H3 request types, admission policy, metrics, the VideoAudioResult branch of the encode stage and the H3 doc are removed; the video runners return to their pre-H3 form. - The specs, README tables and the test_module video routing cover the five models.
WanPipeline.__call__ takes no num_frames; the frame count is set once in create_pipeline, which the runner already passes. Every request, warmup included, failed with "got an unexpected keyword argument 'num_frames'".
The tt_dit pipelines load their own weights, from a Hugging Face repo id or a local directory, so a generation deployment never downloads them in the server: - BaseService downloads only when its DOWNLOADS_WEIGHTS flag is set; GenerationService clears it. - The runners' _weights_dir and its snapshot_download fallback are gone. The Ideogram 4 runner passes the weights path through; its pipeline resolves a repo id itself. - The LTX and MiniMax-H3 runners no longer override load_weights.
The pipeline center-crops each image to the output's aspect ratio and resizes it, so the runner passes first_frame and last_frame as they are instead of resizing them with ImageOps.fit first.
sp_runner forwarded /v1/videos requests over shared memory to video_runner.py, an MPI job spanning four Blackhole Galaxies. No workflow deploys it. Remove it together with what only it used: video_runner.py, the SHM transport (ipc/), the mock video runners, the continuous fan-out path in the device worker, the BaseDeviceRunner.requires_weights opt-out, the MODEL-name fallbacks for the I2V check and the executed step count, and the SP_MESH_4X32, video request timeout and SP warmup timeout settings.
Device discovery blocks while another process holds a chip, and on a mesh spanning several hosts the mesh open waits for every rank. Neither logged anything until it returned, so a hang there showed only as the previous step's line. Log before each.
…' into generations-api # Conflicts: # tt-media-server/docs/api/generation/README.md # tt-media-server/docs/api/generation/common.yaml # tt-media-server/docs/api/generation/models/minimaxai/minimax-h3-ref2va.yaml
Adapts the server to the spec merged from cglagovich/generation-api-spec:
- Model IDs: lightricks/ltx-2.3-22b-distilled, minimaxai/minimax-h3-fl2va,
tenstorrent/minimax-h3-fl2va-lightx2v, tenstorrent/minimax-h3-ref2va-lightx2v.
FastH3 moves under tenstorrent/ as minimax-h3-fl2va-fasth3, keeping its
own text-only schema. The superseded spec files are removed.
- MiniMax-H3: a `size` tier (768p) instead of width/height. FL2VA's
aspect_ratio is `adaptive` with a keyframe (the default then) and a ratio
without one. A reference video's soundtrack counts toward Ref2VA's audio
limits.
- LTX: aspect_ratio, size, duration and fps, each with its one served value,
instead of width, height and frames.
- Video outputs describe their soundtrack ({codec, sample_rate, channels}),
read from the written MP4, or null without one. Only the job fields that
don't apply yet are left out of responses.
- Media inputs: no more earlier-output URLs. Video clips must be MP4 or MOV
with H.264/H.265 and an optional AAC/MP3 track, audio clips WAV or MP3.
- The per-model cap on all data URIs together is dropped.
FastH3 is served text-to-video only, and with keyframes it is untested, so it cannot share FL2VA's request schema as the generation API's naming rule for derived variants requires. Drop it for now: its runner, deployment configs, request, spec and test-client routing.
89c366f dropped it along with the multihost SP path, but the quad runs through the lockstep launcher, which does not depend on sp_runner.
- Check the queue limits before loading a request's media, so a full server returns 429/503 without downloading and decoding first. - Fetch and check the inputs concurrently. A refused input (400, 413) stops the other fetches. - Keep the raw input bytes only where the runner reads them (H3, and video and audio inputs); the other runners get the decoded images. - Remove VideoAudioResult and requires_image_conditioning, which nothing produces or reads any more.
Swagger now shows what docs/api/generation/ says, apart from its notes on the earlier /v1/videos server: - Media, VideoMedia, AudioMedia, Seed and ImageFormat are named schemas with the spec's text, so a field's description no longer hides theirs. - Each submit route has the model's tag, summary, description and example request; every route declares the spec's responses, and the content route its file types. - The job, output and request schemas carry the spec's descriptions, size tables and examples. JobError.code lists the known codes, output `type` and `deleted` are required and `VideoOutput.audio` is optional.
Some hosts refuse httpx's default agent: Wikimedia answers 403, which the downloader reports as a likely expired presigned URL.
- A deployment's route metadata (model ID, tag, summary, description,
example) is a GenerationModel in domain/generation_models.py instead of
ClassVars on the request. The LightX2V deployments take their base
model's request, so their schemas have the spec's names.
- The spec marks optional request fields nullable, as the server accepts
null for them, and drops the photo/art_style oneOf, which rejected a
null for the other key; the description and the server keep the rule.
FLUX.2 klein's `images`, the one list that took null, is a list that
defaults to empty, as for the MiniMax-H3 Ref2VA references.
- The example requests use media that exist: an inline PNG and public
files on MDN and Wikimedia Commons, all within the models' limits.
- The content route answers 404 for a completed job whose result path is
gone, as GET /v1/jobs/{id} does, instead of 409 "Job is completed, not
completed". Its file types come from IMAGE_FORMATS.
# Conflicts: # test_module/benchmark_tests/video_benchmark_tests.py # tt-media-server/docs/api/generation/common.yaml # tt-media-server/docs/api/generation/models/black-forest-labs/flux-3-image-klein.yaml # tt-media-server/docs/api/generation/models/black-forest-labs/flux.2-klein-4b.yaml # tt-media-server/docs/api/generation/models/ideogram-ai/ideogram-4.yaml # tt-media-server/docs/api/generation/models/qwen/qwen-image-edit-2511.yaml # tt-media-server/tt_model_runners/dit_runners.py
The size tier says 1080p is 1920x1088, 8 rows more than 1080 because both sides must be multiples of 64.
The video endpoints the generations API replaced recorded video generation stats, an in-flight gauge and the requested aspect ratio and duration. The generation service now does the same for video jobs: success, failure and cancellation, with the executed step count and the output's size, frames, duration and bytes.
The spec counts a URL that isn't https as against the fetch policy (400). The request schema rejected it, which made it a 422; it now passes the schema and the fetch policy refuses it.
…the spec says - An input image that isn't PNG, JPEG or WebP, judged by its bytes, is a 422. A JPEG holding several pictures, as phones write one with a depth or gain map, is a JPEG. - The generations API refuses an http media URL with a 400, also after a redirect from https. The video API still follows such redirects. - A job past its expires_at is gone (404, not listed) before the retention sweep deletes it. Tests for those, and for what the spec defines and nothing tested: 401 on every route, 429 and 503 with Retry-After, cancel and lifecycle statuses, a failed or oversized fetch, the seed range, malformed data URIs, Wan's encoding and a list of jobs in every status.
CI runs them with pytest-asyncio 0.24, without torch or ffmpeg on PATH, and with the latest pydantic: - The generations_app fixture shuts its job managers down on the test's event loop, which pytest-asyncio 0.24 closes before a sync fixture's teardown. - VideoManager streams frames to the imageio_ffmpeg binary when PATH has no ffmpeg, as its other exports already do. - The OpenAPI dump stubs the runners' libraries that aren't installed. - The specs leave out an enum that only repeats its const: pydantic 2.10 and later don't write it. - ruff 0.15.0, CI's, formats the files these branches touch.
…WebP at 90, not PIL's lossier defaults
…f2VA label docs - Create the job's directory at submit; runners no longer recreate a deleted one - 413 names the cap that applied to a URL fetch - Output url is relative to the base URL (no leading slash) - Ref2VA prompt doc numbers Audio as the packer does; example uses Audio 2 - 400 says "A media URL" This addresses PR comments
…neration deployments
… pixels - Wan encodes its MP4 in the runner and returns the path, as H3 does, so the frames no longer cross the result queue to the API process. Warmup, the deep canary and follower ranks export nothing. - Only LTX remuxes for faststart, in its runner: tt-metal's generate() is the one writer that doesn't add +faststart. The service now only moves the runner's file into place. - For Wan and H3, whose runners decode their own images, submit still fully decodes each image (a corrupt one is a 422) but keeps only its EXIF-upright size in _image_sizes. Other requests keep _images too. - Data URI decoding and the stored request's SHA-256 digests run in a worker thread instead of on the event loop. Addresses review comments 6-9 on #5532.
What the image models share: the ImageFormat and Count fields, closest_ratio, writing an image output, and the image stage recorder. The models follow on their own branches.
POST /v1/generations/minimaxai/minimax-h3-image: text to image, or an edit of 1 to 16 reference images (`<Picture N>` in the prompt), as an async job. The request (domain/minimax_h3_image_request.py) serves 18 fixed sizes: 9 aspect ratios at `4k` (~16 MP, default) or `2k` (~4.2 MP); JPEG by default or PNG; count 1. Each reference is 128-8192 px a side with a ratio of at most 4:1, and together they hold at most H3_IMAGE_REFERENCE_AREA_LIMIT (48 MP, pending the Galaxy sweep) of effective area, each counted after its short side is scaled into 1024-2048 px; anything over is a 422 before queueing. TTMiniMaxH3ImageRunner builds tt-metal's MiniMaxH3ImagePipeline with that same limit, so the pipeline's arenas and warmup cover exactly what the API admits, and runs its warmup() at load. It decodes the references from their files (RUNNER_DECODES_IMAGES) and returns the PIL image, which the service writes in the request's format. A raw Hugging Face release snapshot is converted once on first start, under a file lock. The runner is generations-only: it is in MODEL_SERVICE_RUNNER_MAP's GENERATION set and in no other, so no /v1/images route serves it. The dev catalog entry (BLACKHOLE_GALAXY, EXPERIMENTAL) deploys it with MODEL_RUNNER=tt-minimax-h3-image; docs/api/openapi/minimaxai/ minimax-h3-image.yaml is the served spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verify, measured capacity) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iniMax-H3-Image overlaps its host tail The encode stage (video only so far) also takes a runner's PendingImage: it finishes it if the runner deferred its host tail, encodes it in the request's format, and sends the file's bytes (EncodedImage, ~5 MB at 4K) to the API process instead of 50 MB of pixels. All of it runs while the device works on the next request. The format table and save options move to domain/image_output.py so the stage and the service encode identically; the service still encodes a bare image itself. MiniMax-H3-Image opts in (encodes_images_off_thread); its encode stage turns on defer_image_finish, so the pipeline returns after the device work and the tile stitch runs on the stage. Back to back on a 4x8 Galaxy, 4K: t2i 2.77 -> 2.55 s per image (1299 -> 1410 img/h), 1-ref edit 4.65 -> 4.46 s; a 4K t2i completes 0.25 s sooner. Output is pixel-identical (served guide edit vs the tt-metal test's). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5531 (plus #5532's three image-output commits, cherry-picked); review only the last commit.
Why: H3-Image needs a per-model schema: 18 fixed sizes, 1–16 references, a total reference-area limit.
What:
POST /v1/generations/minimaxai/minimax-h3-image(request, runner, catalog, OpenAPI spec); not on/v1/images/*. Supersedes #5562.Tested: generations tests 177 passed; tt-media-server 1873 passed; pipeline on a 4x8 Galaxy.
🤖 Generated with Claude Code