Skip to content

Serve MiniMax-H3-Image through the generations API - #5563

Draft
cglagovichTT wants to merge 54 commits into
mainfrom
cglagovich/minimax-h3-image-generations
Draft

cglagovichTT wants to merge 54 commits into
mainfrom
cglagovich/minimax-h3-image-generations

Conversation

@cglagovichTT

Copy link
Copy Markdown
Contributor

Stacked on #5531 (plus #5532's three image-output commits, cherry-picked); review only the last commit.

Why: H3-Image needs a per-model schema: 18 fixed sizes, 1–16 references, a total reference-area limit.

What: POST /v1/generations/minimaxai/minimax-h3-image (request, runner, catalog, OpenAPI spec); not on /v1/images/*. Supersedes #5562.

Tested: generations tests 177 passed; tt-media-server 1873 passed; pipeline on a 4x8 Galaxy.

🤖 Generated with Claude Code

cglagovichTT and others added 30 commits October 26, 2026 09:34
OpenAPI design for tt-media-server's per-model generation API: one submit
endpoint per model, POST /v1/generations/{org}/{model}, and a common
/v1/jobs API. Covers Wan2.2 I2V A14B, Qwen-Image-Edit-2511, FLUX.2 klein 4B,
FLUX.3 Image klein and Ideogram 4.

common.yaml holds the jobs API and shared schemas; models/{org}/{model}.yaml
is one complete spec per model deployment, so each can be replaced by the
spec generated from its Pydantic models as it is implemented.
REQUIREMENTS.md records runner behavior the spec leaves out, and follow-ups.

Docs only; no code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
{job_id} and {index} are not valid in a URI fragment; Scalar warned on
these $refs. Encode them as %7B/%7D (resolves the same).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- minimaxai/minimax-h3-fl2va and minimaxai/minimax-h3-ref2va, plus
  tenstorrent/*-lightx2v variants that reuse the base request schema on
  their own routes
- lightricks/ltx-2.3-22b-distilled: one served shape (1920x1080, 6 s,
  25 fps), shape fields kept for later
- common: VideoMedia and AudioMedia inputs, optional audio track on
  VideoOutput, per-type size limits
- REQUIREMENTS: H3 adaptive ratios, input limits, variants; LTX crop to
  1080 in tt-metal; follow-ups for LTX image-to-video and H3 size tiers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PR now holds only the API spec and its README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses review: a reference video's soundtrack is audio conditioning, so it
counts toward the 3-audio and 15 s audio limits; a video with sound still
counts once toward the 12-reference total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per review: the references don't set the shape in today's stack (or SGLang),
so Ref2VA takes a ratio like text-to-video does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Media

Addresses review:
- every model document now declares components.securitySchemes.Bearer
  (referencing common.yaml), as OpenAPI 3.1 requires for its security
  requirement
- Media no longer says an earlier output can be passed by its url: output
  urls are internal runner paths the fetch policy would refuse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the "Elsewhere" comparisons with other providers. Where one also said
how tt-media-server behaves today, keep that part as "tt-media-server today".
Only descriptions change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implements the per-model generation API designed in docs/api/generation for
Wan2.2 I2V A14B, Qwen-Image-Edit-2511, FLUX.2 klein 4B and Ideogram 4: one
submit endpoint per model, POST /v1/generations/{org}/{model}, with the
model's own Pydantic request, and the common /v1/jobs API (status with
outputs and expires_at, content, cancel, delete). A deployment mounts only
the submit endpoint of the model it serves. Each request model is checked
against its spec file in the tests.

- Runners consume the requests directly. The tt_dit pipelines take the output
  size per call, so a deployment serves every size its schema allows; the
  pipeline rejects a size its mesh preset doesn't serve. Warmup covers every
  size and input combination the pipeline doesn't warm up itself.
- Input images are https URLs fetched at submit through media_downloader,
  data URIs, or an earlier output's url. A refused URL is 400, an image over
  30 MB is 413, everything else 422.
- Outputs are written to one directory per job with an outputs.json manifest,
  under TT_GENERATIONS_DIR, and deleted with the job.
- count is fixed at 1, as the spec has it.
- Wan2.2 I2V A14B moves from /v1/videos to the generations API, and its
  multihost SP path is dropped. test_module's Wan I2V clients use the new API.
LTX-2.3 distilled moves from /v1/videos to POST
/v1/generations/lightricks/ltx-2.3-distilled and /v1/jobs.

- The request keeps what LTX served before: prompt and seed, and width,
  height, frames, duration and fps, each allowing only the one shape the
  pipeline is built and traced for (1920x1088, 153 frames at 25 fps).
  Steps and the ignored negative prompt are gone.
- Video outputs report `audio`; LTX's MP4 has an AAC soundtrack. The service
  writes frames, frames with a soundtrack, or an MP4 the pipeline wrote,
  remuxed for faststart.
- A generation job's directory is its result path from submit on. LTX's
  runner writes its video into it, so a file a cancelled or failed job
  leaves behind is deleted with the job.
- The LTX runner is one class in generation_runners.py; the old one, LTX's
  branch of VideoGenerateRequest and its shape helpers are removed.
- start_ltx_fast.sh sets TT_GENERATIONS_DIR and no longer SERVED_MODEL_NAME,
  which the generation API's /v1/models does not use. The runbooks use the
  new routes.
The five MiniMax-H3 deployments move from /v1/videos to the generation
API and /v1/jobs:

- POST /v1/generations/minimaxai/minimax-h3 (FL2VA: optional first and
  last frame)
- POST /v1/generations/minimaxai/minimax-h3-fasth3 (text only)
- POST /v1/generations/minimaxai/minimax-h3-ref2va (reference images,
  videos and audio clips)
- POST /v1/generations/lightx2v/minimax-h3-turbo
- POST /v1/generations/lightx2v/minimax-h3-turbo-ref2va

- The requests keep what H3 served before: prompt, aspect_ratio or width
  and height, duration (4 to 15 s, at 24 fps), seed, and the keyframe and
  reference limits. Ratios accept "16:9", "16x9" and "16/9"; a blank
  prompt is rejected. The multipart upload route, synchronous mode and the
  seed defaulting to 0 are gone, as is the T2VA runner, a strict subset of
  FL2VA.
- Inputs can be videos and audio clips as well as images. The service
  loads and probes them and checks the limits up front (422 per input);
  H3 caps inline data URIs per model (413). The device and the multihost
  ranks get the input files, not decoded pixels, and the runner decodes
  them.
- H3's runner writes the MP4 with its soundtrack into the job's directory.
  The deep canary runs the shortest video on every rank and exports
  nothing.
- The Turbo runners extend the FL2VA and Ref2VA runners.
- The (4, 32) quad keeps the lockstep runner; the request round-trips
  through it with its input files.
- The old H3 request types, admission policy, metrics, the
  VideoAudioResult branch of the encode stage and the H3 doc are removed;
  the video runners return to their pre-H3 form.
- The specs, README tables and the test_module video routing cover the
  five models.
WanPipeline.__call__ takes no num_frames; the frame count is set once in
create_pipeline, which the runner already passes. Every request, warmup
included, failed with "got an unexpected keyword argument 'num_frames'".
The tt_dit pipelines load their own weights, from a Hugging Face repo id
or a local directory, so a generation deployment never downloads them in
the server:

- BaseService downloads only when its DOWNLOADS_WEIGHTS flag is set;
  GenerationService clears it.
- The runners' _weights_dir and its snapshot_download fallback are gone.
  The Ideogram 4 runner passes the weights path through; its pipeline
  resolves a repo id itself.
- The LTX and MiniMax-H3 runners no longer override load_weights.
The pipeline center-crops each image to the output's aspect ratio and
resizes it, so the runner passes first_frame and last_frame as they are
instead of resizing them with ImageOps.fit first.
sp_runner forwarded /v1/videos requests over shared memory to video_runner.py,
an MPI job spanning four Blackhole Galaxies. No workflow deploys it. Remove it
together with what only it used: video_runner.py, the SHM transport (ipc/), the
mock video runners, the continuous fan-out path in the device worker, the
BaseDeviceRunner.requires_weights opt-out, the MODEL-name fallbacks for the I2V
check and the executed step count, and the SP_MESH_4X32, video request timeout
and SP warmup timeout settings.
Device discovery blocks while another process holds a chip, and on a mesh
spanning several hosts the mesh open waits for every rank. Neither logged
anything until it returned, so a hang there showed only as the previous step's
line. Log before each.
…' into generations-api

# Conflicts:
#	tt-media-server/docs/api/generation/README.md
#	tt-media-server/docs/api/generation/common.yaml
#	tt-media-server/docs/api/generation/models/minimaxai/minimax-h3-ref2va.yaml
Adapts the server to the spec merged from cglagovich/generation-api-spec:

- Model IDs: lightricks/ltx-2.3-22b-distilled, minimaxai/minimax-h3-fl2va,
  tenstorrent/minimax-h3-fl2va-lightx2v, tenstorrent/minimax-h3-ref2va-lightx2v.
  FastH3 moves under tenstorrent/ as minimax-h3-fl2va-fasth3, keeping its
  own text-only schema. The superseded spec files are removed.
- MiniMax-H3: a `size` tier (768p) instead of width/height. FL2VA's
  aspect_ratio is `adaptive` with a keyframe (the default then) and a ratio
  without one. A reference video's soundtrack counts toward Ref2VA's audio
  limits.
- LTX: aspect_ratio, size, duration and fps, each with its one served value,
  instead of width, height and frames.
- Video outputs describe their soundtrack ({codec, sample_rate, channels}),
  read from the written MP4, or null without one. Only the job fields that
  don't apply yet are left out of responses.
- Media inputs: no more earlier-output URLs. Video clips must be MP4 or MOV
  with H.264/H.265 and an optional AAC/MP3 track, audio clips WAV or MP3.
- The per-model cap on all data URIs together is dropped.
FastH3 is served text-to-video only, and with keyframes it is untested, so it
cannot share FL2VA's request schema as the generation API's naming rule for
derived variants requires. Drop it for now: its runner, deployment configs,
request, spec and test-client routing.
89c366f dropped it along with the multihost SP path, but the quad runs
through the lockstep launcher, which does not depend on sp_runner.
- Check the queue limits before loading a request's media, so a full
  server returns 429/503 without downloading and decoding first.
- Fetch and check the inputs concurrently. A refused input (400, 413)
  stops the other fetches.
- Keep the raw input bytes only where the runner reads them (H3, and
  video and audio inputs); the other runners get the decoded images.
- Remove VideoAudioResult and requires_image_conditioning, which nothing
  produces or reads any more.
Swagger now shows what docs/api/generation/ says, apart from its notes on
the earlier /v1/videos server:

- Media, VideoMedia, AudioMedia, Seed and ImageFormat are named schemas
  with the spec's text, so a field's description no longer hides theirs.
- Each submit route has the model's tag, summary, description and example
  request; every route declares the spec's responses, and the content
  route its file types.
- The job, output and request schemas carry the spec's descriptions, size
  tables and examples. JobError.code lists the known codes, output `type`
  and `deleted` are required and `VideoOutput.audio` is optional.
Some hosts refuse httpx's default agent: Wikimedia answers 403, which the
downloader reports as a likely expired presigned URL.
- A deployment's route metadata (model ID, tag, summary, description,
  example) is a GenerationModel in domain/generation_models.py instead of
  ClassVars on the request. The LightX2V deployments take their base
  model's request, so their schemas have the spec's names.
- The spec marks optional request fields nullable, as the server accepts
  null for them, and drops the photo/art_style oneOf, which rejected a
  null for the other key; the description and the server keep the rule.
  FLUX.2 klein's `images`, the one list that took null, is a list that
  defaults to empty, as for the MiniMax-H3 Ref2VA references.
- The example requests use media that exist: an inline PNG and public
  files on MDN and Wikimedia Commons, all within the models' limits.
- The content route answers 404 for a completed job whose result path is
  gone, as GET /v1/jobs/{id} does, instead of 409 "Job is completed, not
  completed". Its file types come from IMAGE_FORMATS.
# Conflicts:
#	test_module/benchmark_tests/video_benchmark_tests.py
#	tt-media-server/docs/api/generation/common.yaml
#	tt-media-server/docs/api/generation/models/black-forest-labs/flux-3-image-klein.yaml
#	tt-media-server/docs/api/generation/models/black-forest-labs/flux.2-klein-4b.yaml
#	tt-media-server/docs/api/generation/models/ideogram-ai/ideogram-4.yaml
#	tt-media-server/docs/api/generation/models/qwen/qwen-image-edit-2511.yaml
#	tt-media-server/tt_model_runners/dit_runners.py
friedrich and others added 22 commits November 4, 2026 14:20
The size tier says 1080p is 1920x1088, 8 rows more than 1080 because both
sides must be multiples of 64.
The video endpoints the generations API replaced recorded video generation
stats, an in-flight gauge and the requested aspect ratio and duration. The
generation service now does the same for video jobs: success, failure and
cancellation, with the executed step count and the output's size, frames,
duration and bytes.
The spec counts a URL that isn't https as against the fetch policy (400).
The request schema rejected it, which made it a 422; it now passes the
schema and the fetch policy refuses it.
…the spec says

- An input image that isn't PNG, JPEG or WebP, judged by its bytes, is a
  422. A JPEG holding several pictures, as phones write one with a depth
  or gain map, is a JPEG.
- The generations API refuses an http media URL with a 400, also after a
  redirect from https. The video API still follows such redirects.
- A job past its expires_at is gone (404, not listed) before the
  retention sweep deletes it.

Tests for those, and for what the spec defines and nothing tested: 401
on every route, 429 and 503 with Retry-After, cancel and lifecycle
statuses, a failed or oversized fetch, the seed range, malformed data
URIs, Wan's encoding and a list of jobs in every status.
CI runs them with pytest-asyncio 0.24, without torch or ffmpeg on PATH,
and with the latest pydantic:

- The generations_app fixture shuts its job managers down on the test's
  event loop, which pytest-asyncio 0.24 closes before a sync fixture's
  teardown.
- VideoManager streams frames to the imageio_ffmpeg binary when PATH has
  no ffmpeg, as its other exports already do.
- The OpenAPI dump stubs the runners' libraries that aren't installed.
- The specs leave out an enum that only repeats its const: pydantic 2.10
  and later don't write it.
- ruff 0.15.0, CI's, formats the files these branches touch.
…f2VA label docs

- Create the job's directory at submit; runners no longer recreate a deleted one
- 413 names the cap that applied to a URL fetch
- Output url is relative to the base URL (no leading slash)
- Ref2VA prompt doc numbers Audio as the packer does; example uses Audio 2
- 400 says "A media URL"

This addresses PR comments
… pixels

- Wan encodes its MP4 in the runner and returns the path, as H3 does,
  so the frames no longer cross the result queue to the API process.
  Warmup, the deep canary and follower ranks export nothing.
- Only LTX remuxes for faststart, in its runner: tt-metal's generate()
  is the one writer that doesn't add +faststart. The service now only
  moves the runner's file into place.
- For Wan and H3, whose runners decode their own images, submit still
  fully decodes each image (a corrupt one is a 422) but keeps only its
  EXIF-upright size in _image_sizes. Other requests keep _images too.
- Data URI decoding and the stored request's SHA-256 digests run in a
  worker thread instead of on the event loop.

Addresses review comments 6-9 on #5532.
What the image models share: the ImageFormat and Count fields, closest_ratio,
writing an image output, and the image stage recorder. The models follow on
their own branches.
POST /v1/generations/minimaxai/minimax-h3-image: text to image, or an edit
of 1 to 16 reference images (`<Picture N>` in the prompt), as an async job.
The request (domain/minimax_h3_image_request.py) serves 18 fixed sizes:
9 aspect ratios at `4k` (~16 MP, default) or `2k` (~4.2 MP); JPEG by
default or PNG; count 1. Each reference is 128-8192 px a side with a ratio
of at most 4:1, and together they hold at most
H3_IMAGE_REFERENCE_AREA_LIMIT (48 MP, pending the Galaxy sweep) of
effective area, each counted after its short side is scaled into
1024-2048 px; anything over is a 422 before queueing.

TTMiniMaxH3ImageRunner builds tt-metal's MiniMaxH3ImagePipeline with that
same limit, so the pipeline's arenas and warmup cover exactly what the
API admits, and runs its warmup() at load. It decodes the references from
their files (RUNNER_DECODES_IMAGES) and returns the PIL image, which the
service writes in the request's format. A raw Hugging Face release snapshot
is converted once on first start, under a file lock.

The runner is generations-only: it is in MODEL_SERVICE_RUNNER_MAP's
GENERATION set and in no other, so no /v1/images route serves it. The dev
catalog entry (BLACKHOLE_GALAXY, EXPERIMENTAL) deploys it with
MODEL_RUNNER=tt-minimax-h3-image; docs/api/openapi/minimaxai/
minimax-h3-image.yaml is the served spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verify, measured capacity)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iniMax-H3-Image overlaps its host tail

The encode stage (video only so far) also takes a runner's PendingImage: it finishes it if the runner deferred its
host tail, encodes it in the request's format, and sends the file's bytes (EncodedImage, ~5 MB at 4K) to the API
process instead of 50 MB of pixels. All of it runs while the device works on the next request. The format table and
save options move to domain/image_output.py so the stage and the service encode identically; the service still
encodes a bare image itself.

MiniMax-H3-Image opts in (encodes_images_off_thread); its encode stage turns on defer_image_finish, so the pipeline
returns after the device work and the tile stitch runs on the stage. Back to back on a 4x8 Galaxy, 4K: t2i 2.77 ->
2.55 s per image (1299 -> 1410 img/h), 1-ref edit 4.65 -> 4.46 s; a 4K t2i completes 0.25 s sooner. Output is
pixel-identical (served guide edit vs the tt-metal test's).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants