Skip to content

SDXL Forge (N150): default server config returns HTTP 500 at 1000s request timeout; 512px path verified working #4208

Open

Description

@ctr-lelanchelianTT

Summary

While verifying tt-media-server/RUN_SDXL_FORGE_N150.md on a Wormhole N150 (wormhole_b0), two things came out:

  1. ✅ The documented 512px Forge path works end-to-end and produces a correct image.
  2. ⚠️ A server started without TTXLA_SDXL_RESOLUTION (resolving model_weights_path=stabilityai/stable-diffusion-xl-base-1.0) did not return a generation result within request_processing_timeout_seconds (default 1000) and responded HTTP 500 {"detail":""} — i.e. the default/unconfigured serving path is unusable on N150 as-is. (See caveat in Finding 2 — this run had a confounder and needs a clean repro.)

Environment

  • Host: N150 (wormhole_b0 / n150), KMD 2.4.1, FW bundle 18.12.1
  • Branch feat/sdxl-forge-dashboard-onboarding-main @ d74b043c
  • Runner: tt-xla-sdxl → tt_model_runners/forge_runners/sdxl_forge_runner.py, served via tt-media-server (image service)
  • SFPI toolchain 7.30.0[308] (venv-local), TT_METAL_HOME writable, kernel cache warm

Finding 1 — documented 512px path ✅ VERIFIED

Ran the doc's Step 2A smoke test exactly as written:

export MODEL_RUNNER=tt-xla-sdxl
export TTXLA_SDXL_RESOLUTION=512
export TT_METAL_HOME=/proj_sw/.../tt_metal_home
python smoke_test_sdxl.py

Result — clean success, valid 512×512 RGB PNG:

Device 0: Loading models from hotshotco/SDXL-512 (resolution=512, variant=None, full_on_device=False)
[warmup] async executed in 485.3027 seconds   (UNet compile 347.3s, VAE decode 4.7s)
[smoke] warmup OK in 485.3s
Device 0: Text encoding took 1.1023s
Device 0: UNet diffusion (20 steps) took 11.6066s
Device 0: VAE decode took 3.4504s (full_on_device=False)
[run] executed in 16.1680 seconds
[smoke] inference OK in 16.2s, got 1 image(s)
[smoke] saved smoke_test_sdxl_n150.png ((512, 512))
[smoke] TOTAL 501.6s -- SUCCESS  (Exit code: 0)
  • Steady-state inference 16.2 s/image (text 1.1 s + UNet 20 steps 11.6 s + VAE 3.45 s) — matches the doc's ~15.5 s claim.
  • Warmup here was 485 s (~8 min), faster than the doc's ~14.5 min because the metal kernel cache was already warm; first-ever warmup is longer.
  • Output is a coherent image (astronaut on the moon), not noise.

Finding 2 — default-config server times out → HTTP 500 ⚠️

A server was running started with only MODEL_RUNNER=tt-xla-sdxl + TT_METAL_HOME (no TTXLA_SDXL_RESOLUTION, no MODEL/DEVICE). Its resolved settings were:

Settings resolved: model_runner='tt-xla-sdxl', model_service='image',
model_weights_path='stabilityai/stable-diffusion-xl-base-1.0', max_batch_size=1

A standard request (num_inference_steps=20, guidance_scale=7.5, seed=42) returned:

HTTP 500   13 bytes   1000.003494s
real  16m40.010s
body: {"detail":""}
  • The 1000.003 s wall time exactly equals the default request_processing_timeout_seconds: int = 1000 in config/settings.py:78 (timeout enforced in model_services/base_service.py:152-164 / device_workers/device_worker.py:155-163). So this is the server-side request timeout firing, returning a bare 500 {"detail":""}.
  • Throughout the request the device stayed near idle (~23 W, idle ≈ 20 W) at full AICLK (1000 MHz) → the work was host-bound (CPU-side VAE/CLIP + scheduler on the heavier stable-diffusion-xl-base-1.0 model), not device-bound.

Caveat / confounder (needs clean repro): during this session there was briefly request contention — a second, stale curl was left connected to the single-worker server before the measured request. A single worker = one chip, so the measured request may have queued behind orphaned work. The 1000 s figure should be reproduced with a single, isolated request before treating the exact latency as authoritative. What is solid: the default-config server did not produce an image and returned a 500.

Open questions / observations

  • SDXLForgeRunner.DEFAULT_RESOLUTION = 512 (sdxl_forge_runner.py:27), yet the unconfigured server resolved model_weights_path=stabilityai/stable-diffusion-xl-base-1.0 (the 1024px base model). The relationship between settings.model_weights_path and the runner's resolution selection when TTXLA_SDXL_RESOLUTION is unset is worth confirming — there may be a mismatch where the heavy base-1.0 model is selected even though the runner defaults to 512.
  • A 500 with an empty {"detail":""} body on timeout is not actionable for clients — consider returning 504/408 with a message naming the timeout.

Suggested follow-ups

  1. Reproduce Finding 2 with a single isolated request on a freshly-started default-config server to confirm the timeout (and whether base-1.0 ever completes within any reasonable bound on N150).
  2. Clarify/repair the model_weights_path vs runner-resolution default when TTXLA_SDXL_RESOLUTION is unset.
  3. Improve the timeout response (status code + body).
  4. Consider documenting in RUN_SDXL_FORGE_N150.md that TTXLA_SDXL_RESOLUTION is effectively required for a usable N150 serving config.

Verified by running on N150 on 2026-06-15.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions