Repository navigation
Description
Summary
While verifying tt-media-server/RUN_SDXL_FORGE_N150.md on a Wormhole N150 (wormhole_b0), two things came out:
- ✅ The documented 512px Forge path works end-to-end and produces a correct image.
⚠️ A server started withoutTTXLA_SDXL_RESOLUTION(resolvingmodel_weights_path=stabilityai/stable-diffusion-xl-base-1.0) did not return a generation result withinrequest_processing_timeout_seconds(default 1000) and respondedHTTP 500 {"detail":""}— i.e. the default/unconfigured serving path is unusable on N150 as-is. (See caveat in Finding 2 — this run had a confounder and needs a clean repro.)
Environment
- Host: N150 (
wormhole_b0 / n150), KMD 2.4.1, FW bundle 18.12.1 - Branch
feat/sdxl-forge-dashboard-onboarding-main@d74b043c - Runner:
tt-xla-sdxl→tt_model_runners/forge_runners/sdxl_forge_runner.py, served via tt-media-server (image service) - SFPI toolchain 7.30.0[308] (venv-local),
TT_METAL_HOMEwritable, kernel cache warm
Finding 1 — documented 512px path ✅ VERIFIED
Ran the doc's Step 2A smoke test exactly as written:
export MODEL_RUNNER=tt-xla-sdxl
export TTXLA_SDXL_RESOLUTION=512
export TT_METAL_HOME=/proj_sw/.../tt_metal_home
python smoke_test_sdxl.py
Result — clean success, valid 512×512 RGB PNG:
Device 0: Loading models from hotshotco/SDXL-512 (resolution=512, variant=None, full_on_device=False)
[warmup] async executed in 485.3027 seconds (UNet compile 347.3s, VAE decode 4.7s)
[smoke] warmup OK in 485.3s
Device 0: Text encoding took 1.1023s
Device 0: UNet diffusion (20 steps) took 11.6066s
Device 0: VAE decode took 3.4504s (full_on_device=False)
[run] executed in 16.1680 seconds
[smoke] inference OK in 16.2s, got 1 image(s)
[smoke] saved smoke_test_sdxl_n150.png ((512, 512))
[smoke] TOTAL 501.6s -- SUCCESS (Exit code: 0)
- Steady-state inference 16.2 s/image (text 1.1 s + UNet 20 steps 11.6 s + VAE 3.45 s) — matches the doc's ~15.5 s claim.
- Warmup here was 485 s (~8 min), faster than the doc's ~14.5 min because the metal kernel cache was already warm; first-ever warmup is longer.
- Output is a coherent image (astronaut on the moon), not noise.
Finding 2 — default-config server times out → HTTP 500 ⚠️
A server was running started with only MODEL_RUNNER=tt-xla-sdxl + TT_METAL_HOME (no TTXLA_SDXL_RESOLUTION, no MODEL/DEVICE). Its resolved settings were:
Settings resolved: model_runner='tt-xla-sdxl', model_service='image',
model_weights_path='stabilityai/stable-diffusion-xl-base-1.0', max_batch_size=1
A standard request (num_inference_steps=20, guidance_scale=7.5, seed=42) returned:
HTTP 500 13 bytes 1000.003494s
real 16m40.010s
body: {"detail":""}
- The 1000.003 s wall time exactly equals the default
request_processing_timeout_seconds: int = 1000inconfig/settings.py:78(timeout enforced inmodel_services/base_service.py:152-164/device_workers/device_worker.py:155-163). So this is the server-side request timeout firing, returning a bare500 {"detail":""}. - Throughout the request the device stayed near idle (~23 W, idle ≈ 20 W) at full AICLK (1000 MHz) → the work was host-bound (CPU-side VAE/CLIP + scheduler on the heavier
stable-diffusion-xl-base-1.0model), not device-bound.
Caveat / confounder (needs clean repro): during this session there was briefly request contention — a second, stale curl was left connected to the single-worker server before the measured request. A single worker = one chip, so the measured request may have queued behind orphaned work. The 1000 s figure should be reproduced with a single, isolated request before treating the exact latency as authoritative. What is solid: the default-config server did not produce an image and returned a 500.
Open questions / observations
SDXLForgeRunner.DEFAULT_RESOLUTION = 512(sdxl_forge_runner.py:27), yet the unconfigured server resolvedmodel_weights_path=stabilityai/stable-diffusion-xl-base-1.0(the 1024px base model). The relationship betweensettings.model_weights_pathand the runner's resolution selection whenTTXLA_SDXL_RESOLUTIONis unset is worth confirming — there may be a mismatch where the heavy base-1.0 model is selected even though the runner defaults to 512.- A 500 with an empty
{"detail":""}body on timeout is not actionable for clients — consider returning504/408with a message naming the timeout.
Suggested follow-ups
- Reproduce Finding 2 with a single isolated request on a freshly-started default-config server to confirm the timeout (and whether base-1.0 ever completes within any reasonable bound on N150).
- Clarify/repair the
model_weights_pathvs runner-resolution default whenTTXLA_SDXL_RESOLUTIONis unset. - Improve the timeout response (status code + body).
- Consider documenting in
RUN_SDXL_FORGE_N150.mdthatTTXLA_SDXL_RESOLUTIONis effectively required for a usable N150 serving config.
Verified by running on N150 on 2026-06-15.
Activity