You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Forge SDXL (impl=sdxl-forge, TT_XLA_SDXL) bring-up on a single Wormhole N150 in the nightly CI. Tracks the timeout fix and the remaining spec-test gaps. Related: #4208.
Update — what we did today
Root-caused the nightly "requests sent, zero images, timeout at 3000s" failure: the image eval dispatches all 100 prompts concurrently, but a single N150 serves them serially (max_batch_size=1, ~30s/img effective), so the queue tail hit the 3000s per-request timeout → cascaded into the 6h job cap. Config/resolution were never the problem.
Fix: bumped request_processing_timeout_seconds 3000 → 10000 for (TT_XLA_SDXL, N150).
Re-ran nightly: 225 images generated, zero timeouts, Evals PASS (baseline FID 187.94, CLIP 31.996±2.97) and Benchmarks PASS, ~1h wall-clock (was hitting the 6h cap).
Removed an unbacked P150X8 device from the forge SDXL CI config + spec (no (TT_XLA_SDXL, P150X8) in constants.py).
Remaining (still red)
LoRA not implemented in SDXLForgeRunner → LoRA eval FID == baseline (adapter ignored). Needs the forge LoRA follow-up.
A couple of perf/param thresholds tuned for the faster trace path (load test 11.26s vs 10s; param SSIM 0.9813 vs <0.98).
Activity
Milestone-0 Blackhole-Galaxy readiness work (#4911)
forge SDXL: recalibrate N150 spec-test thresholds for the Forge path …