Skip to content

Forge SDXL on N150 (nightly): timeout fixed; LoRA + perf thresholds remain #4352

Open

Description

@ctr-lelanchelianTT

Description

Forge SDXL (impl=sdxl-forge, TT_XLA_SDXL) bring-up on a single Wormhole N150 in the nightly CI. Tracks the timeout fix and the remaining spec-test gaps. Related: #4208.

Update — what we did today

  • Root-caused the nightly "requests sent, zero images, timeout at 3000s" failure: the image eval dispatches all 100 prompts concurrently, but a single N150 serves them serially (max_batch_size=1, ~30s/img effective), so the queue tail hit the 3000s per-request timeout → cascaded into the 6h job cap. Config/resolution were never the problem.
  • Fix: bumped request_processing_timeout_seconds 3000 → 10000 for (TT_XLA_SDXL, N150).
  • Re-ran nightly: 225 images generated, zero timeouts, Evals PASS (baseline FID 187.94, CLIP 31.996±2.97) and Benchmarks PASS, ~1h wall-clock (was hitting the 6h cap).
  • Removed an unbacked P150X8 device from the forge SDXL CI config + spec (no (TT_XLA_SDXL, P150X8) in constants.py).

Remaining (still red)

  • LoRA not implemented in SDXLForgeRunner → LoRA eval FID == baseline (adapter ignored). Needs the forge LoRA follow-up.
  • A couple of perf/param thresholds tuned for the faster trace path (load test 11.26s vs 10s; param SSIM 0.9813 vs <0.98).

Links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions