[AI][BUG][Iluvatar] Non-deterministic results across repeated identical forward passes (vision tower reproduces it standalone; NaN on some samples) #617

Summary

On Iluvatar BI-V150 (iluvatar backend, torch_fl 2.10.0+corex4.5.0), a repeated forward pass over the same input tensors in the same process does not return the same result. Running the same sample 4 times in one process gives e.g.

0.99418   0.87493   0.99427   0.99427        # prob_1, same batch, same weights, same input tensors

The same model, same weights, same input on the same machine with the vendor DTK torch (2.10.0+corex.4.5.0) is bit-identical across repeated runs (0.99539042 × 3). End to end this costs us accuracy: 4 of 12 reference samples fall outside tolerance, including two NaNs.

I isolated it further: the vision tower alone reproduces it (visual(...) called 4× with the same pixel_values), while a synthetic transformer built from the same primitives is bit-identical, and each individual op I tested (mm / bmm / softmax / layer_norm / rms_norm / conv1d / cumsum / sdpa / exp / sum / mean) is bit-identical across repeats.

The model weights are internal, so I cannot hand you a one-command repro. I have the hardware and can run any diagnostic probe you ask for (see "What I can run for you").

Environment

Container (three bind-mounts are needed on this host — see notes at the end):

H=/usr/local/corex-4.5.0.20260509
IMG=harbor.baai.ac.cn/flagos-dev/pytorch-plugin-fl@sha256:bc1b3c0beb22b301ec501896ea1c77903879526fb7303358669321d609458023

docker run -d --name tfl --privileged --ipc=host --net=host --shm-size 64g \
  -v /dev:/dev -v /lib/modules:/lib/modules -v /sys:/sys \
  -v $H:/opt/host-corex45:ro \
  -v $H/lib64/libcuda.so.1:/usr/local/corex/lib64/libcuda.so.1:ro \
  -v $H/lib64/libixml.so:/usr/local/corex/lib64/libixml.so:ro \
  -v $H/bin/ixsmi:/usr/local/corex/bin/ixsmi:ro \
  -e LD_LIBRARY_PATH=/opt/host-corex45/lib64:/usr/local/corex/lib64:/usr/local/openmpi/lib:/usr/local/lib \
  --entrypoint /bin/bash $IMG -c 'sleep infinity'

Build torch_fl (the image does not ship it):

docker exec tfl bash -lc '
  source /opt/torch-fl-iluvatar-venv/bin/activate
  cd /workspace/Torch-FL        # GitHub clone of flagos-ai/Torch-FL
  FLAGOS_ACCELERATOR=iluvatar COREX_PATH=/usr/local/corex \
    pip install --no-build-isolation --no-deps -e .'
# → Successfully installed torch_fl-2.10.0+corex4.5.0

Model: a 2B Qwen3.5-VL checkpoint (Qwen3_5ForSequenceClassification) in bf16, single forward pass, attn_implementation="sdpa", two images + ~240 text tokens per sample.

Repro shape

import torch_fl, torch
from transformers import AutoProcessor, Qwen3_5ForSequenceClassification

proc = AutoProcessor.from_pretrained(CKPT)
model = Qwen3_5ForSequenceClassification.from_pretrained(
    CKPT, dtype=torch.bfloat16, attn_implementation="sdpa", device_map=None
).to("flagos:0").eval()

inputs = proc(text=[prompt], images=[img1, img2], return_tensors="pt")
inputs = {k: v.to("flagos:0") for k, v in inputs.items() if hasattr(v, "to")}

with torch.inference_mode():
    for _ in range(4):                       # same `inputs` object, no resampling
        logits = model(**inputs).logits.float()
        print(torch.softmax(logits, -1)[0, 1].item())

Full model — repeated runs, bf16, one process

Other repeats on another sample (s12, reference 0.9124):

0.70335  0.71143  0.71143
0.75092  0.70335  0.70335

Note it is not always the first call that differs — in the first table it is run2, in the vision-tower test below it is run0. There is no stable ordering; that is what makes it look like stale/uninitialised memory rather than a warm-up path.

12 reference samples (bf16)

The four bad samples are s08 (formula, 246 tokens), s10 (table, 696), s11 (table, 428), s12 (table, 551) — so prompt length alone does not predict failure; all 8 samples with 222–434 tokens except s08/s11 are fine. The two NaNs appear only when samples are run in sequence; running s10 alone gives 0.983 — i.e. there is also state leaking across calls.

Vendor torch on the same machine, same batch: 12/12 within tolerance, max |Δ| = 8.65e-3.

Isolation so far

Vision tower alone is non-deterministic

mm = proc.image_processor(images=[img1, img2], return_tensors="pt")
mm = {k: v.to("flagos:0") for k, v in mm.items() if hasattr(v, "to")}

with torch.inference_mode():
    for i in range(6):
        out = model.model.visual(mm["pixel_values"], grid_thw=mm["image_grid_thw"])
        e = out.last_hidden_state
        torch.flagos.synchronize()
        f = e.float().cpu()
        print(i, f.mean(), f.std())

run 0   mean 1.11808   std 79.563
run 1   mean 1.10983   std 79.137
run 2   mean 1.10983   std 79.137     ← runs 1..5 are bit-identical to each other
...

grid_thw = [[1,10,28],[1,6,46]], pixel_values shape (556, 1536).

This goes through the vision tower's variable-length attention path: transformers splits q/k/v per image with torch.split(tensor, lengths.tolist(), dim=2) and runs one attention call per image. With attn_implementation="eager" the same non-determinism appears with one image (run0 1.09946 vs runs 1–3 1.09230), and with sdpa + one image all four runs differ (1.09019 / 1.05524 / 1.06326 / 1.05995) — so the pattern is not tied to sdpa and not strictly "first call only".

Vision tower also deviates from vendor torch

Same pixel_values, one forward after a warm-up call, element-wise vs vendor torch:

shape (556, 1024)
vendor: mean 1.08775  std 77.352  absmax 5952
tfl   : mean 1.11767  std 79.532  absmax 6016
|Δ|   : max 5207   mean 2.079   (2.7 % of std)
per-token mean error: median 0.495, max 29.5   (std 77.4)
tokens with per-token mean error > 0.5·std: 0 / 556

So the vision tower output differs by ~2.7 % of its std, spread over all 556 tokens and all 1024 channels — not a localized channel/block. For bf16 I would expect this to be near the rounding floor.

What is not affected

What I suspect / what would help

1. Stale or partially-initialised output buffer. Simple ops are stable under repetition, but a long alternating sequence of different kernels is not — consistent with an allocator/stream reusing device memory before the producing kernel has retired (despite synchronize() between blocks), or a kernel that does not write every output element. Can you add a debug mode that forces per-op synchronisation and disables memory pooling (e.g. FLAGOS_DETERMINISTIC=1) so we can bisect without guessing?
2. Numerical divergence in the vision tower. A ~2.7 %-of-std error over 24 blocks is much larger than bf16 should give. If the Iluvatar backend routes some of those ops to lowered/approximated implementations (e.g. matmul with reduced accumulator precision, or a softmax/layernorm variant using fast math), that would explain both the magnitude and the amplification.
3. The failure is input-length dependent: the 4 bad samples (551–782 tokens, grid_thw up to [1,6,46]) are the longest. Short samples (222–434 tokens) are all fine.

What I can run for you

The machine is idle and I have the weights. Please tell me which of these is most useful:

• FLAGOS_LOG=dispatch dump of the op routing for a single failing forward, so you can see the backend chosen per op;
• per-layer dumps (I already have a harness that hooks every block and dumps mean/std/absmax per layer, vendored vs Torch-FL, JSON out);
• a bisect over the vision tower only (24 blocks, feed the same token tensor through block k repeatedly);
• binary-search the op set by pinning individual ops to another backend via FLAGOS_OP_<name>=..., if the Iluvatar config supports it;
• a reduced standalone repro if you can suggest a public Qwen3.5-VL-shaped checkpoint that triggers it (the two internals I cannot share are the weights and the exact prompt data).

Minor environment notes (may be useful for the Iluvatar guide)

While bringing this up on iluvatar-186 I hit four things that cost time; none of them are the bug above:

1. docs/vendors/iluvatar/installation.md mounts /usr/local/corex-4.5.0/lib64/libcuda.so.1; on this host /usr/local/corex actually symlinks to corex-4.4.0, so the container's bundled libcuda.so.1 must be overridden with the host's 4.5.0 one, otherwise torch.flagos.device_count() returns 0.
2. In the CI image /usr/local/corex/bin/ixsmi is a 0-byte placeholder; ixsmi prints nothing until the host binary is bind-mounted over it.
3. libixthunk.so only exists under the host CoreX 4.5.0 tree, so LD_LIBRARY_PATH must include it or import torch_fl dies with ImportError: libixthunk.so.
4. The image ships transformers 4.57.6 + huggingface_hub 2.x; to use a newer transformers you also need huggingface_hub>=1.5,<2, httpx, typer, psutil, accelerate and a matching torchvision (transformers >= 5.10.4 is required for Qwen3_5ForSequenceClassification).