Deterministic temp=0 generation differs across Apple Silicon targets

[ivanfioravanti]

I am seeing different generated outputs and token counts from the same OpenAI-compatible server/engine when running the same model and request parameters on different Apple Silicon machines.

Specifically, I tested M3 Ultra and M5 Max producing different completions even with:

• temperature: 0
• same model
• same prompt
• same seed
• same max_tokens
• same OpenAI-compatible /v1/chat/completions request shape

M5 Max results:
• aime25: 5/5 passed, 0 failed, runtime 00h:02:18s

#	State	Prompt	Gen	Total	Given	Correct	Test
1	PASSED	86	928	1014	70	70	aime25/aime25_0000
2	PASSED	215	7857	8072	588	588	aime25/aime25_0001
3	PASSED	171	1815	1986	16	16	aime25/aime25_0002
4	PASSED	113	1609	1722	117	117	aime25/aime25_0003
5	PASSED	140	5982	6122	279	279	aime25/aime25_0004

M3 Ultra results:
aime25: 5/5 passed, 0 failed, runtime 00h:02:29s

#	State	Prompt	Gen	Total	Given	Correct	Test
1	PASSED	86	927	1013	70	70	aime25/aime25_0000
2	PASSED	215	6145	6360	588	588	aime25/aime25_0001
3	PASSED	171	1790	1961	16	16	aime25/aime25_0002
4	PASSED	113	1575	1688	117	117	aime25/aime25_0003
5	PASSED	140	3940	4080	279	279	aime25/aime25_0004

This is visible with the standalone repro script available in this gist:

python3 aime25_repro.py
--base-url http://127.0.0.1:8080/v1
-m mlx-community/Qwen3.6-35B-A3B-4bit
--temperature 0
--cases 5
--max-tokens 8192
--seed 1
--no-think

The script sends five embedded AIME25 prompts and prints a summary table with prompt tokens, generated tokens, extracted answer, expected answer, and pass/fail status.

Expected behavior:

With temperature=0, the same model and same request should produce identical outputs, or at least identical generated-token counts and final answers, across M3 Ultra and M5 Max.

Actual behavior:

M3 Ultra and M5 Max generate different outputs/token counts for the same cases. In some runs, one machine reaches the correct final answer while the other either produces a different answer or continues until max_tokens.

This matters because AIME-style evals become inconsistent across hardware: the same eval can pass on one Apple Silicon target and fail on another, despite using deterministic settings.

It would be useful to know whether this is expected due to backend/kernel/numerical differences (I bet this is the way to get more performance out of M5 Neural Accelerators), or whether deterministic decoding should be guaranteed across these Apple Silicon targets.

[lBroth]

Possibly useful data for this issue, from a related investigation
(single-device, but same mechanism class):

While validating our speculative rollback for hybrid GDN caches (#1486)
on an M5 Pro we characterized a within-device, shape-dependent temp=0
divergence (same model, same device, different execution shapes):

generation through a T>1 verify forward (batched-GEMM matmul path) vs
plain T=1 decode (GEMV) diverges at positions where the model is
near-tied. Teacher-forcing the shared prefix and reading the top-2
logit gap at every divergence point gave gaps of 0.000-0.125 (one exact
tie), never large;
the effect needs no speculation at all: at ~2.5K generated tokens,
plain greedy diverges from itself when only the prefill chunk size
changes (2048 vs 512);
it is quality-neutral in our tests (pass@1 unchanged across HumanEval
n=164 + MBPP n=257, with and without the T>1 path).
If cross-device divergence is the same class (different chips give
different kernel tilings and rounding, so different winners at
near-ties), a cheap test would be: at the first cross-device
divergence, teacher-force the shared prefix on both machines and
measure the top-2 gap. A small gap points to tie-class numerics
(expected, benign); a large gap points to something actually wrong.
Method details in #1423 (comment).

I'm not an ML engineer and leaned on AI (Claude) pretty
heavily for this. The numbers held up across a couple of adversarial
re-verification rounds, but expert eyes on the methodology would be
genuinely welcome.