[STORY-014] Smoke-check the widened build on non-K80 hardware + document the open validation gate

[dogkeeper886]

Context

Part of STORY-014 · plan #340 · depends on the widened build

Goal

There is no representative hardware to properly validate the broadened build: no Pascal card, and no non-K80 card on the 470 stack. The maintainer's only non-K80 GPU is an RTX 2060 (sm_75) on a separate host with a different driver — it can't be paired with the K80 (no mixed-arch test) and isn't the 470 target. So the realistic goal is a best-effort smoke check plus an honest, documented validation gate — not a claim that non-K80 support is proven.

Done When

• (Best-effort, if the maintainer runs it) the image executes the sm_75 cubin on the RTX 2060 host with sane output — recorded as a weak sanity check, explicitly not a 470-target / mixed-arch / Pascal validation.
• A runnable smoke-test procedure is documented (in docs/research/470-arch-support-map.md or a sibling) so anyone with a real Pascal card or a same-host mixed box can close the gap.
• The open validation gate is written down clearly: what is proven (build runs on Turing, code-trace for the rest) vs. what is not (P100/Pascal gate crossings, the K80 does not work in tandem with P100 #223 mixed-arch-on-470 scenario), and the candidate ways to close it (K80 does not work in tandem with P100 #223 reporter, cloud instance, CI runner).

Part of #340.

[dogkeeper886]

Concrete validation plan opened as #358: standalone runs of the 9-arch build on GTX 750 / 1060 / RTX 2060 / 3060 (each in its own host, own newer driver). Covers both code paths — the GTX 1060 (sm_61, Pascal) is the key one, the closest buyable proxy to the #223 P100's legacy path. The mixed-arch box + driver-470 + exact sm_60 remain the residual gaps.

[dogkeeper886]

The RTX 2060 host — inventory + smoke-check plan

Standing up the best-effort sm_75 smoke check on the RTX 2060 box referenced in this
issue's Context. Recording the host spec here so the result is attributable and
reproducible.

Host

GPU	NVIDIA GeForce RTX 2060 (TU106), compute cap 7.5, 6144 MiB
Driver	580.159.03 (CUDA 13.0) — not the 470 target
CPU / RAM	Intel i3-14100 (8 threads) / 62 GiB
OS	Rocky Linux 9.8, kernel 5.14.0-687
Docker	29.6.1, Compose v5.3.1
nvidia-container-toolkit	not installed — to be added

Display is attached to the card (~0.5 GiB resident), leaving ~5.5 GiB usable VRAM.

What this run covers, and what it does not

sm_75 is in the shipped sweep (75-real), so the published image already carries cubin
this card executes. Running it here exercises the tensor-core path (cc >= 7.0) — the
half of the coverage matrix in #358 that the K80 can never reach.

It does not validate the 470 driver stack (this is driver 580, via CUDA
backward-compatibility), the #223 mixed-arch box, or Pascal's gate crossings. Per the
Context above, this stays a weak best-effort sanity check.

Plan

1. Install nvidia-container-toolkit, configure the Docker nvidia runtime.
2. Pull the published dogkeeper886/ollama37 rather than building — a local 9-arch build
   yields the same sm_75 cubin at the cost of hours. (Retag to ollama37:latest, which is
   what docker/docker-compose.yml expects.)
3. Create the ollama-data external volume (compose declares it external: true and will
   not create it).
4. Set OLLAMA_FLASH_ATTENTION=1 in docker/.env. This is deliberate and differs from
   the shipped default. Compose leaves FA unset because it is a ~7x loss on the K80
   ([perf] Flash attention is a major slowdown on K80 at long context — decide: default-off, fix kernel, or remove #337/[STORY-015] Plan #346); that finding does not transfer to sm_75, where FlashAttentionSupported()
   (ml/device.go:436) passes any CUDA device with ComputeMajor >= 7 and upstream's own
   gated FA kernels apply. Smoke-testing a tensor-core card with the tensor-core path
   disabled would miss the point of the exercise.
5. Run a small model (~4B q4 — the 26b/32b models in the K80 test set do not fit in 5.5 GiB)
   and record: does the sm_75 cubin load, does it answer coherently, and at what tok/s.

Will report the outcome, plus a runnable procedure so anyone with a Pascal card or a
same-host mixed box can close the parts this run cannot.

Doc note

Both this issue and #340 point at docs/research/470-arch-support-map.md as the home for
the smoke procedure, but that file was removed in 17678ce4 ("Remove the 470
arch-support-map study doc"). The "or a sibling" hedge in Done When still holds — the
procedure will land in a sibling doc under docs/research/.

[dogkeeper886]

Result — sm_75 smoke check on the RTX 2060

Edited. The first version of this comment claimed "shipped defaults: PASS" without
qualification. That was only true for the architecture I happened to test. Corrected in
§2 below: with OLLAMA_FLASH_ATTENTION unset, FA auto-enables on cc>=7.0 for nine
architectures. The default config is not safe on non-K80 cards. The rest of the trace stands.

The published image executes the sm_75 cubin on the RTX 2060 with sane output. The
best-effort Done When box can be ticked. But the default config crashes for a large class
of models, and our own docs are wrong about why.

1. What passed

dogkeeper886/ollama37:latest (digest sha256:32010cd3…, self-reports 2.1.0 — same lineage
as the d83061de sweep), retagged ollama37:latest, docker/docker-compose.yml unmodified
apart from an explicit OLLAMA_FLASH_ATTENTION=0.

inference compute  library=CUDA compute=7.5 driver=13.0
                   description="NVIDIA GeForce RTX 2060"  total="6.0 GiB" available="5.1 GiB"
offload            layers.model=29 layers.offload=29 layers.split=[29]
llama_context      CUDA0 compute buffer = 302.75 MiB
llama runner started in 1.10 seconds

Model deepseek-r1:1.5b — the only entry in k80-vram-by-family-2026-07-10-d83061de.md that
fits 5.1 GiB. Correct output ("Paris", done_reason=stop).

	K80 (sm_37, 4k)	RTX 2060 (sm_75, 4k)
tok/s	34.07	153.89

Scope unchanged: driver 580, not the 470 stack. Single card, not the #223 mixed box.
Turing, not Pascal. A weak sanity check that the multi-arch build yields runnable sm_75 code.

2. CORRECTION — the default config auto-enables FA on cc>=7.0

envconfig.FlashAttention is BoolWithDefault("OLLAMA_FLASH_ATTENTION")
(envconfig/config.go:190). When the env var is empty it returns the caller's default
(config.go:152-163), and llm/server.go:244 passes the model's own preference:

fa := envconfig.FlashAttention(f.FlashAttention())   // fs/ggml/ggml.go:904

GGML.FlashAttention() is a hardcoded architecture whitelist:

gemma3 · gptoss / gpt-oss · qwen3 · qwen3moe · qwen35 · qwen35moe · qwen3next · qwen3vl · qwen3vlmoe

So with nothing set in docker-compose.yml, any of those nine architectures switches FA on,
ml.FlashAttentionSupported() waves it through at cc 7.5, and the run hits the broken path.
The K80 is shielded only because that gate vetoes at cc < 7.

My §1 pass was luck: deepseek-r1:1.5b is general.architecture = qwen2, which is not
whitelisted. Two whitelisted archs — gemma3:4b and gpt-oss — are in the README's own
tested-models list.

This also makes the comment at docker-compose.yml:22-23 factually wrong:

…and we don't auto-enable it on the experimental cc>=7.0 cards.

It does auto-enable it, for nine architectures.

3. OLLAMA_FLASH_ATTENTION=1: deterministic abort

2/2 runs, identical. Model loads (29/29 offload); aborts on the first sampled token:

ollama: llama-sampling.cpp:662: void llama_sampler_dist_apply(...): Assertion `found' failed.
SIGABRT: abort    (signal arrived during cgo execution)

llama_sampler_dist_apply fails found when the cumulative distribution never crosses the
random draw — the signature of NaN / all-zero logits. FA off: clean, 2/2.

4. Trace — why ≥7.0 cards and not the K80

Go gates, llm/server.go:244-256 — all three pass at cc 7.5:

1. envconfig.FlashAttention(f.FlashAttention()) → true (env, or the whitelist in §2)
2. ml.FlashAttentionSupported(gpus) (ml/device.go:436) → ComputeMajor >= 7 → true
3. GGML.SupportsFlashAttention() (fs/ggml/ggml.go:882) → returns true unconditionally for
   qwen35/qwen35moe/qwen3next; others need headCountK == headCountV

Both engines converge on the same CUDA op: llama.cpp via llamarunner, and the Go engine via
ml/backend/ggml/ggml.go:1798 → C.ggml_flash_attn_ext(...).

CUDA side, ggml_cuda_get_best_fattn_kernel (fattn.cu:195) with head dim 128 / f16 KV /
gqa_ratio 6: turing_mma_available(750) true; the f16 VEC branch needs cc >= ADA (890);
gqa_opt_applies true → BEST_FATTN_KERNEL_MMA_F16 (fattn.cu:295), prefill and decode.

Then the toolchain. mma.cuh:22 gates on the CUDA runtime version, not the GPU arch:

#if CUDART_VERSION >= 11080
    ggml_cuda_movmatrix → "movmatrix.sync.aligned.m8n8.trans.b16"   // PTX, CUDA 11.8+
#else
    ggml_cuda_movmatrix → software __shfl_sync fallback              // what we compile
#endif

The image bundles libcudart.so.11.4.148 → CUDART_VERSION = 11040 < 11080. We compile
the fallback, because sm_37 pins us to CUDA 11.4 and CUDA 12 dropped Kepler.

ggml_cuda_movmatrix has one consumer, get_transposed (mma.cuh:213), called from exactly two
lines — both in fattn-mma-f16.cuh (696, 1142). Nothing else in the backend calls it.
On the K80 this code compiles but never executes, since turing_mma_available(370) is false.
Which is why it has never been seen.

FA core is pristine since the re-sync ef14fb5b: the later commits touching common.cuh
(2fbb25af, e7b68b24) only add memory-metrics lines; the K80 references left in ggml-cuda.cu
are the cuBLAS fallback, not FA.

5. Predicted behaviour per model on sm_75

Arch strings must be confirmed from general.architecture at load — sizes below are the
published tags. available on this host is 5.1 GiB.

Model	Size	arch (expected)	Engine	FA w/ no env	CUDA FA kernel	Fits 5.1 GiB	Expected
deepseek-r1:1.5b	1.1 GB	qwen2	llama.cpp	off	—	yes	PASS (observed, 153.9 tok/s)
lfm2.5-thinking:1.2b	731 MB	lfm2 / lfm2moe	llama.cpp	off	—	yes	PASS
lfm2.5:8b	5.2 GB	lfm2moe	llama.cpp	off	—	no (K80 d0 = 6358 MiB)	untestable here
qwen3.5:0.8b	1.0 GB	qwen35	Go	ON	MMA_F16	yes	CRASH
qwen3.5:2b	2.7 GB	qwen35	Go	ON	MMA_F16	yes	CRASH
qwen3.5:4b	3.4 GB	qwen35	Go	ON	MMA_F16	yes	CRASH
qwen3.5:latest	6.6 GB	qwen35moe	Go	ON	MMA_F16	no	untestable here

Notes:

• qwen35 is hybrid: FullAttention layers (attention.go:91 → nn.Attention → FA) and
  GatedDeltaNet linear layers (deltanet.go) that bypass FA. FA is exercised, but only in the
  full-attention layers.
• qwen35 short-circuits SupportsFlashAttention() to true, so the model gate cannot veto.
• model/models/qwen35/ has no vision tower (only attention/cache/deltanet/model.go), yet
  the published qwen3.5:* tags advertise Image. The image path on this fork is unverified and
  probably unsupported — worth its own issue.
• q8_0 KV requires FA. With quantized K, the dispatch takes the non-f16 branch → decode
  (Q->ne[1]==1) selects BEST_FATTN_KERNEL_VEC while prefill stays MMA_F16.

6. Minimum test matrix

Six runs cover every distinguishable path reachable on this host. All ≤ 3.4 GB.

#	Model	Env	Engine	FA	Path exercised	Expect
T1	deepseek-r1:1.5b	defaults	llama.cpp	off	sm_75 cubin, no FA	PASS ✅ done
T2	lfm2.5-thinking:1.2b	defaults	llama.cpp	off	2nd arch, no FA	PASS
T3	qwen3.5:0.8b	no .env at all	Go	auto-ON	default config → MMA_F16	CRASH
T4	qwen3.5:0.8b	FA=0	Go	off	controls for engine; isolates FA	PASS
T5	lfm2.5-thinking:1.2b	FA=1	llama.cpp	on	FA break is engine/arch-independent	CRASH
T6	deepseek-r1:1.5b	FA=1, CUDA_VISIBLE_DEVICES=-1	llama.cpp	on	CPU control — falsifies the CUDA attribution	PASS

T3 is the headline: it needs no configuration at all and proves the shipped default is broken.
T6 is the one that can refute §4 — if CPU also aborts, the fault is the sampler/model, not the
CUDA kernel, and the trace above is wrong.

Optional extensions: T7 qwen3.5:0.8b + FA=1 + KV=q8_0 (adds the VEC decode kernel);
T8 an image prompt to qwen3.5:4b (documents the missing vision tower).

Not covered by any of this, by construction: Pascal gate crossings, the #223 mixed-arch box,
driver 470, and any model above 5.1 GiB.

7. Not proven

That the movmatrix fallback is numerically wrong. It is upstream code and may be correct,
with the fault elsewhere in fattn-mma-f16 under nvcc 11.4. Well-localized suspect, not a
confirmed root cause. T6 has not been run.

Secondary, unverified: __CUDA_ARCH_LIST__ (common.cuh:133) appears to be CUDA 11.5+. If
undefined on 11.4, ggml_cuda_highest_compiled_arch degrades to return arch
(common.cuh:167), so turing_mma_available() would report true for an arch that was never
compiled. Harmless here (75 is in the sweep) but that guard cannot catch a missing cubin.

8. Proposed next steps

• Doc/gate fix — needs no further testing, and is now higher severity than first thought.
  It is not merely that .env.example:19 gives bad advice; the default auto-enables FA for
  nine architectures on every cc>=7.0 card. Either hard-gate FA off in this build, or correct
  docker-compose.yml:22-23 and .env.example:17-20 and document the crash.
• Run T6 before treating §4 as settled.
• Deeper root-cause work (get_transposed probe in the builder; sm_75-only CUDA 11.8 build)
  costs a ~15 GB image and buys a fix for a feature the K80 does not want. Recommend deferring.

Host inventory, and the fact that setting FA=1 was a deliberate extension beyond this issue's
Done When, are in the previous comment.

[dogkeeper886]

Matrix executed — the predictions hold, and the CPU control did not refute them

Ran the T1–T6 matrix from the previous comment on the RTX 2060. Every predicted outcome
was observed. The one test designed to prove me wrong (T6) passed instead.

Full procedure and state: docs/research/sm75-fa-validation-plan.md.

The path, end to end

Model qwen3.5:0.8b (general.architecture = qwen35) · Hardware RTX 2060, cc 7.5 ·
Build libcudart.so.11.4.148 ⇒ CUDART_VERSION 11040 · Config no .env at all

envconfig/config.go:153     Var("OLLAMA_FLASH_ATTENTION") == ""      → fall back to default
llm/server.go:244           fa = envconfig.FlashAttention(f.FlashAttention())
fs/ggml/ggml.go:904           GGML.FlashAttention()   "qwen35" ∈ whitelist    → TRUE
ml/device.go:436              FlashAttentionSupported()  cc 7.5 >= 7          → TRUE
fs/ggml/ggml.go:889           SupportsFlashAttention()   qwen35 short-circuit → TRUE
llm/server.go:262           loadRequest.FlashAttention = true
                              │  qwen35 ∈ OllamaEngineRequired → Go engine
runner/ollamarunner:1277    load request "{... FlashAttention:true ...}"
ml/backend/ggml/ggml.go:1798  if b.flashAttention → C.ggml_flash_attn_ext(...)
ggml-cuda/fattn.cu:195      ggml_cuda_get_best_fattn_kernel
                              turing_mma_available(750) → TRUE     (common.cuh:317)
                              VEC needs cc >= 890 (ADA)  → no
                              gqa_opt_applies            → yes
ggml-cuda/fattn.cu:295      → BEST_FATTN_KERNEL_MMA_F16
                              → _switch_ncols2 → _switch_ncols1 → _mma_f16_case
fattn-mma-f16.cuh:696,1142    get_transposed(...)
mma.cuh:213                   get_transposed → ggml_cuda_movmatrix ×2
mma.cuh:22                    #if CUDART_VERSION >= 11080  →  11040, FALSE
mma.cuh:39                    → software __shfl_sync fallback       ← suspect, NOT confirmed
                              ↓  NaN / all-zero logits
runner/ollamarunner/runner.go:737     panic: failed to sample token       [T3]

Same kernel, different engine. deepseek-r1:1.5b (arch = qwen2, not whitelisted) with
OLLAMA_FLASH_ATTENTION=1 takes the llama.cpp runner, reaches the identical
ggml_cuda_flash_attn_ext → MMA_F16 → movmatrix chain, and dies in llama.cpp's own sampler:

llama-sampling.cpp:662   llama_sampler_dist_apply: Assertion `found' failed  → SIGABRT   [T5]

Where each card leaves the path:

K80    sm_37   ml/device.go:436   ComputeMajor 3 >= 7      → FALSE  → FA never enabled     ✅
V100   sm_70   common.cuh:317     turing_mma_available(700) → FALSE  (CC_TURING = 750)
                                  → WMMA kernel; get_transposed never called    [untested]
T4 / RTX 20xx    cc 7.5   → MMA_F16 → affected  (confirmed on this host)
A100 / RTX 30xx  cc 8.0/8.6 → MMA_F16 → affected  (by trace)
CPU    —       library=cpu, FA enabled, 4.77 tok/s, no abort                    [T6]  ✅

The blast radius is cc >= 7.5, not cc >= 7.0: GGML_CUDA_CC_TURING is 750, so sm_70 takes a
different kernel.

Test environment

GPU	NVIDIA GeForce RTX 2060 (TU106), compute cap 7.5, 6144 MiB total / 5.1 GiB available (display attached)
Driver	580.159.03 · CUDA 13.0 — not the 470 target
CPU / RAM	Intel i3-14100 (8 threads) / 62 GiB
OS	Rocky Linux 9.8 (Blue Onyx), kernel 5.14.0-687
Docker	29.6.1 · Compose v5.3.1 · nvidia-container-toolkit 1.19.1, nvidia runtime
Image	dogkeeper886/ollama37:latest, digest sha256:32010cd31c32…, self-reports 2.1.0 (same lineage as the d83061de sweep)
Bundled CUDA runtime	libcudart.so.11.4.148 · libcublas.so.11.6.5.2 → CUDART_VERSION = 11040
Compose	docker/docker-compose.yml unmodified; only docker/.env varied per test (deleted entirely for T3)
Prompt	"In one sentence, what is the capital of France?", num_ctx=4096, seed=42, num_predict=16 (200 for T4)

Problems found

Passing rows omitted — those are in the matrix below, and they are the controls that make
P1/P2 credible rather than anecdotal.

#	Problem	Where	Evidence	Status
P1	Default config auto-enables flash attention on cc>=7.0, then crashes. No env var needed.	envconfig/config.go:152-163 + fs/ggml/ggml.go:904	T3: .env deleted, load request still FlashAttention:true, architecture=qwen35 → panic: failed to sample token	Confirmed on hardware
P2	FA on sm_75 aborts in both engines.	fattn.cu:295 → mma.cuh:22	T5 llama.cpp → Assertion 'found' failed + SIGABRT; T3 Go → panic. T6 (CPU, FA=1) clean ⇒ fault is in the CUDA path	Confirmed; root cause not isolated
P3	lfm2 (dense) fails to load, any GPU. Only lfm2moe was merged in #383.	vendored llama.cpp	T2: print_info: arch = lfm2 → missing tensor 'token_embd_norm.weight'	Confirmed; unrelated to sm_75
P4	docker-compose.yml:22-23 states the opposite of the truth: "we don't auto-enable it on the experimental cc>=7.0 cards"	docker/docker-compose.yml	P1 disproves it	Confirmed
P5	.env.example:17-20 instructs users into the crash: "On cc>=7.0 cards … set =1 to enable FA"	docker/.env.example	P2	Confirmed
P6	qwen3.5:* tags advertise Image; the fork has no vision tower for qwen35.	model/models/qwen35/ — only attention/cache/deltanet/model.go	Directory listing; not exercised	Untested gap
P7	#340 and #342 point at a deleted doc, docs/research/470-arch-support-map.md	removed in 17678ce4	git log	Confirmed
P8	__CUDA_ARCH_LIST__ may be undefined on CUDA 11.4, degrading ggml_cuda_highest_compiled_arch to return arch — so turing_mma_available() cannot detect a missing cubin	common.cuh:133 / :167	Not verified	Suspected

P3, P6, P7 are separate defects found in passing; they do not belong to this issue.

#	Config	Engine	FA	Observed	Verdict
T1	deepseek-r1:1.5b, defaults	llama.cpp	off	153.9 tok/s, "Paris", done=stop	PASS
T2	lfm2.5-thinking:1.2b, defaults	llama.cpp	—	error loading model: missing tensor 'token_embd_norm.weight'	BLOCKED
T3	qwen3.5:0.8b, no .env at all	Go	true (auto)	25/25 on GPU → panic: failed to sample token	CRASH
T4	qwen3.5:0.8b, FA=0	Go	false	125.4 tok/s, coherent, 0 panics	PASS
T5	deepseek-r1:1.5b, FA=1	llama.cpp	true	Assertion 'found' failed + SIGABRT	CRASH
T6	deepseek-r1:1.5b, FA=1, CPU only	llama.cpp	true	library=cpu, 4.77 tok/s, 0 aborts	PASS

1. The default config is broken on cc>=7.0 — confirmed on hardware

T3 set no environment variable whatsoever. docker/.env was removed entirely; compose
maps the unset var to an empty string. The load request still read:

runner.go:1277 load request="{... FlashAttention:true KvSize:4096 KvCacheType:f16
                GPULayers:25[...] ...}"
ggml.go:140    architecture=qwen35 file_type=Q8_0 num_tensors=536
ggml.go:499    offloaded 25/25 layers to GPU
panic: failed to sample token
  github.com/ollama/ollama/runner/ollamarunner.(*Server).computeBatch
      /usr/local/src/ollama37/runner/ollamarunner/runner.go:737

The whitelist in fs/ggml/ggml.go:904 is confirmed empirically, not merely by reading
BoolWithDefault. Any user pulling qwen3.5, gemma3, gpt-oss or qwen3* onto a
Turing/Ampere card with our published image and stock compose file gets this.

2. The break is neither engine- nor arch-specific

Two different samplers, in two different languages, fail on the same kernel:

• T5, llama.cpp engine, arch qwen2 → C++ llama_sampler_dist_apply → Assertion 'found' failed, SIGABRT
• T3, Go engine, arch qwen35 → Go sampler → panic: failed to sample token

Both engines converge on C.ggml_flash_attn_ext (ml/backend/ggml/ggml.go:1798 for the Go
path). Distinct sampling code, identical upstream cause — consistent with NaN logits arriving
from the shared CUDA FA kernel.

3. T6 was built to refute the CUDA attribution. It failed to.

Same model, same FA=1, GPU removed (CUDA_VISIBLE_DEVICES=-1, no nvidia runtime):

types.go:60  inference compute id=cpu library=cpu total="62.3 GiB"
             "enabling flash attention"   FlashAttention:true
→ 16 tokens, 4.77 tok/s, 0 aborts

Flash attention on CPU is fine. The fault is in the CUDA flash-attention path. The
BEST_FATTN_KERNEL_MMA_F16 → get_transposed → pre-11.8 movmatrix fallback hypothesis
survived the test that could have killed it.

4. FA is the only variable

T3 and T4 differ in exactly one bit — same model, same Go engine, same host, same 25/25
offload. FA=true panics; FA=false yields 125.4 tok/s and coherent reasoning.

5. Still not proven

T6 narrows the fault to the CUDA FA path. It does not single out get_transposed within
that path. The movmatrix fallback remains a well-localized suspect, not a confirmed root
cause — that needs the builder-image probe or an sm_75-only CUDA 11.8 build, both of which I
still recommend deferring (see §8 of the previous comment).

6. New, unrelated finding — lfm2 (dense) does not load

T2 never reached flash attention:

print_info: arch = lfm2
llama_model_load: error loading model: missing tensor 'token_embd_norm.weight'

lfm2.5-thinking:1.2b is arch = lfm2 (dense). Only lfm2moe — lfm2.5:8b — was merged in
#383. This is a load-time architecture gap and is not sm_75-specific; it would fail on the
K80 identically. Filing separately; it does not belong to this issue.

7. What #340 step 2 needs to change

the new ≥7 cards … get FA on for free … the gate is pure-upstream and already
correct for every card in the sweep

"On for free" is now demonstrated — but it is on for free and broken, with no user action.
The gate cannot be left untouched. Recommend, in order:

1. Hard-gate FA off in this build (or correct the whitelist path), independent of root
   cause. This is a real crash on the default config, on the exact cards STORY-014 widened the
   build to reach.
2. Fix docker-compose.yml:22-23 — "we don't auto-enable it on the experimental cc>=7.0
   cards" is false — and .env.example:17-20, which actively instructs users into the crash.
3. Defer root-cause work until a Pascal card or the K80 does not work in tandem with P100 #223 reporter appears.

Scope, unchanged

Driver 580, not the 470 target. One Turing card, not the mixed-arch box. No Pascal. Still a
weak best-effort sanity check on the build, exactly as this issue frames it. What it now
also carries is a reproducible, hardware-confirmed defect in the default configuration.