Skip to content

Kernelize's Deepseek v4.1 Bringup - #5586

Draft
abrown wants to merge 98 commits into
tenstorrent:mainfrom
abrown:kernelize-bringup-ds41
Draft

abrown wants to merge 98 commits into
tenstorrent:mainfrom
abrown:kernelize-bringup-ds41

Conversation

@abrown

@abrown abrown commented Nov 17, 2026 •

Copy link
Copy Markdown

This is a project to upstream Tenstorrent support for Deepseek v4.1. @bgoelTT, as requested, this branches off of tt-rfp-m0, so it will be out of date with main. As it stands now, there are 60 commits here but we believe we could whittle this down to ~13 with some more work. We publish these as-is now since they are the commits verified with the accuracy evals and benchmark sweep.

Related:

bgoelTT and others added 30 commits August 6, 2026 20:24
…asure

Both LLM benchmark parsers normalised tool output to a flat record that kept
mean, p50 and p99 for time to first token, but only the mean for time per
output token and end-to-end latency. No p90 for anything, no standard
deviation for anything. The tools measure all of it; normalisation dropped it.

Checking a real gpt-oss-120b release run showed the loss is not hypothetical:
the raw vLLM result already contained median_tpot_ms, p99_tpot_ms, std_tpot_ms,
median_e2el_ms, p99_e2el_ms, std_e2el_ms and std_ttft_ms — eight fields
collected, written to disk, then discarded on the way into the report.

Changes:

- aiperf.py — emit p90/std for TTFT, and p50/p90/p99/std for TPOT and E2EL.
  metric_stat() already reads raw[metric][stat], so this extends the field map
  rather than adding extraction logic.
- vllm.py — read the same percentiles. Keys beyond mean/median/p99 only appear
  when the run requested them; when absent round_metric yields None, never 0.
  That distinction matters: zero is the best possible latency, so a missing
  value coerced to 0 would score as a perfect result rather than as unmeasured.
- drivers/vllm.py — pass --metric-percentiles 90,99. The driver already passed
  --percentile-metrics, but vLLM defaults the percentile list to "99", which is
  precisely why p90 was never emitted. It costs nothing alongside the existing
  p99 computation.
- display.py — add the six missing column headings, so the new fields render as
  named columns rather than raw keys.

Tests: four new cases covering AIPerf extraction, vLLM extraction, display-name
coverage, and — the one that matters most — that an absent percentile arrives as
None rather than 0 in both parsers.

Full suite: 1574 passed, 10 skipped.

Refs tenstorrent#57

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the previous commit, prompted by checking how real AIPerf output
is read elsewhere in the tree rather than trusting the synthetic test fixture.

`llm_module/drivers/aiperf_prefix_cache` consumes genuine AIPerf exports and
reads the same three metric blocks — time_to_first_token, inter_token_latency,
request_latency — with stats avg, p50, p90, p95, p99 and std. That independently
confirms the extraction added previously, and exposes two gaps in it:

- It reads `avg`/`mean` and `p50`/`median` as alternatives, because AIPerf
  labels those two statistics differently across versions. The benchmark parser
  tried only one spelling each, so an export using the other would parse fine on
  the prefix-cache path and silently yield None here. Added a `_stat_any` helper
  that mirrors the driver's fallback behaviour.
- It extracts p95, which the benchmark parser skipped. Display headings for
  p95_ttft, p95_tpot_ms and p95_e2el_ms already existed, so the field was
  expected and simply never populated. The grading rubric does not read p95, but
  RFP requirement G.2.2 asks for every metric in the record.

Both parsers are now symmetric on p95, and the vLLM driver requests
`--metric-percentiles 90,95,99` so the two tools report the same percentile set
and stay comparable. A shared test asserts an absent percentile is None rather
than 0 in both, which is what caught the asymmetry.

Full suite: 1575 passed, 10 skipped.

Refs tenstorrent#57

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… targets

Groundwork for generating RFP Milestone-0 benchmark fixtures against a simulated
Blackhole Galaxy. Every change here is a real defect found by trying to run the
AIPerf benchmark path end to end; none is specific to simulation.

requirements/llm-aiperf.txt — drop torch, allow transformers 5.x
  Nothing in this venv needs torch: aiperf does not depend on it and no other
  installed distribution declares it. The pin broke two things. torch 2.4.0 is
  incompatible with transformers 5.x (which needs DTensor from
  torch.distributed.tensor, 2.5+), so loading the Gemma-4 tokenizer raised
  ImportError; and the +cpu local-version wheel is published for linux/x86_64
  only, so the venv could not be built on macOS or aarch64 at all. Removing it
  also avoids adding protobuf, which the transformers 4.x SentencePiece path
  would otherwise require. Verified: aiperf installs and
  AutoTokenizer.from_pretrained("google/gemma-4-31B-it") loads in 4.0s
  (GemmaTokenizer, vocab 262144) with no torch present.

llm_module/drivers/aiperf.py — pass --ui-type none
  AIPerf defaults to a full-screen Textual dashboard. With no attached terminal
  it emits alternate-screen escape codes and blocks forever, leaving a 0-byte
  aiperf.log. Benchmarks are always run non-interactively.

workflows/model_spec.py — warn when a model has no reachable targets
  A model whose targets are filed under a differently-spelled key resolved to {}
  in silence: no targets, no warning, and every benchmark point then grading as
  ungradable. gemma-4-31B-it hits this today — the reference file stores
  "gemma-4-31b-it", matching the Forge dev spec and tt-media-server's
  ModelNames constant, while the lookup derives "gemma-4-31B-it" from the prod
  HF repo. Both spellings are deliberate and name the same model on the same
  device, so which spec owns those targets is a question for the model owner;
  this does not guess, it only makes the miss visible.

workflows/model_specs/prod/llm.yaml — add a GPU device spec for gemma-4-31B-it
  The documented bring-your-own-server path (docs/gpu_workflows.md) lets the
  benchmark workflows target an OpenAI-compatible endpoint with no Tenstorrent
  hardware present. Follows the existing Llama-3.1-8B GPU entry.

charts/ — regenerated by workflows.helm_generator for the new device spec.

Upstream issues: tenstorrent#4883 (torch/transformers), tenstorrent#4884
(silent target miss). Both changes are branch-local to the RFP handoff and
carry revert notes.

Also worth reporting to ai-dynamo/aiperf separately: AIPerf hangs rather than
propagating a tokenizer error, writing nothing to its log. Confirmed no requests
reach the server while it sits idle.

Full suite: 1575 passed, 10 skipped.

Refs tenstorrent#57

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…xponent

The Milestone-0 grading rubric weights prefill at 55 of 100 points, but nothing
in the stack measures prefill directly. tt-metal has prefill_t/s only on the demo
path, the normalised serving record has no equivalent, and WorkflowType.
PREFILL_DECODE is not a prefill benchmark — it serves a mock stack for
prefill/decode disaggregation smoke tests. So prefill quality is derived from
time to first token across the sweep.

Adds llm_module/derived_metrics.py:

  prefill throughput     input_sequence_length / mean_ttft_ms, tokens/s
  ttft tail ratio        p99_ttft / p50_ttft
  ttft scaling exponent  fitted b in log(p50_ttft) = a + b*log(isl)

The first two attach per point in the runner, next to apply_target_checks and
following the same Block-in/Block-out shape. The exponent needs several input
lengths at one concurrency, so it is fitted after the sweep completes.

Fitted separately per concurrency level, never pooled. That is the entire point
of the metric: a system can scale cleanly when idle and degrade badly under load,
and one combined slope averages exactly that away. A test pins the behaviour with
a synthetic system that is linear at concurrency 1 and quadratic at 32, and
asserts the pooled fit would have reported neither.

Everything is None when uncomputable, never 0. For a latency-derived figure 0 is
the best possible result, so a coerced zero would score as perfect rather than as
unmeasured — the same invariant established for the parsers in tenstorrent#57.

A concurrency level with fewer than three usable input lengths gets None plus a
warning naming the level, rather than a slope through two points that always fits
perfectly and says nothing about curvature. This is the constraint behind tenstorrent#64.

Verified against the real gpt-oss-120b release run: prefill throughput climbs 486
-> 6,977 tok/s across isl 128 -> 32768 as fixed overhead stops dominating, tail
ratios land between 1.0 and 1.8, and exponents come out 0.51 at concurrency 1 and
0.83 at 32. It correctly declines to fit at concurrencies 3, 7, 15 and 31, where
that run's auto-generated sweep left a single point each.

Full suite: 1597 passed, 10 skipped.

Closes tenstorrent#58

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in the tree could take two benchmark runs and say whether they agree —
searching for report-comparison, regression-comparison or baseline-diff turned up
nothing. The Milestone-0 reproduction check needs exactly that: a Partner reports
numbers, we re-run their submission, and every metric must agree within a stated
margin. Doing it by hand across two models, a full sweep, three runs each and
~15 metrics per point is slow and unreliable, and it decides whether a milestone
is credited.

Adds report_module/comparison.py plus a CLI:

    python -m report_module.comparison \
        --reported partner_run{1,2,3}.json \
        --measured our_run{1,2,3}.json --margin 0.05

Exits non-zero when the reproduction fails, so it drops into CI unchanged.

Three decisions worth recording:

Medians across runs, not single runs. Each side supplies several reports of the
same configuration and the median of each metric is compared. A single run would
let noise decide a commercial outcome; a test pins that one bad run out of three
does not fail an otherwise-agreeing submission.

Benchmark sections are identified by content, not by `kind`. That field is not
stable — current parsers emit "benchmarks" while the real gpt-oss-120b release
report carries the tool name "vllm". Keying off the operating-point fields any
benchmark record must have handles both, and a future tool as well.

A value missing on ONE side fails; missing on BOTH does not. The first would let
an incomplete submission reproduce perfectly. The second is a different thing —
there is nothing to disagree about, so it is reported as a capture gap rather
than a mismatch. This distinction was found by running against the real report:
error_request_count is null throughout it, and under the stricter rule a report
failed to reproduce against ITSELF. That sanity invariant is now a test.

Verified on the gpt-oss-120b release run: self-comparison passes across 17
operating points; perturbing one point's mean_ttft_ms by 12 % fails with exit 1
and isolates that single metric at 12.00 % against the ±5 % margin.

Full suite: 1618 passed, 10 skipped.

Closes tenstorrent#59

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bonus rubric scores two prefix-cache values, and a machine could reach
neither. The decode hit rate was reachable only by reimplementing the
aggregated/disaggregated precedence rule, and the time-to-first-token uplift was
computed inside the Markdown renderer while building a table — it never reached
the report JSON at all. The scorecard generator cannot read a rendered table.

Adds report_module/prefix_cache_uplift.py, applied after the sweep in the
prefix-cache orchestrator, mirroring where apply_scaling_exponents sits in the
benchmark path. It has to run there rather than per run because the uplift
compares a treatment run against its matched baseline run.

The sign is the part that would have gone wrong quietly. The renderer's
ttft_uplift_pct is a latency delta, so a cache that HELPS produces a NEGATIVE
number — correct for a table, inverted for the rubric, which asks for a
percentage reduction where higher is better and 0 % is qualifying. Had the
scorecard read the display field, a 44 % improvement would have scored zero and a
cache that made time to first token 44 % worse would have scored points. The
scored field is therefore named ttft_reduction_pct, carries the opposite sign
deliberately, and both directions are asserted in tests rather than left to
follow from the arithmetic.

Two further decisions, both about not making Partners game things:

The optional mooncake_trace scenario is excluded from the scored summary. If an
optional run could move the score, a Partner would have to work out whether
running it helped or hurt them, which is a perverse thing to make anyone reason
about.

A missing required scenario is reported, not averaged around. Taking the mean of
whichever required scenarios happen to be present would quietly reward dropping
the weakest one. The summary reports scenarios_missing and leaves the decision to
the caller.

Scenario means are taken within a scenario before averaging across scenarios, so
a scenario measured at more operating points does not thereby count for more.

Also single-sources the baseline-matching rule and the delta, which the renderer
now imports. Display and score disagreeing about which baseline matches a
treatment run is exactly the kind of thing that surfaces during a bid dispute.

Full suite: 1640 passed, 10 skipped.

Closes tenstorrent#62

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d zero

Zero failed requests is a binary requirement (RFP G.2.6) that nothing enforced.
A submission could pass acceptance with requests failing at every sweep point,
provided the ones that survived were fast enough — and speed measured over only
the requests that succeeded is not a result.

Fixing it turned up the reason the field looked unusable. Both parsers coerced a
measured zero to None:

    aiperf.py  if isinstance(value, (int, float)) and value:   # 0 is falsy
    vllm.py    return int(v) if v else None                    # 0 -> None

So "no requests failed" and "nobody looked" were the same value, and the field
could not be graded at all. This is the None-vs-zero invariant the derived
metrics already keep, inverted: there, a missing value must not become 0; here, a
measured 0 must not become missing. Both directions destroy the distinction.

All 17 points of the gpt-oss-120b release run carry failed: 0 in the raw tool
output. Every one of them reached the report as null. Worth noting this is also
the true cause of the null error_request_count that tenstorrent#59 worked around — it was
neither an aiperf-versus-vLLM difference nor an unmeasured metric; the tool
measured it and the parser threw it away.

Parsers now return None only when the count is genuinely absent. Acceptance then
grades every benchmark block, not only the first:

  count > 0   blocks, naming the count
  count == 0  passes — a real measurement of the required outcome
  absent      blocks, because the requirement is that zero was CONFIRMED, and
              treating absence as a pass is how an unverified requirement
              silently becomes a satisfied one

Behaviour change worth flagging: a benchmark block with no error count now fails
acceptance where it previously passed. After the parser fix both supported tools
populate it, so this should only catch genuinely unreported cases.

Verified against the release run: 17/17 points now report 0, the clean run
accepts, and injecting 5 failures at one point blocks with that point named.

Note the module named in the issue, workflows/acceptance_criteria.py, is imported
by nothing but its own tests — every live caller goes through
report_module/acceptance_criteria.py, which is what this changes. The legacy copy
is left alone rather than growing a second implementation of the same rule.

Full suite: 1647 passed, 10 skipped.

Closes tenstorrent#61

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds report_module/scorecard.py and a CLI. This ranks competing commercial bids,
so hand-computation was never acceptable: error-prone, not auditable, and not
reproducible if a Partner disputes a score.

Implements Part K end to end. Each graded point's qualifying and excellence
values are derived from that point's own target using the published B.5
multipliers rather than authored per point, so there is no second set of numbers
to drift and a Partner can recompute any value. Per-point weights combine a
log2(input length) factor with the 25/75 concurrency shares and sum to 1. Scaling
quality is fitted per concurrency level and combined by share — never pooled,
because a pooled fit averages away a system that scales cleanly when idle and
degrades under load. Both K.8 rules are applied after scoring and recorded on the
line, so a scorecard shows what a failed reproduction or a waiver actually cost.

Every intermediate is kept, per point, in both the markdown and the JSON: a
disputed score has to be walkable back to the measurement it came from.

An unscoreable line still contributes zero — the rubric has no other option — but
it is listed separately from lines that genuinely scored zero. A line measured
exactly at its qualifying value and a line nobody measured are different facts
about a submission and must not look identical.

The golden test reproduces Appendix F.3 exactly: all ten per-point weights, all
ten p99 fractions, the 0.5649 line fraction and the 12.43 line score.

It also found a defect in the published example. F.2's core total and overall are
sums of ROUNDED group subtotals — 31.99 + 4.70 + 9.87 + 9.04 = 55.60 — while the
unrounded scores sum to 55.5939, which rounds to 55.59. Overall is 69.46, not
69.47. Rounding is a display step; letting it feed back into the arithmetic makes
a total depend on how many decimals the scorecard happens to print, and the error
grows with every line. The code sums at full precision and the RFP is corrected
to match, with a test pinning the distinction so it cannot quietly come back.

Targets still have to be wired in from model_performance_reference.json to score
a real run; that waits on the sweep and target work (tenstorrent#66, tenstorrent#67). The CLI takes a
submission JSON so the engine is usable and testable now.

45 tests. Full suite: 1692 passed, 10 skipped.

Closes tenstorrent#60

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--output-tokens-mean/-stddev only *request* a fixed output length. Without
ignore_eos the server is free to stop at its natural EOS, so the osl axis of a
benchmark sweep stops meaning anything.

Measured against llm-d-inference-sim asking for osl=128: 27-130 tokens returned
(std 41) before this change, 131-135 (std 1.5) after. Output length feeds TPOT,
E2EL and both throughput columns, so entire rows were affected -- a simulated
corpus generated at target reported decode throughput at 0.787x of target for
ISL 131072 purely from truncated generations.

This restores the pairing the sibling drivers already use: aiperf_spec_decode.py
couples ignore_eos:true with output-tokens-mean/-stddev, and the agentic-traces
scenario enforces it too. This driver requesting a fixed length without
enforcing it was the outlier.

Hooks (ruff, ruff-format, pytest 1692 passed, copyright) were run manually via
`pre-commit run --files`; the repo's .git/hooks/pre-commit points at a venv
without pre_commit installed, hence --no-verify.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cy KV budget

max_tokens_all_users defaults to max_context, so the token budget shared across
all users equalled ONE full-context request. get_benchmark_max_concurrency then
correctly derives a concurrency of 1 for a full-context request, and the
long-context sweep points were silently benchmarked far below the concurrency
they registered:

  ISL    8192 / OSL 1024: 32 requested -> 28 measured
  ISL   32768 / OSL 1024: 32 requested ->  7 measured
  ISL  131072 / OSL 1024: 32 requested ->  1 measured

That is neither what the RFP asks bidders to build -- max context AT max
concurrency -- nor what the rubric intends to grade, and it also fragmented the
TTFT-scaling-exponent fit across concurrency buckets until it had too few points
per bucket to fit at all.

Setting the budget to max_context * max_concurrency (262144 * 32) restores
concurrency 32 at every ISL, verified end to end against llm-d-inference-sim:
the two clamped points now run as bench_32768_1024_32_n32 and
bench_131072_1024_32_n32.

This is a bring-your-own-server endpoint, which like SUPER_CLUSTER is not bound
by a single device's KV pool. SUPER_CLUSTER already gets context * concurrency in
_infer_data; extending that branch to cover remote GPU endpoints would be the
more general fix and would make this override unnecessary.

Hooks run manually via `pre-commit run --files` (pytest 1692 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… throughput

The `tput` perf target is defined as decode interactivity x concurrency -- the
median per-request decode rate (1000 / TPOT), summed over the concurrent users.
The three LLM parsers instead fed it the tools' own output_token_throughput
(vllm: output_throughput), which is total output tokens divided by the whole
benchmark duration and therefore charges decode for time spent in prefill and in
queueing.

The two quantities agree only when prefill is negligible, which is why this went
unnoticed: they diverge as input length grows, and the metric degrades even
though the system's decode rate is unchanged. On a simulated corpus generated to
exactly meet target at concurrency 32:

  ISL     tput_user   wall-clock  decode agg
          ratio       ratio       ratio
      128 1.046       1.042       1.046
     1024 1.046       1.038       1.046
     8192 1.045       1.002       1.045
    32768 1.044       0.889 FAIL  1.044
   131072 1.049       0.456 FAIL  1.049

Both failures were artefacts: at ISL 131072 the machine spends roughly three
times as long prefilling as decoding, so a wall-clock output-token rate cannot
approach 35 t/s/u x 32 no matter how fast decode is. The target was not missed,
it was measured against the wrong quantity.

tps_decode_throughput now carries the decode aggregate, matching both its own
documented description ("across all users") and how
test_module/stress_tests/stress_tests_record_builder already derives the same
field (mean_tps * actual_max_con). The tools' wall-clock figure is still reported
as the new tps_output_throughput, labelled as such and not graded, so nothing is
lost. The helper returns None rather than 0 when a rate or concurrency is
missing, since a zero would score as a perfect result.

Hooks run manually via `pre-commit run --files` (pytest 1699 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not skew TTFT

AIPerf generates 100 unique dataset entries by default (--num-dataset-entries)
and reuses them once --request-count exceeds that. A 128-request sweep point
therefore sends 28 duplicate prompts, which the server answers from its prefix
cache with a TTFT of roughly the dispatch overhead.

The damage is that this moves the mean and not the median, so it does not look
like an artefact. Measured at ISL 8192, concurrency 32, 128 requests against
llm-d-inference-sim configured to exactly meet target:

  entries=100 (default): mean 1021.7 ms  median 1278.6 ms  28 cache hits
  entries=128 (fixed):   mean 1294.2 ms  median 1293.5 ms   0 cache hits

The reported mean TTFT was 21% below the truth, and every graded mid-context
point on the loaded corner read 25-31% *faster* than a target the system was in
fact only just meeting. With the fix, mean and median agree and the residual gap
to target is the simulator's dummy tokenizer under-counting prompt tokens by
~11%, which is a known simulator limitation rather than a measurement error.

Not simulator-specific: prefix caching is enabled by default in vLLM, so any
sweep point with num_prompts > 100 understated TTFT on real hardware too. The
concurrency-1 points (num_prompts 8) were never affected, which is why this
surfaced only once long-context points began running at their registered
concurrency of 32.

Hooks run manually via `pre-commit run --files` (pytest 1699 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ilures

The acceptance gate blocked every graded point with "No error_request_count
reported, so zero failed requests cannot be confirmed", so no corpus could pass
Part I regardless of its latency numbers.

The gate is right to block on absence -- RFP G.2.6 requires zero failures to be
*confirmed*, and treating silence as a pass is how an unverified requirement
becomes a satisfied one. The bug is that AIPerf's silence is not silence: it
omits error_request_count from the export entirely when nothing failed, and only
emits it (with error_isl / total_error_isl) once at least one request has failed.
The parser read that absence as "nobody looked", inverting the meaning of the
healthiest output AIPerf can produce.

error_summary is the right signal and AIPerf always writes it: an empty list on a
clean run, one entry per distinct failure otherwise. That is an affirmative "the
tool looked", which is exactly what the gate needs in order to tell a confirmed
zero from silence. Verified against aiperf 0.5.0 using the simulator's
--failure-injection-rate:

  rate=0:  error_summary: []            error_request_count absent, request_count 8
  rate=50: error_summary: [4 entries]   error_request_count 5,      request_count 3

Fixed in the parser rather than the gate: the parser is the layer that knows one
tool's export dialect, so the gate stays tool-agnostic and its guarantee is
untouched. An export with neither key still yields None and still blocks.

The summary is summed rather than merely tested for emptiness, so a partial export
cannot report zero failures while listing some. Checked against all 20 real corpus
exports (each now 0, previously None) and the injected-failure export (5).

Hooks run manually via `pre-commit run --files` (pytest 1703 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benchmarks summary could report a negative number of passing blocks -- a real
20-point sweep printed "-2/20 passed, 10 failed, 2 waived, 10 NA", which is the
first number a reader sees.

`waived` is an overlay on the other counts, not a fourth exclusive bucket. A block
lands in it when a tier check failed without blocking, and that happens in two
ways: on its own, where the block still passed the gate and was never in a
non-pass bucket; or alongside a real blocker, where the block is already counted
in `failed`. That overlap is deliberate --
test_benchmark_failing_complete_tier_still_blocks_at_complete_status asserts it,
so a masked tier is surfaced rather than silently dropped just because the block
failed elsewhere. Subtracting len(waived) therefore removed passing blocks that
were never excluded, and removed failed blocks twice.

One block that both failed and carried an informational note was enough to
reproduce it (passed == -1). Now passed/failed/skipped/na partition total, so the
summary line reads as a partition, and waived is worded as an annotation ("N with
waived check(s)") rather than listed as though it were a further share.

Note for anyone using a corpus as a scoring oracle: waivers exist here because
gemma-4-31B-it is status EXPERIMENTAL, which enforces zero target tiers by design
-- required_target_tiers is empty, so "a forge/new bring-up model can fail every
performance benchmark and still be released". A block failing functional,
complete AND target is still accepted at that status. Only TOP_PERF enforces the
target tier. The waivers are correct; only the arithmetic was wrong.

Hooks run manually via `pre-commit run --files` (pytest 1708 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Milestone-0 scoring chain had a manual gap: nothing built the document
report_module.scorecard consumes. GradedPoint was constructed only in scorecard.py
and its own tests, so a submission was hand-transcribed -- roughly ten points times
eleven fields -- which is the one step of an auditable ranking process with no
audit trail.

    python -m report_module.submission --partner "Acme" --run workflow_logs

Runs after the workflows, reads their reports, writes the submission. Not a change
to any workflow.

Derived, because the mapping is exact: each graded point's targets (taken from its
own target_checks["target"], so they travel with the measurement they grade),
the measured percentiles and throughputs, the fitted scaling exponent per
concurrency level, and the two bonus figures when a prefix-cache report is present
(read via prefix_cache_uplift.SCORED_FIELDS so the two cannot drift apart).

Supplied on the command line: eval margins, contribution quality, assistance units
and the reproduction outcome. Deliberately not guessed -- how a set of per-task
eval ratios becomes one margin is a scoring policy expressed nowhere in this
repository, and inventing an aggregation would invent published terms. The per-task
ratios found in the reports go to _evidence so whoever sets the margin can see what
they are setting it from.

run_to_run_cov is computed from repeat runs under a definition this module chooses
rather than one the rubric publishes, so the definition is stated in the output and
--run-to-run-cov overrides it.

Two properties the tests pin, both about not inventing data. A point measured once
yields no CoV rather than 0.0, since zero variation would score full marks for
having done nothing, and splitting a sweep across runs must not manufacture a
stable-looking score. Points merge across runs by (concurrency, input length) with
the earliest run winning, so a Partner may run the two concurrency corners
separately while scored figures still come from one consistent run.

Validation happens here, where the message can name the cause: a sweep that is not
exactly two concurrency corners is rejected with a pointer to
ONLY_BENCHMARK_TARGETS=1, rather than surfacing later as a weighting error from
inside point_weights().

Verified against the two simulated corpora: assembles 10 points at concurrency
1 and 32 and scores a core total of 32.92, identical to an independently
hand-built submission for the same data.

Hooks run manually via `pre-commit run --files` (pytest 1726 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…enstorrent#4910)

* Add DeepSeek-V4-Flash-0731 GPU reference eval config and thresholds

Add a GPU bring-your-own-server dev spec for deepseek-ai/DeepSeek-V4-Flash-0731
and an EvalConfig (r1_gpqa_diamond, terminal_bench_2_1, swe_bench_verified)
with GPU reference scores measured with thinking enabled at reasoning_effort=max
(GPQA 90.40, Terminal-Bench 2.1 74.16, SWE-Bench Verified 87.20).

Co-authored-by: Cursor <cursoragent@cursor.com>

* Report DeepSeek-V4-Flash author scores at max reasoning effort

Align published_score for DeepSeek-V4-Flash-0731 with the author's
max-effort numbers, since the GPU references are collected at
reasoning_effort=max: GPQA-Diamond 87.4->88.1, Terminal-Bench 2.1
56.6->82.7 (repointed to the -0731 card, which reports code-agent tasks
at max effort), SWE-bench Verified 78.6->79.0.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
tenstorrent#4909)

* Add DeepSeek-V4-Flash-0731 GPU reference eval config and thresholds

Add a GPU bring-your-own-server dev spec for deepseek-ai/DeepSeek-V4-Flash-0731
and an EvalConfig (r1_gpqa_diamond, terminal_bench_2_1, swe_bench_verified)
with GPU reference scores measured with thinking enabled at reasoning_effort=max
(GPQA 90.40, Terminal-Bench 2.1 74.16, SWE-Bench Verified 87.20).

Co-authored-by: Cursor <cursoragent@cursor.com>

* Report DeepSeek-V4-Flash author scores at max reasoning effort

Align published_score for DeepSeek-V4-Flash-0731 with the author's
max-effort numbers, since the GPU references are collected at
reasoning_effort=max: GPQA-Diamond 87.4->88.1, Terminal-Bench 2.1
56.6->82.7 (repointed to the -0731 card, which reports code-agent tasks
at max effort), SWE-bench Verified 78.6->79.0.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add Mistral-Small-4-119B-2603 GPU reference eval config and thresholds

Add a GPU bring-your-own-vLLM dev spec and eval config for
Mistral-Small-4-119B-2603 covering GPQA-Diamond, AIME 2025, and
Terminal-Bench 2.1. Set gpu_reference_score thresholds from the reference
run (reasoning_effort=high, temperature=0.7, top_p=0.95): GPQA 77.27
(153/198), AIME25 86.67 (26/30), Terminal-Bench 2.1 32.58 (29/89).

For terminus-2, reasoning_effort is a top-level agent argument that the
agent forwards into LiteLLM itself, so it must stay out of llm_kwargs to
avoid a duplicate-keyword TypeError.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
* forge SDXL: recalibrate N150 spec-test thresholds for the Forge path (tenstorrent#4575)

The forge SDXL N150 nightly reaches spec_tests now that the request
timeout fix lets the job complete. Evals + benchmarks pass, but three
spec_tests fail because their targets were tuned for the faster
all-on-device trace path, not the Forge UNet-only path (~11-15s/img):

  - ImageGenerationLoadTest 20-step: 11.1s vs 10s target
  - ImageGenerationLoadTest 30-step: 14.9s vs 14s target
  - ImageGenerationParamTest diff-params SSIM: 0.9813 vs <0.98 bound

Add sdxl+n150 model_targets overrides (20-step 10->12s, 30-step
14->16s, diff_params_max_ssim 0.98->0.985), scoped to n150 so t3k and
galaxy targets are unchanged. Update the matrix-expansion unit test and
add a t3k regression guard so the two paths can't silently drift.

Clears 3/5 spec_test failures; the remaining 2 (LoRA eval + LoRA load)
need LoRA support in SDXLForgeRunner, tracked separately.

Refs tenstorrent#4352

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* comment out (tenstorrent#4886)

* uplift TLE (tenstorrent#4860)

* read scheduler diagnostic (tenstorrent#4880)

* read diagnostic

* fix build + format

* Add BLACKHOLE_GALAXY mapping to utils device_utils (tenstorrent#4889)

* Determine BLACKHOLE_GALAXY device config for Milestone-0 (readiness 5.4)

Settle the 32-chip Blackhole Galaxy (4x8 2D torus) device-level configuration
that blocks the M0 sweep/target work, and add the scaffolding it produces:

- Add BLACKHOLE_GALAXY device specs to the dev catalog for gemma-4-31B-it
  (real tt_transformers config, full 256K context) plus partner-fillable stubs
  for deepseek-ai/DeepSeek-V4-Flash-0731 (1M native context, new deepseek_v4_flash
  stub impl) and mistralai/Mistral-Small-4-119B-2603 (256K context). All pin the
  fixed torus mesh graph descriptor, fabric_config FABRIC_1D_RING, and per-system
  max_concurrency 128; the logical mesh (MESH_DEVICE) is left to the operator.
- Express and enforce targets per system for BLACKHOLE_GALAXY so a data-parallel
  choice cannot move the bar: DeviceTypes.expresses_targets_per_system() and
  get_perf_reference() now resolve from the device key and skip DP scaling.
  Wormhole Galaxy DP specs keep the per-replica behaviour (regression-guarded).
- Add tests/test_perf_reference_per_system.py and the determination record under
  docs/rfp/.

Ref: llm-gauntlet#68 (parent tenstorrent#56)
Co-authored-by: Cursor <cursoragent@cursor.com>

* Enforce scaling-quality three-point sweep coverage (readiness 5.7)

The scaling-quality rubric fits time-to-first-token vs. input length
separately at each graded concurrency level, so every graded concurrency
level must carry at least three distinct input lengths or its regression is
meaningless (RFP Appendix B.1/B.2/F.1). The auto-generated sweep pairs each
input length with concurrency [1, per-shape-allowed-max], and the allowed max
shrinks as input length grows, so at the Milestone-0 Blackhole Galaxy config
(max_context=262144, max_concurrency=128) with the token pool defaulted to
max_context only ISLs 128 and 1024 reach concurrency 128 -- two points, the
exact failure this issue guards against.

- workflow_types: DeviceTypes.grades_scaling_quality() -> True for
  BLACKHOLE_GALAXY.
- benchmark_config: reusable validator scaling_quality_coverage_violations
  plus feasibility helpers input_lengths_reaching_concurrency,
  max_gradeable_concurrency (120 at the default pool), and
  min_token_pool_for_concurrency (278528 for concurrency 128).
- benchmark_configs: get_llm_configs fails fast for scaling-quality-graded
  devices when the post-cap graded set violates the rule, with an actionable
  message. Vacuous for the current catalog; activates when blackhole_galaxy
  targets are authored (tenstorrent#66/tenstorrent#67).
- dev/llm.yaml: each M0 BLACKHOLE_GALAXY spec carries a [TBD - Performance]
  max_tokens_all_users_override placeholder (floor 278528) to lift a third
  input length to the top graded concurrency.
- tests + determination doc (docs/rfp/m0-scaling-quality-sweep-coverage.md),
  cross-linked with the 5.4 device-config doc.

Issue: llm-gauntlet#64 (Blocker) - Parent: tenstorrent#56
Co-authored-by: Cursor <cursoragent@cursor.com>

* Verify AIPerf benchmark path end-to-end on Blackhole Galaxy (readiness 7.1)

Confirm the `--tools aiperf` path works end to end against a model "served" on a
32-chip Blackhole Galaxy, using a mock OpenAI-compatible server
(llm-d-inference-sim) in place of real hardware. The harness exercises the real
repo code path -- AIPerfDriver -> AIPerfParser -> apply_target_checks ->
report_module -- so only the inference server and tokenizer are stand-ins.

- scripts/verify/run_aiperf_bh_galaxy_e2e.sh: starts the seeded mock, builds the
  LLM_AIPERF venv, runs the harness, tears down. Every knob overridable via env.
- scripts/verify/aiperf_bh_galaxy_e2e.py: drives the sweep, grades tiers, and
  renders the report/summary. Sweep grid (--isls/--concurrencies/--osl) and
  targets (--targets-json) are configurable.
- scripts/verify/targets_deepseek_v4_flash.json: DeepSeek-V4-Flash tiered targets
  derived from the Appendix B sheet (interactivity 100/25 t/s/u; TTFT 1410/2820 ms
  from prefill throughput at the system operating point), derivation recorded inline.
- docs/rfp/m0-aiperf-e2e-verification.md: determination doc -- approach, real vs
  mocked, both confirming runs (generic + DeepSeek targets, 6/6 PASS), and the
  remaining real-silicon gap.

Grades on the two concurrency-independent per-request bars (TTFT + interactivity);
aggregate/prefill throughput kept as documented system context so the bar does not
move with the sweep point's concurrency.

Readiness item 7.1 (llm-gauntlet#72). RFP requirements G.1.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Author DeepSeek-V4-Flash-0731 blackhole_galaxy targets from Appendix B sheet (tenstorrent#66/tenstorrent#67)

Seed the per-system Blackhole Galaxy performance targets for
deepseek-ai/DeepSeek-V4-Flash-0731 from the Milestone-0 Appendix B AIPerf sheet.

- model_performance_reference.json: new "DeepSeek-V4-Flash-0731" -> "blackhole_galaxy"
  entry with three graded points at the sheet's operating concurrency (64),
  bracketing the 8,192 mean ISL: ISLs {4096, 8192, 16384}, OSL 128. Each stores a
  single theoretical target -- interactivity tput_user=100 t/s/u (per-user, sheet
  target), aggregate decode tput=6413 t/s (sheet t/s/BH-GLX, per-system so used
  verbatim), and ttft from a linear-prefill model anchored at the sheet's
  8192->1410ms point (per-request prefill rate 371,835/64 = 5,810 tok/s): 705/1410/2820.
- dev/llm.yaml: set the DeepSeek BLACKHOLE_GALAXY spec max_concurrency to 64 (the
  sheet's max-num-seqs @ max context; was a placeholder 128) and enable
  max_tokens_all_users_override=1056768 (=64*(16384+128)) so all three graded ISLs
  reach concurrency 64 -- otherwise the pool defaults to max_context and ISL 16384
  demotes below 64, breaking the scaling-quality three-point rule (readiness 5.7,
  enforced by get_llm_configs).
- docs/rfp/m0-deepseek-v4-flash-targets.md: determination doc with the full
  derivation and open decisions for tenstorrent#66/tenstorrent#67 + Partner (TTFT off the anchor is
  modeled not measured; single graded concurrency level; repo-default tiering vs the
  sheet's per-metric downrates; dev placeholders pending the contributed impl).

Validated: get_perf_reference resolves the 3 points per-system (unscaled),
get_llm_configs builds the sweep with graded set {(4096,64),(8192,64),(16384,64)}
and the three-point guard passes; perf/scaling-quality/model-spec/benchmark-config
suites green (105 passed).

Builds on tenstorrent#68 (device config), tenstorrent#64 (scaling-quality rule), tenstorrent#72 (AIPerf E2E).

Co-authored-by: Cursor <cursoragent@cursor.com>

* Clean up RFP branch: drop llm-gauntlet references, revert accidental tt-media-server changes

- docs/rfp/*, scripts/verify/*, dev/llm.yaml: remove all references to the
  llm-gauntlet repo (repo name, issue URLs, and bare #NN issue numbers that would
  mis-link on tt-inference-server). Determinations now reference the Milestone-0
  readiness item / RFP section numbers instead.
- tt-media-server/: revert changes that were pulled into this branch by accident
  (blaze_runner scheduler-diagnostics additions and the tt-llm-engine submodule
  pointer bump) back to the tt-rfp-m0 base.
- workflows/model_spec.py: make the DeepSeek-V4-Flash stub impl code_path an
  obvious "<FILL IN>" placeholder for the Partner instead of a real-looking path.

Dev catalog still expands the DeepSeek spec (code_path="<FILL IN>", 3 perf-reference
points, max_concurrency=64); perf/scaling-quality/model-spec suites green.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Update workflows/model_specs/dev/llm.yaml

set this value to accommodate max_concurrency=32 full-context requests

Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com>

* Update workflows/model_specs/dev/llm.yaml from target sheet

Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com>

* Update workflows/model_specs/dev/llm.yaml from target sheet to be sweeping all the way up to max-num-seqs + max-seq-len

Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com>

* Update workflows/model_specs/dev/llm.yaml

Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com>

---------

Co-authored-by: Logeshwaran Elanchelian <ctr-lelanchelian@ext.tenstorrent.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ljubica Muravljov <lmuravljov@tenstorrent.com>
Co-authored-by: Kosta Novokmet <knovokmet@tenstorrent.com>
Co-authored-by: Veljko Cankovic <vcankovic@tenstorrent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com>
Settles readiness §9.2 (llm-gauntlet tenstorrent#78) before the remaining sweep entries are
authored (tenstorrent#66), so the same value is not written twice under two different
meanings. It already happened once: PR tenstorrent#4911 stored the roofline sheet's figures
directly and flagged the open question. The sheet's figures are peaks, and the
per-metric downrates still applied, so those entries set the bar 2x (TTFT) and 4x
(throughput) harder than intended.

The convention, now written down in docs/rfp/m0-target-convention.md: `theoretical`
holds the PUBLISHED target, which is peak x downrate (prefill 0.50, interactivity
and decode 0.25). The peak never enters the file. Downrating at authoring time is
forced rather than chosen — the two factors differ by metric and perf_targets_map
carries one percentage per tier applied to every metric, so it cannot express
them.

DeepSeek-V4-Flash-0731 re-derived accordingly: ttft 1410/2820/5640, tput_user 25,
tput 1603. Cross-check that catches exactly this class of error next time —
tput_user x concurrency should approximate tput (25 x 64 = 1600 ~ 1603).

Three defects found while making the gate work, each of which fails silently and
in the safe-looking direction:

All three Milestone-0 models sat at status EXPERIMENTAL, whose
required_target_tiers is empty. Because evals_enforced reuses that same signal,
accuracy failures were waived too — a submission could fail every performance
target AND every eval and still be accepted. The issue describes only the
performance half. Now FUNCTIONAL, with one tier named `functional` at 1.0 holding
the published value. The tier is named for the status that enforces it:
required_target_tiers matches on NAME, so a tier called `target` under FUNCTIONAL
would be computed, reported and then ignored. TOP_PERF is the only status that
enforces `target`, and it asserts near-peak optimisation, which is not a claim
Milestone-0 makes about a first-of-kind bring-up.

_verdict() read the `target` tier by hardcoded name. With any perf_targets_map
that does not define that name it returned NA, and an NA point is counted
ungradable rather than failed — so a point missing its target by any margin was
accepted. It now grades the strictest tier actually present, which is unchanged
for every model on the default ladder.

DeviceModelSpec.perf_targets_map existed, was carried through expand_to_specs and
was never read when deriving tiers; the model-wide map applied to every device.
Setting it did nothing, silently, despite readiness §9.2 describing it as the
supported override. It is needed here because gemma-4-31B-it's template also
covers P300X2 and GPU, which must keep the ordinary ladder.

tolerance now reaches the check. It defaulted to 0.0 and was never populated from
the reference file, so the gate demanded beating target outright while G.2.4
published an allowance. Read per point beside `targets` — a policy knob, not a
measured metric — defaulting to 0.0 so no existing model changes. Milestone-0
publishes 0.10.

Verified end to end on the DeepSeek mid point (target 2820 ms): 20% over blocks
and names the point, 5% over passes on tolerance, 10% under passes, and the same
20%-over point is waived at EXPERIMENTAL — isolating the status as the cause.

22 new tests, including a guard that each M0 spec's tier names are a subset of
its status's enforced tiers, read straight from the committed YAML.

Full suite: 1766 passed, 10 skipped.

Refs tenstorrent#78

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_module/submission.py, added in tenstorrent#4911 and now the documented submission
path in RFP I.1, read target_checks["target"] by name. Under the Milestone-0
single-tier configuration (tenstorrent#78) that tier is called `functional`, so the lookup
found nothing, every graded point was silently skipped, and the assembler would
have produced a submission with no points at all — scoring zero across the board
with no error.

Same defect as the one just fixed in _verdict, in a second place, which is the
argument for not having the rule written twice. Adds
llm_module.target_checks.graded_tier as the single definition of "the tier a
report is graded against": the strictest tier actually present, which is
unchanged for every model on the default ladder. submission.py now calls it, and
its docstring no longer names a tier.

Verified both ways round: the assembler finds the targets whether the tier is
named `functional` or `target`, and with the full ladder present graded_tier
still selects `target`.

3 new tests. Full suite: 1769 passed, 10 skipped.

Refs tenstorrent#78

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…h corners

Extends the blackhole_galaxy sweep from 3 points to 22: eleven input lengths at
both concurrency corners. Powers of two 1K..512K, then a context-saturating
point. The three existing entries are preserved exactly — 4K/8K/16K at
concurrency 64 still read 1410/2820/5640 ms.

The KV pool was the blocker, and the fix was already written down. The value was
1,056,768 (= 64 x 16,512), sized only for the three ISLs then graded, while its
own inline comment read `64 * 1024 * 1024`. The comment held the intent: 64
concurrent requests at the full 1M context, 67,108,864 tokens. Under the old
value every input length above 16K demoted — 32K->32, 64K->16, 128K->8, 256K->4,
512K->2 — which breaks the three-input-lengths-per-graded-level rule (tenstorrent#64) and
leaves seven concurrency levels where Appendix B.5's per-point weights are
defined over exactly two. At 67,108,864 every input length reaches 64, and the
longest does so exactly: 67,108,864 / (1,048,448 + 128) = 64.

A full 2^20 input cannot be swept. 1,048,576 + 128 exceeds max_context, and
get_benchmark_max_concurrency answers that by silently returning concurrency 1
rather than rejecting the point — so the mistake would have surfaced as a stray
concurrency level, not an error. The top input is max_context - osl = 1,048,448,
which saturates the window exactly.

Targets follow the convention: linear prefill over the downrated rate
(371,835 x 0.50 = 185,917.5 tok/s aggregate), shared by the requests in flight,
so the loaded corner is exactly 64x the idle one at every input length.

Two corrections to the existing values:

`tput` is now 25 x concurrency (25 idle, 1,600 loaded) rather than the sheet's
1,603. Measured aggregate decode is *defined* as tput_user x concurrency
(llm_module.parsers.base.decode_throughput), so a target derived any other way
means a system hitting interactivity exactly still misses the bar. The 3-token
gap was the sheet's independent rounding.

Values are stored to 4 decimal places, not 1. At 1 dp the idle target at ISL
1,024 rounds to 5.5 and x64 gives 352.0 against the loaded corner's 352.5 — the
convention promises a Partner can recompute any published value, and that promise
fails at 1 dp.

Verified with ONLY_BENCHMARK_TARGETS=1 (the setting RFP I.1 mandates): exactly 22
points, all graded, two concurrency levels, eleven distinct input lengths each,
scorecard weights summing to 1.0000 with the loaded corner at 0.75 and the
heaviest point being the longest input under load. Rough sweep runtime ~35 min.

The JSON was patched by brace-matching the DeepSeek object rather than
round-tripping the file, because that file has duplicate keys (tenstorrent#101) and a
load/dump silently drops 217 lines.

7 new tests covering the sweep shape: both corners at every length, no point
exceeding the context window, targets linear in input length at each corner, the
loaded corner exactly 64x the idle one, and tput = tput_user x concurrency.

Full suite: 1774 passed, 10 skipped.

Refs tenstorrent#66
Eighteen graded points: nine input lengths (1K..128K by powers of two, then a
context-saturating 262,016) at both concurrency corners, 1 and 32.

Targets are 123,072 tok/s aggregate prefill and 35.0 t/s/u interactivity, both
supplied already downrated, so they are stored verbatim — unlike the DeepSeek
sheet row, which gives peaks and separate downrate factors. TTFT is the linear
prefill model over that rate divided by the requests in flight, making the loaded
corner exactly 32x the idle one at every input length. `tput` is 35 x concurrency
because that is how the measurement is defined.

Note for anyone deriving from earlier work: an aggregate prefill figure of
184,608 tok/s circulated and is 1.5x the real target. The llm-gauntlet simulator
profile still carries it and must be corrected before any calibration is taken
from its output.

Filed under `gemma-4-31B-it`, upper-case B, which is what the Milestone-0
tt_transformers spec derives. The pre-existing lower-case `gemma-4-31b-it` is the
FORGE spec's key — it derives that spelling to match tt-media-server's ModelNames
constant and owns the p300x2 entry. Both keys now exist and neither is renamed;
renaming either would move targets off the spec that reads them, which is why
tenstorrent#4884 only added a warning. The case-mismatch warning for this spec is
now gone because the key it wants exists.

Two spec fixes, both latent until this point:

max_tokens_all_users_override read `32 * 256 * 1024` unquoted. YAML does not
evaluate arithmetic, so the value loaded as the STRING '32 * 256 * 1024' and
get_benchmark_max_concurrency raised TypeError on `str // int`. Nothing caught it
because the model had no blackhole_galaxy targets, so the sweep never reached that
code path — adding targets would have broken it immediately. The comment's
arithmetic was right; only the literal was wrong. Now 8388608, which is also
exactly what the sweep needs: 8,388,608 / (262,016 + 128) = 32.

max_concurrency 128 -> 32, the expected max-num-seqs at full context and the
concurrency the targets are authored at. 128 was a placeholder.

A full 2^18 input cannot be swept: 262,144 + 128 exceeds max_context, and
get_benchmark_max_concurrency responds by silently returning concurrency 1 rather
than rejecting the point, so the mistake would surface as a stray concurrency
level. The top input is max_context - osl.

Verified with ONLY_BENCHMARK_TARGETS=1: exactly 18 points, all graded, two
concurrency levels with nine distinct input lengths each, scorecard weights
summing to 1.0000 with the loaded corner at 0.75, longest point fitting the
context window exactly. Rough runtime ~31 min.

The sweep-shape tests are now parametrised over both Milestone-0 models, plus one
asserting the two gemma keys stay distinct.

Full suite: 1780 passed, 10 skipped.

Refs tenstorrent#66
The RFP now covers two models: gemma-4-31B-it and DeepSeek-V4-Flash-0731.

Mistral-Small-4-119B-2603 keeps its dev-catalog scaffold — it is real work and the
model may return in a later milestone — but no longer carries any Milestone-0
grading configuration: `perf_targets_map` removed, `status` back to EXPERIMENTAL,
and it never had blackhole_galaxy targets. A test asserts that, because
`status: FUNCTIONAL` plus a perf_targets_map is the signature of a graded model
and the obvious mistake is to add it back by making the three specs look alike.
The header comment says so too, at the point someone would edit.

Also removes the stale blackhole_galaxy entry for Mistral-Small-3.1-24B-Instruct-2503.
That is the placeholder tenstorrent#66's problem statement calls out — three values copied
verbatim from gemma-4-31b-it's p300x2 row — and it was not merely misleading: since
tenstorrent#4911 added the scaling-quality coverage guard it made `get_llm_configs` RAISE for
that spec on BLACKHOLE_GALAXY, because its single graded point gives concurrency 1
only one input length. Pre-existing, confirmed by stashing today's work. Removing
the entry leaves the device row intact and its points ungraded, which is what an
unmeasured model should look like.

blackhole_galaxy targets in the reference file are now exactly the two RFP models.
Verified every BLACKHOLE_GALAXY spec builds its sweep without raising, and only
those two are graded.

Full suite: 1779 passed, 10 skipped.

Refs tenstorrent#66
test_an_entry_without_a_tolerance_still_defaults_to_zero read
Mistral-Small-3.1-24B's blackhole_galaxy entry, which the previous commit removed
as a stale placeholder. Broken by that commit and pushed with it — my verification
chain piped pytest into tail, so the shell saw tail's exit code rather than
pytest's.

Rewritten to assert the property across every non-Milestone-0 model in the
reference file instead of one hand-picked entry: only the two RFP models opt in to
a tolerance, and every other targeted point must still default to 0.0. Now covers
~600 points rather than 1, and cannot be broken by removing any single entry.

Full suite: 1779 passed, 10 skipped (verified on pytest's own exit code).
The multipliers convert the mean TTFT target into a qualifying value for each
graded percentile, and they decide 38 of the 100 core points. They were plausible
round numbers — 0.90 / 1.30 / 1.80 — chosen before any distribution had been
looked at.

Measured on the gpt-oss-120b release run, 16 operating points on real silicon:

  p50 / mean   median 1.000   range 0.795-1.150   published multiplier 0.90
  p99 / mean   median 1.192   range 1.000-1.852   published multiplier 1.80

Both were wrong, in opposite directions. At 1.80, a submission that merely met its
mean target with an ordinary tail scored 14.9 of the 22 points on the rubric's
heaviest line, free. At 0.90 the median line asked p50 to sit 10 % BELOW the mean,
which essentially never happens, so those 5 points were unreachable by anyone.
Tail discipline at 4.0 saturated outright: real ratios are ~1.19, so every system
took all 4 points. Together that is 21.9 of 42 tail-sensitive points handed to a
submission for meeting the target and nothing more — the opposite of what K.2
requires, which is that meeting the target scores zero.

Now p50 1.00 and tail discipline 1.25 -> 1.05, both at the measured values. p99 is
set at 1.35 rather than the observed 1.19, deliberately: tails widen under load and
on a first-of-kind bring-up, and a qualifying value at the median observation would
put 22 points out of reach of a system with an ordinary tail, which is a worse
failure than a generous one. p90 was not captured in that run — it predates the
percentile work in tenstorrent#57 — so 1.12 is interpolated and is the least certain.

Appendix F.2 and F.3 are recomputed, including the example's measured p99 column:
those values were chosen to give a spread against 1.80, and against 1.35 three of
the ten points clamped to zero, which teaches nothing. Re-chosen to model a system
with a tight tail that widens under load — the shape the line exists to reward.
The p99 line now returns 8.72 of 22 and the core total 50.56.

Two tests were tuned to the old example rather than to the invariant, and are now
robust to it: the summed-before-rounding test asserts an exact identity instead of
a divergence that only appeared with particular numbers, and the prefill-dominance
test asserts prefill is the largest group rather than clearing a fixed share.

Full suite: 1779 passed, 10 skipped.

Refs tenstorrent#118
The RFP previously said Milestone-0 "adopts unchanged" the benchmarking tool's own
service-level thresholds. That makes a constant in llm_module a published
requirement by reference: editing it would move the bar a Partner is graded
against, silently and with no review of the RFP.

Appendix B.3 now states the values as Milestone-0's own — the same numbers today —
and this pins the tool's constants to them, so a change to either is a test
failure naming the other. The Appendix governs; the tool is checked against it,
not the reverse.

Refs tenstorrent#74
…03 / 1.02

They were 1.15, which asked a Partner to be 15 % more accurate than the reference
running the same weights. Accuracy is a property of the model, not the silicon: a
correct implementation scores about equal to the reference, so 1.15 was unreachable
by engineering and 22 of the 100 core points were dead weight — the same defect as
the p99 multiplier, in a different place.

The achievement actually being rewarded is holding reference accuracy on a
different compute class at a different numerical precision, and then edging past
it. 1.03 for agentic and 1.02 for standard put the line in the band submissions
will really occupy.

Agentic is the higher of the two because its task sets are smaller — 89 tasks
against 198 samples — so an above-parity result needs more headroom before it means
anything. Both remain close to the noise floor of their task sets, which is stated
in the RFP rather than hidden: these lines separate a submission that holds
reference accuracy from one that does not, and are not fine instruments above that.

F.2's example values move from 1.09 and 1.04 to 1.02 and 1.01 — against the new
range the old ones clamped to full marks, which demonstrates nothing. Core total
53.69, overall 67.56.

One test hardcoded the forgone score in the failed-reproduction note. It now
derives it from the baseline: the point is that the cost is stated, not that it is
any particular number.

Full suite: 1784 passed, 10 skipped.

Refs tenstorrent#118
Accuracy leaves Part K entirely. It remains a hard pass/fail gate in Part I via
requirements H.5 — unchanged and still blocking — but it no longer earns rubric
points.

The reason is that rewarding accuracy above the reference invites Partners to chase
fractions of a point on a quantity that is a property of the model's weights rather
than of their implementation. A correct implementation scores about equal to the
reference; anything above that is mostly sampling noise on 89-task and 198-sample
sets. Points that reward noise are worse than no points.

Setting the excellence values to 1.00 would have expressed the same intent but left
22 weighted points unearnable by construction: qualifying == excellence makes the
fraction 0/0, so the best achievable core score would silently have been 78 against
a stated 100, and every scorecard would have reported two lines as unscoreable when
they were in fact deliberately not scored. Removing them says it once, in the
weights.

Core is therefore **78**, not 100, and every point in it is winnable. The bonus
stays 20. F.2 rolls up to core 40.69 and overall 54.56; prefill's share of core rises
from 53 % to 66 %, which is the balance the rubric was already trying to express.

report_module/submission.py loses --agentic-eval and --standard-eval, which now feed
nothing. Per-task eval ratios are still written to _evidence, because whoever
adjudicates the Part I gate should be able to see them without opening the run.

Four tests referenced the eval lines only as convenient examples of a once-scored
line and now use run_to_run_cov or contribution_quality instead. Two that asserted
100 and a fixed prefill share are updated to 78 and to comparisons that hold
regardless of what the example scores.

Full suite: 1781 passed, 10 skipped.
…m fork to vllm-tt-plugin + upstream vLLM (tenstorrent#4907)

* initial changes for switching to vllm plugin instead of the fork

* remove uv cache after building stage so we can shrink docker image
The vLLM platform plugin now lives in its own repository (tenstorrent#4907, cherry-picked in
the previous commit), and validate_setup's failure message is where a Partner
actually learns how to install it. It said `git clone` with no revision, which
gets whatever the default branch holds today.

That matters more than it looks: the plugin owns the vLLM version pin and its
dependency overrides, so a different plugin revision is a different vLLM. The
Milestone-0 baseline pins the plugin to a specific commit, and a Partner following
this message verbatim would have installed a different one and had no way to know.

The message now shows the checkout step and says why, while making clear it is
skippable when nothing pins it.
kbot added 27 commits November 12, 2026 22:05
V41_DECODE_STAGE_BATCH=1: the decode step stages its packed inputs and
Engram rows with one ttnn.copy_host_to_device_tensors call (one wake-up
of the per-device dispatch workers instead of three): served, the
one-row step's staging 2.7 -> 2.2 ms, the period 59.0 -> 58.5, tput_user
17.1 -> 17.2 to 17.3 at points 1, 5 and 8.

Action-Id: ds41/optimize/102
V41_DECODE_SDPA=flash: the one-live-row decode capture runs the
flash-decode SDPA program (the live rows keys in units of 32 over the
grid, a log-sum-exp merge per output tile, the partials through the
merge cores L1). Right on every live-row pattern in both KV formats
against the local path (PCC 0.9998), the c1 chain 102 -> 73 us a layer,
the one-row step 49.2 -> 48.0 ms of device; served, tput_user 17.2 ->
17.6 at points 1, 5, 8 with the replay-state, production-order L1 and
generator gates as before.

Action-Id: ds41/optimize/113
TT_METAL_MESH_WRITE_INLINE=1: the decode step one batched host-to-device
round runs its per-device writes on the calling thread (tt-metal
mesh-write-inline) instead of as one task per device on the dispatch
worker pool, whose wake-up after the ~47 ms device wait cost the round
~2 ms served. Bit for bit (the generator gate four texts); served, the
staging 2.2 -> 0.7 ms a step, the period 58.5 -> 56.8 ms, tput_user 17.3
-> 17.8 at points 1, 5, 8.

Action-Id: ds41/optimize/113
TT_V41_SPEC=3 (the verify decoder with the DSpark drafter in its
capture) with the plugin's ragged contract (the speculative-config of
vLLM's custom_class method naming vllm_tt_plugin.model_owned_spec, 3
tokens: K + 1 tokens a step through the scheduler's accepted counts) and
TT_V41_SPEC_GREEDY_ALL=1 (every request verified greedily: the smoke's
knob, since the benchmark samples at temperature 1 and the round has no
sampler yet).

Action-Id: ds41/optimize/154
The round's warm-up captures 16 vd: variants (a round per user index and
draft count) and the plain one-slot step per user index beside the
32-slot step, ~0.8 GB of trace region, after which the catalog's
per-length prefill captures overran the 2 GiB region of the plain
configuration. 3 GiB, at a cost of 1 GB of the DRAM margin.

Action-Id: ds41/optimize/154
The catalog's speculative entry drops TT_V41_SPEC_GREEDY_ALL: a sampling
request's round draws the served sampler on every verify row and accepts
while the draw equals the next draft, so the benchmark's temperature-1
requests are speculated with the same output law as plain serving
instead of being verified greedily.

Action-Id: ds41/optimize/154
The catalog sets TT_V41_SPEC_RINGS=slot: the DSpark drafter keeps a ring
per decode slot, so every request speculates (one ring set let the last
joined request speculate and ran the others plain, which at c32 was the
plain step plus one eighth of a round). A step of one live slot is the
fused c1 round; a step of several runs every slot of the pool as a block
of 1 + K verify rows and one standalone draft call for the four slots of
every mesh row.

Action-Id: ds41/optimize/159
submission.merge_points and run_to_run_cov keyed a point on the report's
measured input_sequence_length, which is AIPerf's mean prompt length and
can differ from the registered length by a token. Repeat runs could
become separate points and their variation could be computed over the
wrong groups. registered_point reads the sweep identity from the
target_checks block title; point_key falls back to measured fields when
a block has no such title. comparison.collect_points uses the same
identity. Tests cover token-apart repeats, merged variation, title
parsing and fallback behavior.

Action-Id: ds41/optimize/161
The four flags (TT_SCHED_DEFER_FINAL, TT_V41_DEFER_FINAL,
TT_V41_RESTORE_TRACE, TT_V41_RESTORE_TRACE_MAX_WIDTH 131072) in the
DeepSeek-V4.1-Flash entry's env_vars, so the served configuration runs
every request's final prefill chunk after the wave's bulk prefill (c32
mean TPOT 0.76x at 8K and 0.12x at 128K of the serial path's on the
plain tree), composed here with the speculative round.

Action-Id: ds41/optimize/171
TT_SCHED_DEFER_FINAL "hold" in the served entry: the plugin admits no
prefill while a cohort decodes, so a c32 point's replacements form the
next cohort together instead of running their bulks between the
stragglers' decode steps.

Action-Id: ds41/optimize/174
TT_SCHED_FINAL_STEP=1 beside TT_SCHED_DEFER_FINAL=hold (vllm-tt-plugin):
a multi-chunk request's bulk prefill runs to n - 1 tokens and is joined
by the stock path, and the cohort's parked requests take their first
token from one decode step instead of a serial final chunk each. The
generator's parking and the restore captures (TT_V41_DEFER_FINAL,
TT_V41_RESTORE_TRACE and its width cap) come out: nothing is parked or
restored on this path.

Action-Id: ds41/optimize/192
Under TT_SCHED_FINAL_STEP a stepped prompt reaches the generator as n -
1 tokens, so the sweep's prompts (the listed length and the one or two
below it) need the final-chunk captures at +-2 of each listed length:
with the default slack of 1, 6 of 128 finals at 64K and 1 at 16K ran
eagerly (4 to 13 s each). TT_V41_TRACE_PREFILL_SLACK=2 in the catalog
entry.

Action-Id: ds41/optimize/194
TT_V41_PREFILL_HOST_BATCH=1: the generator's state reset and each
chunk's staging issue their host-to-device copies as one
ttnn.copy_host_to_device_tensors each instead of ~55 single copies a
one-chunk call, bit for bit; the flag-on arm of the served pair at the
c32 short points.

Action-Id: ds41/optimize/208
TT_V41_CHUNK 10240 with max_num_batched_tokens to match, and 5120 added
to TT_V41_SHORT_CHUNK so prompts of 2,049 to 5,120 tokens keep their one
5,120-row chunk. Measurements showed the larger chunk at -9 % of prefill
device time per token at 16K and from 128K up; the per-chunk items halve
per token and the padded-row items do not move.

Action-Id: ds41/optimize/209
TT_V41_PREFILL_RESET_DEVICE 1 and TT_V41_PREFILL_INDEX_BOUND 1 beside
the 10,240 chunk at the native 1,053,696 context: whether the index
bound, by not building the candidate source's resident ratio-1 mask,
lets the 1M score bucket's generic capture fit the chip's DRAM, which it
did not without it.

Action-Id: ds41/optimize/215
TT_DETOK_PRIME_TOKENS=16 in the catalog entry: the vllm-tt-plugin's
general entry point replaces tokenizers' DecodeStream with one primed
with the prompt's last 16 ids instead of the whole prompt, which vLLM
0.24.0's streaming detokenizer decodes twice at a request's first token
(0.26 us an id; 32 requests' first tokens from one decode step paid it
serially, 9.7 s at 1M, in the API server's output loop while the engine
decoded on). The emitted text is unchanged.

Action-Id: ds41/optimize/230
TT_V41_PACK=1 with TT_V41_PACK_LAYOUTS and the scheduler's TT_SCHED_PACK
(the same list: 4x1028,3x1028,2x1028,2x2052): a prefill step admits the
whole short prompts of a listed layout at the head of the queue and the
generator runs them as one packed 5,120-row chunk, each prompt joining
its decode slot from its own segment. The points it moves are the short
c32 ones (12 and 13: four 1,028-token or two 2,052-token prompts a
chunk); every other prompt takes the single path as before, and at c1
the flags are inert.

Action-Id: ds41/optimize/233
The packed captures are the last of the warm-up, after the decode
variants, the 352 joins, the per-length prefills and the 43 generic
chunk captures, and the declared 3 GiB region had room for three of the
four layouts (the fourth failed at its first 57 KB with 0 B free). A
pack capture is ~37 MB and the four with their 160 segment joins ~185
MB; the region grows by 256 MiB a chip, which the 1M bucket's capture
leaves free since the index bound.

Action-Id: ds41/optimize/239
set_runtime_env_vars expands $VAR and ${VAR} with os.path.expandvars,
which leaves the shell's default-value form untouched, and a value the
spec sets always replaces the launch environment's. A spec that wants a
default the operator can override at launch has no way to say so. The
form ${VAR:-default} now takes VAR from the launch environment when it
is set and non-empty, else the default, before the existing expansion;
plain values keep replacing the environment. Two tests: the helper's
cases, and a spec value overridden by the launch environment while a
plain value is not.

Action-Id: ds41/optimize/244
The served packing arm is chosen by the launch environment: the four
flags that differ between the arms are written ${FLAG:-default}, so the
entry serves hold-all (one-chunk prompts held for the cohort and drained
as the listed packs, each first token delivered as produced) unless the
server starts with TT_SCHED_RELEASE_FIRST=cohort (+release),
TT_SCHED_DEFER_FINAL=hold (pack-only) or that and TT_V41_PACK=0
TT_SCHED_PACK=0 (off). The comment lists the four arms.

Action-Id: ds41/optimize/244
Keep the embedded serving template in step with the plugin renderer.
Correct few-shot user merging and assistant transitions without changing
other serving parameters or single-user benchmark prompt formatting.

Action-Id: ds41/optimize/282
Add --eval-tasks independently of sample-limit controls for standard
and agentic evaluations. Reject unknown, duplicate or limited task
selections and preserve configured budgets, repeats and task objects.
Carry selection through runtime serialization and dependency setup.

Action-Id: ds41/optimize/283
Declare the V4.1 thinking and tool interface and pin the common eval
harness and Harbor reference. Run full tasks in explicit order with
independent coverage and accuracy verdicts, owned cleanup and resource
bounds. Isolate agentic Docker storage and networking from shared state.

Action-Id: ds41/main/optimize/284
Allow an explicit dotenv path so a full evaluation campaign cannot
inherit another workflow secret or experimental setting. Keep the
campaign credential file private and preserve reasoning in sample logs
for extraction and token-budget diagnostics.

Action-Id: ds41/main/optimize/284
Inspect inference HTTP failures as well as per-sample errors before
calling a standard task operationally complete. Do not turn transport
failures with a full-looking sample file into an accuracy verdict.

Action-Id: ds41/main/optimize/284
Resolve frontend tokenizer files from the same immutable snapshot as
the model and checkpoint encoder, rather than a mutable cached main
reference during a long evaluation campaign.

Action-Id: ds41/main/optimize/284
if not 0 < args.max_hours <= 72 or not 0 < args.max_artifact_gib <= 100:
parser.error("Campaign bounds must not exceed 72 hours / 100 GiB")
args.output.mkdir(parents=True, exist_ok=False, mode=0o700)
os.chmod(args.output, 0o700)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❗Cycode: SAST violation: 'Permissive file assignment (Python)'.

Risk Score: 59 (MEDIUM)
Severity: High

Description

Permissive file assignment exposes sensitive information by granting unnecessary read, write, or execute permissions to users without ownership privileges.

Cycode Remediation Guideline

✅ Do


  • Do keep file permissions as restrictive as possible to minimize the risk of unauthorized access. Use the principle of least privilege to only grant permissions that are absolutely necessary for the operation of the application.
    os.chmod("my_private_file.txt", 0o600) # only you have full read and write access
  • Do prefer assigning file permissions to 'groups' rather than 'other' when you need to extend privileges to users who are not the owners. This approach helps in limiting access to a more controlled set of users.

❌ Don't


  • Do not set a permissive umask value, as this can lead to overly permissive default permissions for new files and directories
  os.umask(0) # unsafe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants