Repository navigation
Conversation
…asure Both LLM benchmark parsers normalised tool output to a flat record that kept mean, p50 and p99 for time to first token, but only the mean for time per output token and end-to-end latency. No p90 for anything, no standard deviation for anything. The tools measure all of it; normalisation dropped it. Checking a real gpt-oss-120b release run showed the loss is not hypothetical: the raw vLLM result already contained median_tpot_ms, p99_tpot_ms, std_tpot_ms, median_e2el_ms, p99_e2el_ms, std_e2el_ms and std_ttft_ms — eight fields collected, written to disk, then discarded on the way into the report. Changes: - aiperf.py — emit p90/std for TTFT, and p50/p90/p99/std for TPOT and E2EL. metric_stat() already reads raw[metric][stat], so this extends the field map rather than adding extraction logic. - vllm.py — read the same percentiles. Keys beyond mean/median/p99 only appear when the run requested them; when absent round_metric yields None, never 0. That distinction matters: zero is the best possible latency, so a missing value coerced to 0 would score as a perfect result rather than as unmeasured. - drivers/vllm.py — pass --metric-percentiles 90,99. The driver already passed --percentile-metrics, but vLLM defaults the percentile list to "99", which is precisely why p90 was never emitted. It costs nothing alongside the existing p99 computation. - display.py — add the six missing column headings, so the new fields render as named columns rather than raw keys. Tests: four new cases covering AIPerf extraction, vLLM extraction, display-name coverage, and — the one that matters most — that an absent percentile arrives as None rather than 0 in both parsers. Full suite: 1574 passed, 10 skipped. Refs tenstorrent#57 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the previous commit, prompted by checking how real AIPerf output is read elsewhere in the tree rather than trusting the synthetic test fixture. `llm_module/drivers/aiperf_prefix_cache` consumes genuine AIPerf exports and reads the same three metric blocks — time_to_first_token, inter_token_latency, request_latency — with stats avg, p50, p90, p95, p99 and std. That independently confirms the extraction added previously, and exposes two gaps in it: - It reads `avg`/`mean` and `p50`/`median` as alternatives, because AIPerf labels those two statistics differently across versions. The benchmark parser tried only one spelling each, so an export using the other would parse fine on the prefix-cache path and silently yield None here. Added a `_stat_any` helper that mirrors the driver's fallback behaviour. - It extracts p95, which the benchmark parser skipped. Display headings for p95_ttft, p95_tpot_ms and p95_e2el_ms already existed, so the field was expected and simply never populated. The grading rubric does not read p95, but RFP requirement G.2.2 asks for every metric in the record. Both parsers are now symmetric on p95, and the vLLM driver requests `--metric-percentiles 90,95,99` so the two tools report the same percentile set and stay comparable. A shared test asserts an absent percentile is None rather than 0 in both, which is what caught the asymmetry. Full suite: 1575 passed, 10 skipped. Refs tenstorrent#57 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… targets
Groundwork for generating RFP Milestone-0 benchmark fixtures against a simulated
Blackhole Galaxy. Every change here is a real defect found by trying to run the
AIPerf benchmark path end to end; none is specific to simulation.
requirements/llm-aiperf.txt — drop torch, allow transformers 5.x
Nothing in this venv needs torch: aiperf does not depend on it and no other
installed distribution declares it. The pin broke two things. torch 2.4.0 is
incompatible with transformers 5.x (which needs DTensor from
torch.distributed.tensor, 2.5+), so loading the Gemma-4 tokenizer raised
ImportError; and the +cpu local-version wheel is published for linux/x86_64
only, so the venv could not be built on macOS or aarch64 at all. Removing it
also avoids adding protobuf, which the transformers 4.x SentencePiece path
would otherwise require. Verified: aiperf installs and
AutoTokenizer.from_pretrained("google/gemma-4-31B-it") loads in 4.0s
(GemmaTokenizer, vocab 262144) with no torch present.
llm_module/drivers/aiperf.py — pass --ui-type none
AIPerf defaults to a full-screen Textual dashboard. With no attached terminal
it emits alternate-screen escape codes and blocks forever, leaving a 0-byte
aiperf.log. Benchmarks are always run non-interactively.
workflows/model_spec.py — warn when a model has no reachable targets
A model whose targets are filed under a differently-spelled key resolved to {}
in silence: no targets, no warning, and every benchmark point then grading as
ungradable. gemma-4-31B-it hits this today — the reference file stores
"gemma-4-31b-it", matching the Forge dev spec and tt-media-server's
ModelNames constant, while the lookup derives "gemma-4-31B-it" from the prod
HF repo. Both spellings are deliberate and name the same model on the same
device, so which spec owns those targets is a question for the model owner;
this does not guess, it only makes the miss visible.
workflows/model_specs/prod/llm.yaml — add a GPU device spec for gemma-4-31B-it
The documented bring-your-own-server path (docs/gpu_workflows.md) lets the
benchmark workflows target an OpenAI-compatible endpoint with no Tenstorrent
hardware present. Follows the existing Llama-3.1-8B GPU entry.
charts/ — regenerated by workflows.helm_generator for the new device spec.
Upstream issues: tenstorrent#4883 (torch/transformers), tenstorrent#4884
(silent target miss). Both changes are branch-local to the RFP handoff and
carry revert notes.
Also worth reporting to ai-dynamo/aiperf separately: AIPerf hangs rather than
propagating a tokenizer error, writing nothing to its log. Confirmed no requests
reach the server while it sits idle.
Full suite: 1575 passed, 10 skipped.
Refs tenstorrent#57
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…xponent The Milestone-0 grading rubric weights prefill at 55 of 100 points, but nothing in the stack measures prefill directly. tt-metal has prefill_t/s only on the demo path, the normalised serving record has no equivalent, and WorkflowType. PREFILL_DECODE is not a prefill benchmark — it serves a mock stack for prefill/decode disaggregation smoke tests. So prefill quality is derived from time to first token across the sweep. Adds llm_module/derived_metrics.py: prefill throughput input_sequence_length / mean_ttft_ms, tokens/s ttft tail ratio p99_ttft / p50_ttft ttft scaling exponent fitted b in log(p50_ttft) = a + b*log(isl) The first two attach per point in the runner, next to apply_target_checks and following the same Block-in/Block-out shape. The exponent needs several input lengths at one concurrency, so it is fitted after the sweep completes. Fitted separately per concurrency level, never pooled. That is the entire point of the metric: a system can scale cleanly when idle and degrade badly under load, and one combined slope averages exactly that away. A test pins the behaviour with a synthetic system that is linear at concurrency 1 and quadratic at 32, and asserts the pooled fit would have reported neither. Everything is None when uncomputable, never 0. For a latency-derived figure 0 is the best possible result, so a coerced zero would score as perfect rather than as unmeasured — the same invariant established for the parsers in tenstorrent#57. A concurrency level with fewer than three usable input lengths gets None plus a warning naming the level, rather than a slope through two points that always fits perfectly and says nothing about curvature. This is the constraint behind tenstorrent#64. Verified against the real gpt-oss-120b release run: prefill throughput climbs 486 -> 6,977 tok/s across isl 128 -> 32768 as fixed overhead stops dominating, tail ratios land between 1.0 and 1.8, and exponents come out 0.51 at concurrency 1 and 0.83 at 32. It correctly declines to fit at concurrencies 3, 7, 15 and 31, where that run's auto-generated sweep left a single point each. Full suite: 1597 passed, 10 skipped. Closes tenstorrent#58 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in the tree could take two benchmark runs and say whether they agree —
searching for report-comparison, regression-comparison or baseline-diff turned up
nothing. The Milestone-0 reproduction check needs exactly that: a Partner reports
numbers, we re-run their submission, and every metric must agree within a stated
margin. Doing it by hand across two models, a full sweep, three runs each and
~15 metrics per point is slow and unreliable, and it decides whether a milestone
is credited.
Adds report_module/comparison.py plus a CLI:
python -m report_module.comparison \
--reported partner_run{1,2,3}.json \
--measured our_run{1,2,3}.json --margin 0.05
Exits non-zero when the reproduction fails, so it drops into CI unchanged.
Three decisions worth recording:
Medians across runs, not single runs. Each side supplies several reports of the
same configuration and the median of each metric is compared. A single run would
let noise decide a commercial outcome; a test pins that one bad run out of three
does not fail an otherwise-agreeing submission.
Benchmark sections are identified by content, not by `kind`. That field is not
stable — current parsers emit "benchmarks" while the real gpt-oss-120b release
report carries the tool name "vllm". Keying off the operating-point fields any
benchmark record must have handles both, and a future tool as well.
A value missing on ONE side fails; missing on BOTH does not. The first would let
an incomplete submission reproduce perfectly. The second is a different thing —
there is nothing to disagree about, so it is reported as a capture gap rather
than a mismatch. This distinction was found by running against the real report:
error_request_count is null throughout it, and under the stricter rule a report
failed to reproduce against ITSELF. That sanity invariant is now a test.
Verified on the gpt-oss-120b release run: self-comparison passes across 17
operating points; perturbing one point's mean_ttft_ms by 12 % fails with exit 1
and isolates that single metric at 12.00 % against the ±5 % margin.
Full suite: 1618 passed, 10 skipped.
Closes tenstorrent#59
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bonus rubric scores two prefix-cache values, and a machine could reach neither. The decode hit rate was reachable only by reimplementing the aggregated/disaggregated precedence rule, and the time-to-first-token uplift was computed inside the Markdown renderer while building a table — it never reached the report JSON at all. The scorecard generator cannot read a rendered table. Adds report_module/prefix_cache_uplift.py, applied after the sweep in the prefix-cache orchestrator, mirroring where apply_scaling_exponents sits in the benchmark path. It has to run there rather than per run because the uplift compares a treatment run against its matched baseline run. The sign is the part that would have gone wrong quietly. The renderer's ttft_uplift_pct is a latency delta, so a cache that HELPS produces a NEGATIVE number — correct for a table, inverted for the rubric, which asks for a percentage reduction where higher is better and 0 % is qualifying. Had the scorecard read the display field, a 44 % improvement would have scored zero and a cache that made time to first token 44 % worse would have scored points. The scored field is therefore named ttft_reduction_pct, carries the opposite sign deliberately, and both directions are asserted in tests rather than left to follow from the arithmetic. Two further decisions, both about not making Partners game things: The optional mooncake_trace scenario is excluded from the scored summary. If an optional run could move the score, a Partner would have to work out whether running it helped or hurt them, which is a perverse thing to make anyone reason about. A missing required scenario is reported, not averaged around. Taking the mean of whichever required scenarios happen to be present would quietly reward dropping the weakest one. The summary reports scenarios_missing and leaves the decision to the caller. Scenario means are taken within a scenario before averaging across scenarios, so a scenario measured at more operating points does not thereby count for more. Also single-sources the baseline-matching rule and the delta, which the renderer now imports. Display and score disagreeing about which baseline matches a treatment run is exactly the kind of thing that surfaces during a bid dispute. Full suite: 1640 passed, 10 skipped. Closes tenstorrent#62 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d zero
Zero failed requests is a binary requirement (RFP G.2.6) that nothing enforced.
A submission could pass acceptance with requests failing at every sweep point,
provided the ones that survived were fast enough — and speed measured over only
the requests that succeeded is not a result.
Fixing it turned up the reason the field looked unusable. Both parsers coerced a
measured zero to None:
aiperf.py if isinstance(value, (int, float)) and value: # 0 is falsy
vllm.py return int(v) if v else None # 0 -> None
So "no requests failed" and "nobody looked" were the same value, and the field
could not be graded at all. This is the None-vs-zero invariant the derived
metrics already keep, inverted: there, a missing value must not become 0; here, a
measured 0 must not become missing. Both directions destroy the distinction.
All 17 points of the gpt-oss-120b release run carry failed: 0 in the raw tool
output. Every one of them reached the report as null. Worth noting this is also
the true cause of the null error_request_count that tenstorrent#59 worked around — it was
neither an aiperf-versus-vLLM difference nor an unmeasured metric; the tool
measured it and the parser threw it away.
Parsers now return None only when the count is genuinely absent. Acceptance then
grades every benchmark block, not only the first:
count > 0 blocks, naming the count
count == 0 passes — a real measurement of the required outcome
absent blocks, because the requirement is that zero was CONFIRMED, and
treating absence as a pass is how an unverified requirement
silently becomes a satisfied one
Behaviour change worth flagging: a benchmark block with no error count now fails
acceptance where it previously passed. After the parser fix both supported tools
populate it, so this should only catch genuinely unreported cases.
Verified against the release run: 17/17 points now report 0, the clean run
accepts, and injecting 5 failures at one point blocks with that point named.
Note the module named in the issue, workflows/acceptance_criteria.py, is imported
by nothing but its own tests — every live caller goes through
report_module/acceptance_criteria.py, which is what this changes. The legacy copy
is left alone rather than growing a second implementation of the same rule.
Full suite: 1647 passed, 10 skipped.
Closes tenstorrent#61
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds report_module/scorecard.py and a CLI. This ranks competing commercial bids, so hand-computation was never acceptable: error-prone, not auditable, and not reproducible if a Partner disputes a score. Implements Part K end to end. Each graded point's qualifying and excellence values are derived from that point's own target using the published B.5 multipliers rather than authored per point, so there is no second set of numbers to drift and a Partner can recompute any value. Per-point weights combine a log2(input length) factor with the 25/75 concurrency shares and sum to 1. Scaling quality is fitted per concurrency level and combined by share — never pooled, because a pooled fit averages away a system that scales cleanly when idle and degrades under load. Both K.8 rules are applied after scoring and recorded on the line, so a scorecard shows what a failed reproduction or a waiver actually cost. Every intermediate is kept, per point, in both the markdown and the JSON: a disputed score has to be walkable back to the measurement it came from. An unscoreable line still contributes zero — the rubric has no other option — but it is listed separately from lines that genuinely scored zero. A line measured exactly at its qualifying value and a line nobody measured are different facts about a submission and must not look identical. The golden test reproduces Appendix F.3 exactly: all ten per-point weights, all ten p99 fractions, the 0.5649 line fraction and the 12.43 line score. It also found a defect in the published example. F.2's core total and overall are sums of ROUNDED group subtotals — 31.99 + 4.70 + 9.87 + 9.04 = 55.60 — while the unrounded scores sum to 55.5939, which rounds to 55.59. Overall is 69.46, not 69.47. Rounding is a display step; letting it feed back into the arithmetic makes a total depend on how many decimals the scorecard happens to print, and the error grows with every line. The code sums at full precision and the RFP is corrected to match, with a test pinning the distinction so it cannot quietly come back. Targets still have to be wired in from model_performance_reference.json to score a real run; that waits on the sweep and target work (tenstorrent#66, tenstorrent#67). The CLI takes a submission JSON so the engine is usable and testable now. 45 tests. Full suite: 1692 passed, 10 skipped. Closes tenstorrent#60 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--output-tokens-mean/-stddev only *request* a fixed output length. Without ignore_eos the server is free to stop at its natural EOS, so the osl axis of a benchmark sweep stops meaning anything. Measured against llm-d-inference-sim asking for osl=128: 27-130 tokens returned (std 41) before this change, 131-135 (std 1.5) after. Output length feeds TPOT, E2EL and both throughput columns, so entire rows were affected -- a simulated corpus generated at target reported decode throughput at 0.787x of target for ISL 131072 purely from truncated generations. This restores the pairing the sibling drivers already use: aiperf_spec_decode.py couples ignore_eos:true with output-tokens-mean/-stddev, and the agentic-traces scenario enforces it too. This driver requesting a fixed length without enforcing it was the outlier. Hooks (ruff, ruff-format, pytest 1692 passed, copyright) were run manually via `pre-commit run --files`; the repo's .git/hooks/pre-commit points at a venv without pre_commit installed, hence --no-verify. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cy KV budget max_tokens_all_users defaults to max_context, so the token budget shared across all users equalled ONE full-context request. get_benchmark_max_concurrency then correctly derives a concurrency of 1 for a full-context request, and the long-context sweep points were silently benchmarked far below the concurrency they registered: ISL 8192 / OSL 1024: 32 requested -> 28 measured ISL 32768 / OSL 1024: 32 requested -> 7 measured ISL 131072 / OSL 1024: 32 requested -> 1 measured That is neither what the RFP asks bidders to build -- max context AT max concurrency -- nor what the rubric intends to grade, and it also fragmented the TTFT-scaling-exponent fit across concurrency buckets until it had too few points per bucket to fit at all. Setting the budget to max_context * max_concurrency (262144 * 32) restores concurrency 32 at every ISL, verified end to end against llm-d-inference-sim: the two clamped points now run as bench_32768_1024_32_n32 and bench_131072_1024_32_n32. This is a bring-your-own-server endpoint, which like SUPER_CLUSTER is not bound by a single device's KV pool. SUPER_CLUSTER already gets context * concurrency in _infer_data; extending that branch to cover remote GPU endpoints would be the more general fix and would make this override unnecessary. Hooks run manually via `pre-commit run --files` (pytest 1692 passed); the repo's .git/hooks/pre-commit points at a venv without pre_commit installed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… throughput
The `tput` perf target is defined as decode interactivity x concurrency -- the
median per-request decode rate (1000 / TPOT), summed over the concurrent users.
The three LLM parsers instead fed it the tools' own output_token_throughput
(vllm: output_throughput), which is total output tokens divided by the whole
benchmark duration and therefore charges decode for time spent in prefill and in
queueing.
The two quantities agree only when prefill is negligible, which is why this went
unnoticed: they diverge as input length grows, and the metric degrades even
though the system's decode rate is unchanged. On a simulated corpus generated to
exactly meet target at concurrency 32:
ISL tput_user wall-clock decode agg
ratio ratio ratio
128 1.046 1.042 1.046
1024 1.046 1.038 1.046
8192 1.045 1.002 1.045
32768 1.044 0.889 FAIL 1.044
131072 1.049 0.456 FAIL 1.049
Both failures were artefacts: at ISL 131072 the machine spends roughly three
times as long prefilling as decoding, so a wall-clock output-token rate cannot
approach 35 t/s/u x 32 no matter how fast decode is. The target was not missed,
it was measured against the wrong quantity.
tps_decode_throughput now carries the decode aggregate, matching both its own
documented description ("across all users") and how
test_module/stress_tests/stress_tests_record_builder already derives the same
field (mean_tps * actual_max_con). The tools' wall-clock figure is still reported
as the new tps_output_throughput, labelled as such and not graded, so nothing is
lost. The helper returns None rather than 0 when a rate or concurrency is
missing, since a zero would score as a perfect result.
Hooks run manually via `pre-commit run --files` (pytest 1699 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not skew TTFT AIPerf generates 100 unique dataset entries by default (--num-dataset-entries) and reuses them once --request-count exceeds that. A 128-request sweep point therefore sends 28 duplicate prompts, which the server answers from its prefix cache with a TTFT of roughly the dispatch overhead. The damage is that this moves the mean and not the median, so it does not look like an artefact. Measured at ISL 8192, concurrency 32, 128 requests against llm-d-inference-sim configured to exactly meet target: entries=100 (default): mean 1021.7 ms median 1278.6 ms 28 cache hits entries=128 (fixed): mean 1294.2 ms median 1293.5 ms 0 cache hits The reported mean TTFT was 21% below the truth, and every graded mid-context point on the loaded corner read 25-31% *faster* than a target the system was in fact only just meeting. With the fix, mean and median agree and the residual gap to target is the simulator's dummy tokenizer under-counting prompt tokens by ~11%, which is a known simulator limitation rather than a measurement error. Not simulator-specific: prefix caching is enabled by default in vLLM, so any sweep point with num_prompts > 100 understated TTFT on real hardware too. The concurrency-1 points (num_prompts 8) were never affected, which is why this surfaced only once long-context points began running at their registered concurrency of 32. Hooks run manually via `pre-commit run --files` (pytest 1699 passed); the repo's .git/hooks/pre-commit points at a venv without pre_commit installed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ilures The acceptance gate blocked every graded point with "No error_request_count reported, so zero failed requests cannot be confirmed", so no corpus could pass Part I regardless of its latency numbers. The gate is right to block on absence -- RFP G.2.6 requires zero failures to be *confirmed*, and treating silence as a pass is how an unverified requirement becomes a satisfied one. The bug is that AIPerf's silence is not silence: it omits error_request_count from the export entirely when nothing failed, and only emits it (with error_isl / total_error_isl) once at least one request has failed. The parser read that absence as "nobody looked", inverting the meaning of the healthiest output AIPerf can produce. error_summary is the right signal and AIPerf always writes it: an empty list on a clean run, one entry per distinct failure otherwise. That is an affirmative "the tool looked", which is exactly what the gate needs in order to tell a confirmed zero from silence. Verified against aiperf 0.5.0 using the simulator's --failure-injection-rate: rate=0: error_summary: [] error_request_count absent, request_count 8 rate=50: error_summary: [4 entries] error_request_count 5, request_count 3 Fixed in the parser rather than the gate: the parser is the layer that knows one tool's export dialect, so the gate stays tool-agnostic and its guarantee is untouched. An export with neither key still yields None and still blocks. The summary is summed rather than merely tested for emptiness, so a partial export cannot report zero failures while listing some. Checked against all 20 real corpus exports (each now 0, previously None) and the injected-failure export (5). Hooks run manually via `pre-commit run --files` (pytest 1703 passed); the repo's .git/hooks/pre-commit points at a venv without pre_commit installed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benchmarks summary could report a negative number of passing blocks -- a real
20-point sweep printed "-2/20 passed, 10 failed, 2 waived, 10 NA", which is the
first number a reader sees.
`waived` is an overlay on the other counts, not a fourth exclusive bucket. A block
lands in it when a tier check failed without blocking, and that happens in two
ways: on its own, where the block still passed the gate and was never in a
non-pass bucket; or alongside a real blocker, where the block is already counted
in `failed`. That overlap is deliberate --
test_benchmark_failing_complete_tier_still_blocks_at_complete_status asserts it,
so a masked tier is surfaced rather than silently dropped just because the block
failed elsewhere. Subtracting len(waived) therefore removed passing blocks that
were never excluded, and removed failed blocks twice.
One block that both failed and carried an informational note was enough to
reproduce it (passed == -1). Now passed/failed/skipped/na partition total, so the
summary line reads as a partition, and waived is worded as an annotation ("N with
waived check(s)") rather than listed as though it were a further share.
Note for anyone using a corpus as a scoring oracle: waivers exist here because
gemma-4-31B-it is status EXPERIMENTAL, which enforces zero target tiers by design
-- required_target_tiers is empty, so "a forge/new bring-up model can fail every
performance benchmark and still be released". A block failing functional,
complete AND target is still accepted at that status. Only TOP_PERF enforces the
target tier. The waivers are correct; only the arithmetic was wrong.
Hooks run manually via `pre-commit run --files` (pytest 1708 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Milestone-0 scoring chain had a manual gap: nothing built the document
report_module.scorecard consumes. GradedPoint was constructed only in scorecard.py
and its own tests, so a submission was hand-transcribed -- roughly ten points times
eleven fields -- which is the one step of an auditable ranking process with no
audit trail.
python -m report_module.submission --partner "Acme" --run workflow_logs
Runs after the workflows, reads their reports, writes the submission. Not a change
to any workflow.
Derived, because the mapping is exact: each graded point's targets (taken from its
own target_checks["target"], so they travel with the measurement they grade),
the measured percentiles and throughputs, the fitted scaling exponent per
concurrency level, and the two bonus figures when a prefix-cache report is present
(read via prefix_cache_uplift.SCORED_FIELDS so the two cannot drift apart).
Supplied on the command line: eval margins, contribution quality, assistance units
and the reproduction outcome. Deliberately not guessed -- how a set of per-task
eval ratios becomes one margin is a scoring policy expressed nowhere in this
repository, and inventing an aggregation would invent published terms. The per-task
ratios found in the reports go to _evidence so whoever sets the margin can see what
they are setting it from.
run_to_run_cov is computed from repeat runs under a definition this module chooses
rather than one the rubric publishes, so the definition is stated in the output and
--run-to-run-cov overrides it.
Two properties the tests pin, both about not inventing data. A point measured once
yields no CoV rather than 0.0, since zero variation would score full marks for
having done nothing, and splitting a sweep across runs must not manufacture a
stable-looking score. Points merge across runs by (concurrency, input length) with
the earliest run winning, so a Partner may run the two concurrency corners
separately while scored figures still come from one consistent run.
Validation happens here, where the message can name the cause: a sweep that is not
exactly two concurrency corners is rejected with a pointer to
ONLY_BENCHMARK_TARGETS=1, rather than surfacing later as a weighting error from
inside point_weights().
Verified against the two simulated corpora: assembles 10 points at concurrency
1 and 32 and scores a core total of 32.92, identical to an independently
hand-built submission for the same data.
Hooks run manually via `pre-commit run --files` (pytest 1726 passed); the repo's
.git/hooks/pre-commit points at a venv without pre_commit installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…enstorrent#4910) * Add DeepSeek-V4-Flash-0731 GPU reference eval config and thresholds Add a GPU bring-your-own-server dev spec for deepseek-ai/DeepSeek-V4-Flash-0731 and an EvalConfig (r1_gpqa_diamond, terminal_bench_2_1, swe_bench_verified) with GPU reference scores measured with thinking enabled at reasoning_effort=max (GPQA 90.40, Terminal-Bench 2.1 74.16, SWE-Bench Verified 87.20). Co-authored-by: Cursor <cursoragent@cursor.com> * Report DeepSeek-V4-Flash author scores at max reasoning effort Align published_score for DeepSeek-V4-Flash-0731 with the author's max-effort numbers, since the GPU references are collected at reasoning_effort=max: GPQA-Diamond 87.4->88.1, Terminal-Bench 2.1 56.6->82.7 (repointed to the -0731 card, which reports code-agent tasks at max effort), SWE-bench Verified 78.6->79.0. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
tenstorrent#4909) * Add DeepSeek-V4-Flash-0731 GPU reference eval config and thresholds Add a GPU bring-your-own-server dev spec for deepseek-ai/DeepSeek-V4-Flash-0731 and an EvalConfig (r1_gpqa_diamond, terminal_bench_2_1, swe_bench_verified) with GPU reference scores measured with thinking enabled at reasoning_effort=max (GPQA 90.40, Terminal-Bench 2.1 74.16, SWE-Bench Verified 87.20). Co-authored-by: Cursor <cursoragent@cursor.com> * Report DeepSeek-V4-Flash author scores at max reasoning effort Align published_score for DeepSeek-V4-Flash-0731 with the author's max-effort numbers, since the GPU references are collected at reasoning_effort=max: GPQA-Diamond 87.4->88.1, Terminal-Bench 2.1 56.6->82.7 (repointed to the -0731 card, which reports code-agent tasks at max effort), SWE-bench Verified 78.6->79.0. Co-authored-by: Cursor <cursoragent@cursor.com> * Add Mistral-Small-4-119B-2603 GPU reference eval config and thresholds Add a GPU bring-your-own-vLLM dev spec and eval config for Mistral-Small-4-119B-2603 covering GPQA-Diamond, AIME 2025, and Terminal-Bench 2.1. Set gpu_reference_score thresholds from the reference run (reasoning_effort=high, temperature=0.7, top_p=0.95): GPQA 77.27 (153/198), AIME25 86.67 (26/30), Terminal-Bench 2.1 32.58 (29/89). For terminus-2, reasoning_effort is a top-level agent argument that the agent forwards into LiteLLM itself, so it must stay out of llm_kwargs to avoid a duplicate-keyword TypeError. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* forge SDXL: recalibrate N150 spec-test thresholds for the Forge path (tenstorrent#4575) The forge SDXL N150 nightly reaches spec_tests now that the request timeout fix lets the job complete. Evals + benchmarks pass, but three spec_tests fail because their targets were tuned for the faster all-on-device trace path, not the Forge UNet-only path (~11-15s/img): - ImageGenerationLoadTest 20-step: 11.1s vs 10s target - ImageGenerationLoadTest 30-step: 14.9s vs 14s target - ImageGenerationParamTest diff-params SSIM: 0.9813 vs <0.98 bound Add sdxl+n150 model_targets overrides (20-step 10->12s, 30-step 14->16s, diff_params_max_ssim 0.98->0.985), scoped to n150 so t3k and galaxy targets are unchanged. Update the matrix-expansion unit test and add a t3k regression guard so the two paths can't silently drift. Clears 3/5 spec_test failures; the remaining 2 (LoRA eval + LoRA load) need LoRA support in SDXLForgeRunner, tracked separately. Refs tenstorrent#4352 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * comment out (tenstorrent#4886) * uplift TLE (tenstorrent#4860) * read scheduler diagnostic (tenstorrent#4880) * read diagnostic * fix build + format * Add BLACKHOLE_GALAXY mapping to utils device_utils (tenstorrent#4889) * Determine BLACKHOLE_GALAXY device config for Milestone-0 (readiness 5.4) Settle the 32-chip Blackhole Galaxy (4x8 2D torus) device-level configuration that blocks the M0 sweep/target work, and add the scaffolding it produces: - Add BLACKHOLE_GALAXY device specs to the dev catalog for gemma-4-31B-it (real tt_transformers config, full 256K context) plus partner-fillable stubs for deepseek-ai/DeepSeek-V4-Flash-0731 (1M native context, new deepseek_v4_flash stub impl) and mistralai/Mistral-Small-4-119B-2603 (256K context). All pin the fixed torus mesh graph descriptor, fabric_config FABRIC_1D_RING, and per-system max_concurrency 128; the logical mesh (MESH_DEVICE) is left to the operator. - Express and enforce targets per system for BLACKHOLE_GALAXY so a data-parallel choice cannot move the bar: DeviceTypes.expresses_targets_per_system() and get_perf_reference() now resolve from the device key and skip DP scaling. Wormhole Galaxy DP specs keep the per-replica behaviour (regression-guarded). - Add tests/test_perf_reference_per_system.py and the determination record under docs/rfp/. Ref: llm-gauntlet#68 (parent tenstorrent#56) Co-authored-by: Cursor <cursoragent@cursor.com> * Enforce scaling-quality three-point sweep coverage (readiness 5.7) The scaling-quality rubric fits time-to-first-token vs. input length separately at each graded concurrency level, so every graded concurrency level must carry at least three distinct input lengths or its regression is meaningless (RFP Appendix B.1/B.2/F.1). The auto-generated sweep pairs each input length with concurrency [1, per-shape-allowed-max], and the allowed max shrinks as input length grows, so at the Milestone-0 Blackhole Galaxy config (max_context=262144, max_concurrency=128) with the token pool defaulted to max_context only ISLs 128 and 1024 reach concurrency 128 -- two points, the exact failure this issue guards against. - workflow_types: DeviceTypes.grades_scaling_quality() -> True for BLACKHOLE_GALAXY. - benchmark_config: reusable validator scaling_quality_coverage_violations plus feasibility helpers input_lengths_reaching_concurrency, max_gradeable_concurrency (120 at the default pool), and min_token_pool_for_concurrency (278528 for concurrency 128). - benchmark_configs: get_llm_configs fails fast for scaling-quality-graded devices when the post-cap graded set violates the rule, with an actionable message. Vacuous for the current catalog; activates when blackhole_galaxy targets are authored (tenstorrent#66/tenstorrent#67). - dev/llm.yaml: each M0 BLACKHOLE_GALAXY spec carries a [TBD - Performance] max_tokens_all_users_override placeholder (floor 278528) to lift a third input length to the top graded concurrency. - tests + determination doc (docs/rfp/m0-scaling-quality-sweep-coverage.md), cross-linked with the 5.4 device-config doc. Issue: llm-gauntlet#64 (Blocker) - Parent: tenstorrent#56 Co-authored-by: Cursor <cursoragent@cursor.com> * Verify AIPerf benchmark path end-to-end on Blackhole Galaxy (readiness 7.1) Confirm the `--tools aiperf` path works end to end against a model "served" on a 32-chip Blackhole Galaxy, using a mock OpenAI-compatible server (llm-d-inference-sim) in place of real hardware. The harness exercises the real repo code path -- AIPerfDriver -> AIPerfParser -> apply_target_checks -> report_module -- so only the inference server and tokenizer are stand-ins. - scripts/verify/run_aiperf_bh_galaxy_e2e.sh: starts the seeded mock, builds the LLM_AIPERF venv, runs the harness, tears down. Every knob overridable via env. - scripts/verify/aiperf_bh_galaxy_e2e.py: drives the sweep, grades tiers, and renders the report/summary. Sweep grid (--isls/--concurrencies/--osl) and targets (--targets-json) are configurable. - scripts/verify/targets_deepseek_v4_flash.json: DeepSeek-V4-Flash tiered targets derived from the Appendix B sheet (interactivity 100/25 t/s/u; TTFT 1410/2820 ms from prefill throughput at the system operating point), derivation recorded inline. - docs/rfp/m0-aiperf-e2e-verification.md: determination doc -- approach, real vs mocked, both confirming runs (generic + DeepSeek targets, 6/6 PASS), and the remaining real-silicon gap. Grades on the two concurrency-independent per-request bars (TTFT + interactivity); aggregate/prefill throughput kept as documented system context so the bar does not move with the sweep point's concurrency. Readiness item 7.1 (llm-gauntlet#72). RFP requirements G.1. Co-authored-by: Cursor <cursoragent@cursor.com> * Author DeepSeek-V4-Flash-0731 blackhole_galaxy targets from Appendix B sheet (tenstorrent#66/tenstorrent#67) Seed the per-system Blackhole Galaxy performance targets for deepseek-ai/DeepSeek-V4-Flash-0731 from the Milestone-0 Appendix B AIPerf sheet. - model_performance_reference.json: new "DeepSeek-V4-Flash-0731" -> "blackhole_galaxy" entry with three graded points at the sheet's operating concurrency (64), bracketing the 8,192 mean ISL: ISLs {4096, 8192, 16384}, OSL 128. Each stores a single theoretical target -- interactivity tput_user=100 t/s/u (per-user, sheet target), aggregate decode tput=6413 t/s (sheet t/s/BH-GLX, per-system so used verbatim), and ttft from a linear-prefill model anchored at the sheet's 8192->1410ms point (per-request prefill rate 371,835/64 = 5,810 tok/s): 705/1410/2820. - dev/llm.yaml: set the DeepSeek BLACKHOLE_GALAXY spec max_concurrency to 64 (the sheet's max-num-seqs @ max context; was a placeholder 128) and enable max_tokens_all_users_override=1056768 (=64*(16384+128)) so all three graded ISLs reach concurrency 64 -- otherwise the pool defaults to max_context and ISL 16384 demotes below 64, breaking the scaling-quality three-point rule (readiness 5.7, enforced by get_llm_configs). - docs/rfp/m0-deepseek-v4-flash-targets.md: determination doc with the full derivation and open decisions for tenstorrent#66/tenstorrent#67 + Partner (TTFT off the anchor is modeled not measured; single graded concurrency level; repo-default tiering vs the sheet's per-metric downrates; dev placeholders pending the contributed impl). Validated: get_perf_reference resolves the 3 points per-system (unscaled), get_llm_configs builds the sweep with graded set {(4096,64),(8192,64),(16384,64)} and the three-point guard passes; perf/scaling-quality/model-spec/benchmark-config suites green (105 passed). Builds on tenstorrent#68 (device config), tenstorrent#64 (scaling-quality rule), tenstorrent#72 (AIPerf E2E). Co-authored-by: Cursor <cursoragent@cursor.com> * Clean up RFP branch: drop llm-gauntlet references, revert accidental tt-media-server changes - docs/rfp/*, scripts/verify/*, dev/llm.yaml: remove all references to the llm-gauntlet repo (repo name, issue URLs, and bare #NN issue numbers that would mis-link on tt-inference-server). Determinations now reference the Milestone-0 readiness item / RFP section numbers instead. - tt-media-server/: revert changes that were pulled into this branch by accident (blaze_runner scheduler-diagnostics additions and the tt-llm-engine submodule pointer bump) back to the tt-rfp-m0 base. - workflows/model_spec.py: make the DeepSeek-V4-Flash stub impl code_path an obvious "<FILL IN>" placeholder for the Partner instead of a real-looking path. Dev catalog still expands the DeepSeek spec (code_path="<FILL IN>", 3 perf-reference points, max_concurrency=64); perf/scaling-quality/model-spec suites green. Co-authored-by: Cursor <cursoragent@cursor.com> * Update workflows/model_specs/dev/llm.yaml set this value to accommodate max_concurrency=32 full-context requests Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com> * Update workflows/model_specs/dev/llm.yaml from target sheet Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com> * Update workflows/model_specs/dev/llm.yaml from target sheet to be sweeping all the way up to max-num-seqs + max-seq-len Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com> * Update workflows/model_specs/dev/llm.yaml Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com> --------- Co-authored-by: Logeshwaran Elanchelian <ctr-lelanchelian@ext.tenstorrent.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ljubica Muravljov <lmuravljov@tenstorrent.com> Co-authored-by: Kosta Novokmet <knovokmet@tenstorrent.com> Co-authored-by: Veljko Cankovic <vcankovic@tenstorrent.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Benjamin Goel <bgoel@tenstorrent.com>
Settles readiness §9.2 (llm-gauntlet tenstorrent#78) before the remaining sweep entries are authored (tenstorrent#66), so the same value is not written twice under two different meanings. It already happened once: PR tenstorrent#4911 stored the roofline sheet's figures directly and flagged the open question. The sheet's figures are peaks, and the per-metric downrates still applied, so those entries set the bar 2x (TTFT) and 4x (throughput) harder than intended. The convention, now written down in docs/rfp/m0-target-convention.md: `theoretical` holds the PUBLISHED target, which is peak x downrate (prefill 0.50, interactivity and decode 0.25). The peak never enters the file. Downrating at authoring time is forced rather than chosen — the two factors differ by metric and perf_targets_map carries one percentage per tier applied to every metric, so it cannot express them. DeepSeek-V4-Flash-0731 re-derived accordingly: ttft 1410/2820/5640, tput_user 25, tput 1603. Cross-check that catches exactly this class of error next time — tput_user x concurrency should approximate tput (25 x 64 = 1600 ~ 1603). Three defects found while making the gate work, each of which fails silently and in the safe-looking direction: All three Milestone-0 models sat at status EXPERIMENTAL, whose required_target_tiers is empty. Because evals_enforced reuses that same signal, accuracy failures were waived too — a submission could fail every performance target AND every eval and still be accepted. The issue describes only the performance half. Now FUNCTIONAL, with one tier named `functional` at 1.0 holding the published value. The tier is named for the status that enforces it: required_target_tiers matches on NAME, so a tier called `target` under FUNCTIONAL would be computed, reported and then ignored. TOP_PERF is the only status that enforces `target`, and it asserts near-peak optimisation, which is not a claim Milestone-0 makes about a first-of-kind bring-up. _verdict() read the `target` tier by hardcoded name. With any perf_targets_map that does not define that name it returned NA, and an NA point is counted ungradable rather than failed — so a point missing its target by any margin was accepted. It now grades the strictest tier actually present, which is unchanged for every model on the default ladder. DeviceModelSpec.perf_targets_map existed, was carried through expand_to_specs and was never read when deriving tiers; the model-wide map applied to every device. Setting it did nothing, silently, despite readiness §9.2 describing it as the supported override. It is needed here because gemma-4-31B-it's template also covers P300X2 and GPU, which must keep the ordinary ladder. tolerance now reaches the check. It defaulted to 0.0 and was never populated from the reference file, so the gate demanded beating target outright while G.2.4 published an allowance. Read per point beside `targets` — a policy knob, not a measured metric — defaulting to 0.0 so no existing model changes. Milestone-0 publishes 0.10. Verified end to end on the DeepSeek mid point (target 2820 ms): 20% over blocks and names the point, 5% over passes on tolerance, 10% under passes, and the same 20%-over point is waived at EXPERIMENTAL — isolating the status as the cause. 22 new tests, including a guard that each M0 spec's tier names are a subset of its status's enforced tiers, read straight from the committed YAML. Full suite: 1766 passed, 10 skipped. Refs tenstorrent#78 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_module/submission.py, added in tenstorrent#4911 and now the documented submission path in RFP I.1, read target_checks["target"] by name. Under the Milestone-0 single-tier configuration (tenstorrent#78) that tier is called `functional`, so the lookup found nothing, every graded point was silently skipped, and the assembler would have produced a submission with no points at all — scoring zero across the board with no error. Same defect as the one just fixed in _verdict, in a second place, which is the argument for not having the rule written twice. Adds llm_module.target_checks.graded_tier as the single definition of "the tier a report is graded against": the strictest tier actually present, which is unchanged for every model on the default ladder. submission.py now calls it, and its docstring no longer names a tier. Verified both ways round: the assembler finds the targets whether the tier is named `functional` or `target`, and with the full ladder present graded_tier still selects `target`. 3 new tests. Full suite: 1769 passed, 10 skipped. Refs tenstorrent#78 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…h corners Extends the blackhole_galaxy sweep from 3 points to 22: eleven input lengths at both concurrency corners. Powers of two 1K..512K, then a context-saturating point. The three existing entries are preserved exactly — 4K/8K/16K at concurrency 64 still read 1410/2820/5640 ms. The KV pool was the blocker, and the fix was already written down. The value was 1,056,768 (= 64 x 16,512), sized only for the three ISLs then graded, while its own inline comment read `64 * 1024 * 1024`. The comment held the intent: 64 concurrent requests at the full 1M context, 67,108,864 tokens. Under the old value every input length above 16K demoted — 32K->32, 64K->16, 128K->8, 256K->4, 512K->2 — which breaks the three-input-lengths-per-graded-level rule (tenstorrent#64) and leaves seven concurrency levels where Appendix B.5's per-point weights are defined over exactly two. At 67,108,864 every input length reaches 64, and the longest does so exactly: 67,108,864 / (1,048,448 + 128) = 64. A full 2^20 input cannot be swept. 1,048,576 + 128 exceeds max_context, and get_benchmark_max_concurrency answers that by silently returning concurrency 1 rather than rejecting the point — so the mistake would have surfaced as a stray concurrency level, not an error. The top input is max_context - osl = 1,048,448, which saturates the window exactly. Targets follow the convention: linear prefill over the downrated rate (371,835 x 0.50 = 185,917.5 tok/s aggregate), shared by the requests in flight, so the loaded corner is exactly 64x the idle one at every input length. Two corrections to the existing values: `tput` is now 25 x concurrency (25 idle, 1,600 loaded) rather than the sheet's 1,603. Measured aggregate decode is *defined* as tput_user x concurrency (llm_module.parsers.base.decode_throughput), so a target derived any other way means a system hitting interactivity exactly still misses the bar. The 3-token gap was the sheet's independent rounding. Values are stored to 4 decimal places, not 1. At 1 dp the idle target at ISL 1,024 rounds to 5.5 and x64 gives 352.0 against the loaded corner's 352.5 — the convention promises a Partner can recompute any published value, and that promise fails at 1 dp. Verified with ONLY_BENCHMARK_TARGETS=1 (the setting RFP I.1 mandates): exactly 22 points, all graded, two concurrency levels, eleven distinct input lengths each, scorecard weights summing to 1.0000 with the loaded corner at 0.75 and the heaviest point being the longest input under load. Rough sweep runtime ~35 min. The JSON was patched by brace-matching the DeepSeek object rather than round-tripping the file, because that file has duplicate keys (tenstorrent#101) and a load/dump silently drops 217 lines. 7 new tests covering the sweep shape: both corners at every length, no point exceeding the context window, targets linear in input length at each corner, the loaded corner exactly 64x the idle one, and tput = tput_user x concurrency. Full suite: 1774 passed, 10 skipped. Refs tenstorrent#66
Eighteen graded points: nine input lengths (1K..128K by powers of two, then a context-saturating 262,016) at both concurrency corners, 1 and 32. Targets are 123,072 tok/s aggregate prefill and 35.0 t/s/u interactivity, both supplied already downrated, so they are stored verbatim — unlike the DeepSeek sheet row, which gives peaks and separate downrate factors. TTFT is the linear prefill model over that rate divided by the requests in flight, making the loaded corner exactly 32x the idle one at every input length. `tput` is 35 x concurrency because that is how the measurement is defined. Note for anyone deriving from earlier work: an aggregate prefill figure of 184,608 tok/s circulated and is 1.5x the real target. The llm-gauntlet simulator profile still carries it and must be corrected before any calibration is taken from its output. Filed under `gemma-4-31B-it`, upper-case B, which is what the Milestone-0 tt_transformers spec derives. The pre-existing lower-case `gemma-4-31b-it` is the FORGE spec's key — it derives that spelling to match tt-media-server's ModelNames constant and owns the p300x2 entry. Both keys now exist and neither is renamed; renaming either would move targets off the spec that reads them, which is why tenstorrent#4884 only added a warning. The case-mismatch warning for this spec is now gone because the key it wants exists. Two spec fixes, both latent until this point: max_tokens_all_users_override read `32 * 256 * 1024` unquoted. YAML does not evaluate arithmetic, so the value loaded as the STRING '32 * 256 * 1024' and get_benchmark_max_concurrency raised TypeError on `str // int`. Nothing caught it because the model had no blackhole_galaxy targets, so the sweep never reached that code path — adding targets would have broken it immediately. The comment's arithmetic was right; only the literal was wrong. Now 8388608, which is also exactly what the sweep needs: 8,388,608 / (262,016 + 128) = 32. max_concurrency 128 -> 32, the expected max-num-seqs at full context and the concurrency the targets are authored at. 128 was a placeholder. A full 2^18 input cannot be swept: 262,144 + 128 exceeds max_context, and get_benchmark_max_concurrency responds by silently returning concurrency 1 rather than rejecting the point, so the mistake would surface as a stray concurrency level. The top input is max_context - osl. Verified with ONLY_BENCHMARK_TARGETS=1: exactly 18 points, all graded, two concurrency levels with nine distinct input lengths each, scorecard weights summing to 1.0000 with the loaded corner at 0.75, longest point fitting the context window exactly. Rough runtime ~31 min. The sweep-shape tests are now parametrised over both Milestone-0 models, plus one asserting the two gemma keys stay distinct. Full suite: 1780 passed, 10 skipped. Refs tenstorrent#66
The RFP now covers two models: gemma-4-31B-it and DeepSeek-V4-Flash-0731. Mistral-Small-4-119B-2603 keeps its dev-catalog scaffold — it is real work and the model may return in a later milestone — but no longer carries any Milestone-0 grading configuration: `perf_targets_map` removed, `status` back to EXPERIMENTAL, and it never had blackhole_galaxy targets. A test asserts that, because `status: FUNCTIONAL` plus a perf_targets_map is the signature of a graded model and the obvious mistake is to add it back by making the three specs look alike. The header comment says so too, at the point someone would edit. Also removes the stale blackhole_galaxy entry for Mistral-Small-3.1-24B-Instruct-2503. That is the placeholder tenstorrent#66's problem statement calls out — three values copied verbatim from gemma-4-31b-it's p300x2 row — and it was not merely misleading: since tenstorrent#4911 added the scaling-quality coverage guard it made `get_llm_configs` RAISE for that spec on BLACKHOLE_GALAXY, because its single graded point gives concurrency 1 only one input length. Pre-existing, confirmed by stashing today's work. Removing the entry leaves the device row intact and its points ungraded, which is what an unmeasured model should look like. blackhole_galaxy targets in the reference file are now exactly the two RFP models. Verified every BLACKHOLE_GALAXY spec builds its sweep without raising, and only those two are graded. Full suite: 1779 passed, 10 skipped. Refs tenstorrent#66
test_an_entry_without_a_tolerance_still_defaults_to_zero read Mistral-Small-3.1-24B's blackhole_galaxy entry, which the previous commit removed as a stale placeholder. Broken by that commit and pushed with it — my verification chain piped pytest into tail, so the shell saw tail's exit code rather than pytest's. Rewritten to assert the property across every non-Milestone-0 model in the reference file instead of one hand-picked entry: only the two RFP models opt in to a tolerance, and every other targeted point must still default to 0.0. Now covers ~600 points rather than 1, and cannot be broken by removing any single entry. Full suite: 1779 passed, 10 skipped (verified on pytest's own exit code).
The multipliers convert the mean TTFT target into a qualifying value for each graded percentile, and they decide 38 of the 100 core points. They were plausible round numbers — 0.90 / 1.30 / 1.80 — chosen before any distribution had been looked at. Measured on the gpt-oss-120b release run, 16 operating points on real silicon: p50 / mean median 1.000 range 0.795-1.150 published multiplier 0.90 p99 / mean median 1.192 range 1.000-1.852 published multiplier 1.80 Both were wrong, in opposite directions. At 1.80, a submission that merely met its mean target with an ordinary tail scored 14.9 of the 22 points on the rubric's heaviest line, free. At 0.90 the median line asked p50 to sit 10 % BELOW the mean, which essentially never happens, so those 5 points were unreachable by anyone. Tail discipline at 4.0 saturated outright: real ratios are ~1.19, so every system took all 4 points. Together that is 21.9 of 42 tail-sensitive points handed to a submission for meeting the target and nothing more — the opposite of what K.2 requires, which is that meeting the target scores zero. Now p50 1.00 and tail discipline 1.25 -> 1.05, both at the measured values. p99 is set at 1.35 rather than the observed 1.19, deliberately: tails widen under load and on a first-of-kind bring-up, and a qualifying value at the median observation would put 22 points out of reach of a system with an ordinary tail, which is a worse failure than a generous one. p90 was not captured in that run — it predates the percentile work in tenstorrent#57 — so 1.12 is interpolated and is the least certain. Appendix F.2 and F.3 are recomputed, including the example's measured p99 column: those values were chosen to give a spread against 1.80, and against 1.35 three of the ten points clamped to zero, which teaches nothing. Re-chosen to model a system with a tight tail that widens under load — the shape the line exists to reward. The p99 line now returns 8.72 of 22 and the core total 50.56. Two tests were tuned to the old example rather than to the invariant, and are now robust to it: the summed-before-rounding test asserts an exact identity instead of a divergence that only appeared with particular numbers, and the prefill-dominance test asserts prefill is the largest group rather than clearing a fixed share. Full suite: 1779 passed, 10 skipped. Refs tenstorrent#118
The RFP previously said Milestone-0 "adopts unchanged" the benchmarking tool's own service-level thresholds. That makes a constant in llm_module a published requirement by reference: editing it would move the bar a Partner is graded against, silently and with no review of the RFP. Appendix B.3 now states the values as Milestone-0's own — the same numbers today — and this pins the tool's constants to them, so a change to either is a test failure naming the other. The Appendix governs; the tool is checked against it, not the reverse. Refs tenstorrent#74
…03 / 1.02 They were 1.15, which asked a Partner to be 15 % more accurate than the reference running the same weights. Accuracy is a property of the model, not the silicon: a correct implementation scores about equal to the reference, so 1.15 was unreachable by engineering and 22 of the 100 core points were dead weight — the same defect as the p99 multiplier, in a different place. The achievement actually being rewarded is holding reference accuracy on a different compute class at a different numerical precision, and then edging past it. 1.03 for agentic and 1.02 for standard put the line in the band submissions will really occupy. Agentic is the higher of the two because its task sets are smaller — 89 tasks against 198 samples — so an above-parity result needs more headroom before it means anything. Both remain close to the noise floor of their task sets, which is stated in the RFP rather than hidden: these lines separate a submission that holds reference accuracy from one that does not, and are not fine instruments above that. F.2's example values move from 1.09 and 1.04 to 1.02 and 1.01 — against the new range the old ones clamped to full marks, which demonstrates nothing. Core total 53.69, overall 67.56. One test hardcoded the forgone score in the failed-reproduction note. It now derives it from the baseline: the point is that the cost is stated, not that it is any particular number. Full suite: 1784 passed, 10 skipped. Refs tenstorrent#118
Accuracy leaves Part K entirely. It remains a hard pass/fail gate in Part I via requirements H.5 — unchanged and still blocking — but it no longer earns rubric points. The reason is that rewarding accuracy above the reference invites Partners to chase fractions of a point on a quantity that is a property of the model's weights rather than of their implementation. A correct implementation scores about equal to the reference; anything above that is mostly sampling noise on 89-task and 198-sample sets. Points that reward noise are worse than no points. Setting the excellence values to 1.00 would have expressed the same intent but left 22 weighted points unearnable by construction: qualifying == excellence makes the fraction 0/0, so the best achievable core score would silently have been 78 against a stated 100, and every scorecard would have reported two lines as unscoreable when they were in fact deliberately not scored. Removing them says it once, in the weights. Core is therefore **78**, not 100, and every point in it is winnable. The bonus stays 20. F.2 rolls up to core 40.69 and overall 54.56; prefill's share of core rises from 53 % to 66 %, which is the balance the rubric was already trying to express. report_module/submission.py loses --agentic-eval and --standard-eval, which now feed nothing. Per-task eval ratios are still written to _evidence, because whoever adjudicates the Part I gate should be able to see them without opening the run. Four tests referenced the eval lines only as convenient examples of a once-scored line and now use run_to_run_cov or contribution_quality instead. Two that asserted 100 and a fixed prefill share are updated to 78 and to comparisons that hold regardless of what the example scores. Full suite: 1781 passed, 10 skipped.
…m fork to vllm-tt-plugin + upstream vLLM (tenstorrent#4907) * initial changes for switching to vllm plugin instead of the fork * remove uv cache after building stage so we can shrink docker image
The vLLM platform plugin now lives in its own repository (tenstorrent#4907, cherry-picked in the previous commit), and validate_setup's failure message is where a Partner actually learns how to install it. It said `git clone` with no revision, which gets whatever the default branch holds today. That matters more than it looks: the plugin owns the vLLM version pin and its dependency overrides, so a different plugin revision is a different vLLM. The Milestone-0 baseline pins the plugin to a specific commit, and a Partner following this message verbatim would have installed a different one and had no way to know. The message now shows the checkout step and says why, while making clear it is skippable when nothing pins it.
added 27 commits
November 12, 2026 22:05
…projection Action-Id: ds41/optimize/101
V41_DECODE_STAGE_BATCH=1: the decode step stages its packed inputs and Engram rows with one ttnn.copy_host_to_device_tensors call (one wake-up of the per-device dispatch workers instead of three): served, the one-row step's staging 2.7 -> 2.2 ms, the period 59.0 -> 58.5, tput_user 17.1 -> 17.2 to 17.3 at points 1, 5 and 8. Action-Id: ds41/optimize/102
V41_DECODE_SDPA=flash: the one-live-row decode capture runs the flash-decode SDPA program (the live rows keys in units of 32 over the grid, a log-sum-exp merge per output tile, the partials through the merge cores L1). Right on every live-row pattern in both KV formats against the local path (PCC 0.9998), the c1 chain 102 -> 73 us a layer, the one-row step 49.2 -> 48.0 ms of device; served, tput_user 17.2 -> 17.6 at points 1, 5, 8 with the replay-state, production-order L1 and generator gates as before. Action-Id: ds41/optimize/113
TT_METAL_MESH_WRITE_INLINE=1: the decode step one batched host-to-device round runs its per-device writes on the calling thread (tt-metal mesh-write-inline) instead of as one task per device on the dispatch worker pool, whose wake-up after the ~47 ms device wait cost the round ~2 ms served. Bit for bit (the generator gate four texts); served, the staging 2.2 -> 0.7 ms a step, the period 58.5 -> 56.8 ms, tput_user 17.3 -> 17.8 at points 1, 5, 8. Action-Id: ds41/optimize/113
TT_V41_SPEC=3 (the verify decoder with the DSpark drafter in its capture) with the plugin's ragged contract (the speculative-config of vLLM's custom_class method naming vllm_tt_plugin.model_owned_spec, 3 tokens: K + 1 tokens a step through the scheduler's accepted counts) and TT_V41_SPEC_GREEDY_ALL=1 (every request verified greedily: the smoke's knob, since the benchmark samples at temperature 1 and the round has no sampler yet). Action-Id: ds41/optimize/154
The round's warm-up captures 16 vd: variants (a round per user index and draft count) and the plain one-slot step per user index beside the 32-slot step, ~0.8 GB of trace region, after which the catalog's per-length prefill captures overran the 2 GiB region of the plain configuration. 3 GiB, at a cost of 1 GB of the DRAM margin. Action-Id: ds41/optimize/154
The catalog's speculative entry drops TT_V41_SPEC_GREEDY_ALL: a sampling request's round draws the served sampler on every verify row and accepts while the draw equals the next draft, so the benchmark's temperature-1 requests are speculated with the same output law as plain serving instead of being verified greedily. Action-Id: ds41/optimize/154
The catalog sets TT_V41_SPEC_RINGS=slot: the DSpark drafter keeps a ring per decode slot, so every request speculates (one ring set let the last joined request speculate and ran the others plain, which at c32 was the plain step plus one eighth of a round). A step of one live slot is the fused c1 round; a step of several runs every slot of the pool as a block of 1 + K verify rows and one standalone draft call for the four slots of every mesh row. Action-Id: ds41/optimize/159
submission.merge_points and run_to_run_cov keyed a point on the report's measured input_sequence_length, which is AIPerf's mean prompt length and can differ from the registered length by a token. Repeat runs could become separate points and their variation could be computed over the wrong groups. registered_point reads the sweep identity from the target_checks block title; point_key falls back to measured fields when a block has no such title. comparison.collect_points uses the same identity. Tests cover token-apart repeats, merged variation, title parsing and fallback behavior. Action-Id: ds41/optimize/161
The four flags (TT_SCHED_DEFER_FINAL, TT_V41_DEFER_FINAL, TT_V41_RESTORE_TRACE, TT_V41_RESTORE_TRACE_MAX_WIDTH 131072) in the DeepSeek-V4.1-Flash entry's env_vars, so the served configuration runs every request's final prefill chunk after the wave's bulk prefill (c32 mean TPOT 0.76x at 8K and 0.12x at 128K of the serial path's on the plain tree), composed here with the speculative round. Action-Id: ds41/optimize/171
TT_SCHED_DEFER_FINAL "hold" in the served entry: the plugin admits no prefill while a cohort decodes, so a c32 point's replacements form the next cohort together instead of running their bulks between the stragglers' decode steps. Action-Id: ds41/optimize/174
TT_SCHED_FINAL_STEP=1 beside TT_SCHED_DEFER_FINAL=hold (vllm-tt-plugin): a multi-chunk request's bulk prefill runs to n - 1 tokens and is joined by the stock path, and the cohort's parked requests take their first token from one decode step instead of a serial final chunk each. The generator's parking and the restore captures (TT_V41_DEFER_FINAL, TT_V41_RESTORE_TRACE and its width cap) come out: nothing is parked or restored on this path. Action-Id: ds41/optimize/192
Under TT_SCHED_FINAL_STEP a stepped prompt reaches the generator as n - 1 tokens, so the sweep's prompts (the listed length and the one or two below it) need the final-chunk captures at +-2 of each listed length: with the default slack of 1, 6 of 128 finals at 64K and 1 at 16K ran eagerly (4 to 13 s each). TT_V41_TRACE_PREFILL_SLACK=2 in the catalog entry. Action-Id: ds41/optimize/194
TT_V41_PREFILL_HOST_BATCH=1: the generator's state reset and each chunk's staging issue their host-to-device copies as one ttnn.copy_host_to_device_tensors each instead of ~55 single copies a one-chunk call, bit for bit; the flag-on arm of the served pair at the c32 short points. Action-Id: ds41/optimize/208
TT_V41_CHUNK 10240 with max_num_batched_tokens to match, and 5120 added to TT_V41_SHORT_CHUNK so prompts of 2,049 to 5,120 tokens keep their one 5,120-row chunk. Measurements showed the larger chunk at -9 % of prefill device time per token at 16K and from 128K up; the per-chunk items halve per token and the padded-row items do not move. Action-Id: ds41/optimize/209
TT_V41_PREFILL_RESET_DEVICE 1 and TT_V41_PREFILL_INDEX_BOUND 1 beside the 10,240 chunk at the native 1,053,696 context: whether the index bound, by not building the candidate source's resident ratio-1 mask, lets the 1M score bucket's generic capture fit the chip's DRAM, which it did not without it. Action-Id: ds41/optimize/215
TT_DETOK_PRIME_TOKENS=16 in the catalog entry: the vllm-tt-plugin's general entry point replaces tokenizers' DecodeStream with one primed with the prompt's last 16 ids instead of the whole prompt, which vLLM 0.24.0's streaming detokenizer decodes twice at a request's first token (0.26 us an id; 32 requests' first tokens from one decode step paid it serially, 9.7 s at 1M, in the API server's output loop while the engine decoded on). The emitted text is unchanged. Action-Id: ds41/optimize/230
TT_V41_PACK=1 with TT_V41_PACK_LAYOUTS and the scheduler's TT_SCHED_PACK (the same list: 4x1028,3x1028,2x1028,2x2052): a prefill step admits the whole short prompts of a listed layout at the head of the queue and the generator runs them as one packed 5,120-row chunk, each prompt joining its decode slot from its own segment. The points it moves are the short c32 ones (12 and 13: four 1,028-token or two 2,052-token prompts a chunk); every other prompt takes the single path as before, and at c1 the flags are inert. Action-Id: ds41/optimize/233
The packed captures are the last of the warm-up, after the decode variants, the 352 joins, the per-length prefills and the 43 generic chunk captures, and the declared 3 GiB region had room for three of the four layouts (the fourth failed at its first 57 KB with 0 B free). A pack capture is ~37 MB and the four with their 160 segment joins ~185 MB; the region grows by 256 MiB a chip, which the 1M bucket's capture leaves free since the index bound. Action-Id: ds41/optimize/239
set_runtime_env_vars expands $VAR and ${VAR} with os.path.expandvars,
which leaves the shell's default-value form untouched, and a value the
spec sets always replaces the launch environment's. A spec that wants a
default the operator can override at launch has no way to say so. The
form ${VAR:-default} now takes VAR from the launch environment when it
is set and non-empty, else the default, before the existing expansion;
plain values keep replacing the environment. Two tests: the helper's
cases, and a spec value overridden by the launch environment while a
plain value is not.
Action-Id: ds41/optimize/244
The served packing arm is chosen by the launch environment: the four
flags that differ between the arms are written ${FLAG:-default}, so the
entry serves hold-all (one-chunk prompts held for the cohort and drained
as the listed packs, each first token delivered as produced) unless the
server starts with TT_SCHED_RELEASE_FIRST=cohort (+release),
TT_SCHED_DEFER_FINAL=hold (pack-only) or that and TT_V41_PACK=0
TT_SCHED_PACK=0 (off). The comment lists the four arms.
Action-Id: ds41/optimize/244
Keep the embedded serving template in step with the plugin renderer. Correct few-shot user merging and assistant transitions without changing other serving parameters or single-user benchmark prompt formatting. Action-Id: ds41/optimize/282
Add --eval-tasks independently of sample-limit controls for standard and agentic evaluations. Reject unknown, duplicate or limited task selections and preserve configured budgets, repeats and task objects. Carry selection through runtime serialization and dependency setup. Action-Id: ds41/optimize/283
Declare the V4.1 thinking and tool interface and pin the common eval harness and Harbor reference. Run full tasks in explicit order with independent coverage and accuracy verdicts, owned cleanup and resource bounds. Isolate agentic Docker storage and networking from shared state. Action-Id: ds41/main/optimize/284
Allow an explicit dotenv path so a full evaluation campaign cannot inherit another workflow secret or experimental setting. Keep the campaign credential file private and preserve reasoning in sample logs for extraction and token-budget diagnostics. Action-Id: ds41/main/optimize/284
Inspect inference HTTP failures as well as per-sample errors before calling a standard task operationally complete. Do not turn transport failures with a full-looking sample file into an accuracy verdict. Action-Id: ds41/main/optimize/284
Resolve frontend tokenizer files from the same immutable snapshot as the model and checkpoint encoder, rather than a mutable cached main reference during a long evaluation campaign. Action-Id: ds41/main/optimize/284
| if not 0 < args.max_hours <= 72 or not 0 < args.max_artifact_gib <= 100: | ||
| parser.error("Campaign bounds must not exceed 72 hours / 100 GiB") | ||
| args.output.mkdir(parents=True, exist_ok=False, mode=0o700) | ||
| os.chmod(args.output, 0o700) |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Permissive file assignment (Python)'.
Risk Score: 59 (MEDIUM)
Severity: High
Description
Permissive file assignment exposes sensitive information by granting unnecessary read, write, or execute permissions to users without ownership privileges.
Cycode Remediation Guideline
✅ Do
- Do keep file permissions as restrictive as possible to minimize the risk of unauthorized access. Use the principle of least privilege to only grant permissions that are absolutely necessary for the operation of the application.
os.chmod("my_private_file.txt", 0o600) # only you have full read and write access
- Do prefer assigning file permissions to 'groups' rather than 'other' when you need to extend privileges to users who are not the owners. This approach helps in limiting access to a more controlled set of users.
❌ Don't
- Do not set a permissive
umaskvalue, as this can lead to overly permissive default permissions for new files and directories
os.umask(0) # unsafe
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a project to upstream Tenstorrent support for Deepseek v4.1. @bgoelTT, as requested, this branches off of
tt-rfp-m0, so it will be out of date with main. As it stands now, there are 60 commits here but we believe we could whittle this down to ~13 with some more work. We publish these as-is now since they are the commits verified with the accuracy evals and benchmark sweep.Related: