Noisy benchmarks review

[IanButterworth]

Using https://perf.julialang.org/?tab=benchmarks&bv=noisy I asked claude to review the top offenders

cc. @vtjnash

---

Top run-to-run noise on the Nanosoldier x86_64 runner over the last ~6 months (Nov 2025 → May 2026), from julia-ci-timing. Noise = median |log(v[i]/v[i-1])| between consecutive runs, as %. Filtered to median latency ≥ 50 ns and noise ≥ 5%.

Group	Benchmark	Latest	Noise	Likely cause	Action
sparse	transpose/("adjoint", (20000, 20000))	89 ns	365%	Measures lazy Adjoint wrapper construction, not transpose	#344
sparse	transpose/("adjoint", (20000, 10000))	80 ns	120%	Same	#344
sparse	transpose/("transpose", (20000, 10000))	85 ns	49%	Same	#344
sparse	transpose/("transpose", (20000, 20000))	84 ns	15%	Same	#344
find	findprev/("Vector{Bool}", "50-50")	2.3 µs	194%	Bimodal 775/2330 ns; doesn't repro on aarch64; suspected x86 µop-cache alignment flip	File upstream (JuliaLang/julia)
sparse	index/("spmat", "row", "logical", 1000)	4.6 µs	14%	Sparse indexing path	Investigate
collection	deletion/("Set", "String", "filter")	27 µs	14%	String hash bucket order non-deterministic across processes	Use samerandstring
collection	deletion/("IdDict", "String", "filter!")	11 µs	12%	IdDict ordering depends on String pointer addresses	Use samerandstring or non-String keys
collection	deletion/("Set", "String", "filter!")	4.9 µs	11%	Same	Use samerandstring
collection	set operations/("Set", "Int", "union", "Set", "Set")	11 µs	13%	Set bucket order sensitive to layout	Audit setup determinism
collection	set operations/("Set", "Int", "union", "BitSet", "BitSet")	11 µs	12%	Same	Audit setup determinism
collection	set operations/("Set", "Int", "union", "BitSet")	11 µs	12%	Same	Audit setup determinism
collection	set operations/("Set", "Int", "union", "Vector")	11 µs	12%	Same	Audit setup determinism
collection	set operations/("Set", "Int", "union", "Vector", "Vector")	11 µs	10%	Same	Audit setup determinism
collection	set operations/("Set", "Int", "union", "Set")	11 µs	9%	Same	Audit setup determinism
union	array/("skipmissing", "perf_sumskipmissing", "Union{Nothing, Int64}", 0)	4.6 µs	12%	Union-typed getindex codegen	Investigate
array	index/("sumcolon", SubArray{Float32,2,…ReshapedArray…})	6.4 µs	11%	Likely codegen sensitivity for nested SubArray/ReshapedArray	Investigate
sort	length = 10/mixed eltype with by order	340 ns	11%	Tiny mixed-type sort; likely codegen	Probably yellow-flag only
array	index/("sumcartesian", ReinterpretArray{Int32,3,Float64,…})	815 ns	10%	Reinterpret indexing codegen	Investigate
array	index/("sumcartesian_view", ReinterpretArray{Int32,3,Float64,…})	818 ns	10%	Same	Investigate
linalg	arithmetic/("*", Bidiagonal, Vector, 256)	430 ns	9%	Sub-µs, near measurement floor	Yellow-flag only
linalg	arithmetic/("+", Diagonal, Diagonal, 256)	395 ns	9%	Same	Yellow-flag only
find	findall/("Vector{Bool}", "90-10")	860 ns	9%	Same family as findprev Vector{Bool}	File upstream
find	findall/("Vector{Bool}", "10-90")	439 ns	9%	Same	File upstream
union	array/("perf_binaryop", "*", "Int8", "(true, true)")	18 µs	8%	Union-typed codegen	Investigate
sparse	index/("spmat", "row", "array", 1000)	12 µs	8%	Sparse indexing	Investigate
linalg	blas/gemv!	240 µs	7%	Likely runner-side (cache/affinity), not BaseBenchmarks	Runner-side
sparse	matmul/("At_mul_Bt!", "dense 40x4000, sparse 40x40 → dense 4000x40")	3.2 ms	7%	Same	Runner-side

Suggested order of attack

1. Merge #344 — removes the 4 worst entries.
2. Audit setup blocks in CollectionBenchmarks.jl (and any String-keyed Dict/Set/IdDict) to route all randomness through RandUtils.samerand/samesprand/samerandstring. This addresses ~7 entries.
3. File a Julia issue for findprev/findall on Vector{Bool} documenting the bimodality with the dashboard link.
4. Investigate the union/SubArray/ReinterpretArray codegen-sensitive entries individually.