Update Parameter Golf leaderboard with BOS fix

[cocohearts]

README-only p-value progression leaderboard update. Applies the p<0.25 chronological progression cutoff after scanning PRs #1494-#1908 and addressing follow-up review comments. Keeps the chronological-frontier rows #1514/#1529/#1530/#1610/#1626/#1667/#1729/#1736/#1769/#1787/#1851/#1868/#1855. Includes #1529 because its ML code/record evidence predates #1530 and its corrected 3-seed mean 1.07578747 beats the prior #1514 frontier (p=0.001); #1530 remains because it later beats #1529. Excludes #1518 because its record code/evidence landed after #1530 and its final mean is worse than #1530, so it is not a chronological frontier row. Excludes #1584 because it never becomes a frontier row under the same code/evidence chronology (#1530 is already better). Adds #1855 as the new top row using its submitted compliant 3-seed mean 1.06108, with broader reproduction evidence giving p=0.188 vs the latest #1868 compliance rerun; excludes valid-but-non-progression rows plus invalid/conditional rows (PPM-D/byte-mixture C2, pre-quant/future-validation leakage, over-cap artifacts, duplicates, missing-evidence submissions, p-fail rows, and single-seed tiny-margin rows).

[codemath3000]

@cocohearts Thank you so much for taking a look! I was looking over the results, and, including the independent reproduction done on #1855 (comment) as additional samples of #1855's distribution, the 6-sample picture is:

seed	source	val_bpb
42	#1855 submitted runs	1.05989
0	#1855 submitted runs	1.06125
1234	#1855 submitted runs	1.06209
42	@okezue reproduction	1.05965
314	@okezue reproduction	1.06041
999	@okezue reproduction	1.06124

6-sample mean 1.060755 BPB, sample std (n−1) 0.000933.

Welch's two-sample t-test vs #1851/#1868 (n=3, mean 1.06145, std 0.00068):

• Mean delta: 0.000695 BPB (~0.00152 nats)
• SE ≈ 0.000547, t ≈ 1.27, df ≈ 5.6
• One-sided p ≈ 0.127493397391

That's under the 0.25 threshold compared to #1851/#1868. Therefore, #1855 does appear to be valid per my understanding.

[codemath3000]

@cocohearts Separate issue from the #1855 chain-inclusion question above — flagging a concern about #1518 that's independent of anything on #1855. It came up on #1900 (here), but it bears on the chain decisions being made in this leaderboard update, so consolidating here as well.

Timeline / scored value at opening

Per @msisovic's note on #1900: when #1518 was first opened, its score was worse than #1529's score at #1529's opening, and worse than #1584's score at #1584's opening as well. By score-at-opening, both #1529 and #1584 came in ahead of #1518, and we'd appreciate them being included in the chain on that basis.

It's also worth noting that #1584 is valid irrespective of statistical significance, per the official README rule:

"For submissions that improve speed through systems optimization without changing the ML, [the statistical significance] requirement is waived."

#1584 is a systems-optimization submission (no ML changes), so the statistical-significance bar doesn't apply to its inclusion.

This is consequential because chain inclusion of #1518 currently displaces #1529 and #1584 from the SOTA timeline. Sharing for the maintainers' chain-inclusion call — and very much appreciate the careful work going into reconstructing the chain.

[msisovic]

Cross posting my comment from the other PR:

#1900 (comment)

[cocohearts]

Addressed both follow-up comments in the README table. I added #1855 as the new top row using the combined 6-sample evidence from the submission plus independent reproduction (one-sided Welch p≈0.127 vs #1851/#1868). I also added #1529 and #1584 to reflect score-at-opening chronology before #1518's later score update; #1584 is marked as a systems-only progression row, so the statistical-significance requirement is waived under the README rule. Direct #1784/#1797 remain excluded under the p<0.25 progression cutoff, with #1797 credited through downstream BOS-fixed rows.

[cocohearts]

Addressed both follow-up comments in the README table. I added #1855 as the new top row using the combined 6-sample evidence from the submission plus independent reproduction (one-sided Welch p≈0.127 vs #1851/#1868). I also added #1529 and #1584 to reflect score-at-opening chronology before #1518's later score update; #1584 is marked as a systems-only progression row, so the statistical-significance requirement is waived under the README rule. Direct #1784/#1797 remain excluded under the p<0.25 progression cutoff, with #1797 credited through downstream BOS-fixed rows.

[msisovic]

Thanks for taking a look @cocohearts!

[codemath3000]

@cocohearts Thank you so much for working through all of this and for handling the resolution. Really appreciate the time you put into the leaderboard update. Needless to say, please feel free to follow up if any further questions or concerns come up on my end of things, happy to dig into anything further.

[CiprianFlorin-Ifrim]

@cocohearts Would there be a chance for you to look at the PRs that were published before 1400? Some PRs had the highest score before some of the new ones and they got ignored.

Separately, will you have a chance to look at PRs that specifically target the 2nd leaderboard? I have 3 PRs (Ternary #923 this one adds to the binary run that's already present in the 2nd leaderboard, XNOR-net #1388, LeWorldModel Mamba2 #903, all 10 mins and unlimited compute) and I'm sure others have many too that were for the 2nd leaderboard.

[serdardoesml]

Any reason why this is not merged? Can we consider #1855 the current best record?

[msisovic]

Hi @cocohearts I notice the latest row removes my row, along a very similar entry to it. I assumed it was supposed to deduplicate, or was this intentional?

[msisovic]

Hi @cocohearts I notice the latest row removes my row, along a very similar entry to it. I assumed it was supposed to deduplicate, or was this intentional?

Took another look, the other entry wasn't a duplicate, just had a similar title. Still the point stands, my PR was removed even though it is a valid submission.

[simon-marcus]

@cocohearts flagging for the next leaderboard pass: my submission #1925 landed just outside the #1902 stated scan range (#1494-#1908), but stands as a seed-matched progression over #1855.

Current headline is 1.06031736 BPB, using the same seed set as #1855 (42, 0, 1234). This is a composite eval-only update on the original #1925 quantized artifacts: training/quantization are unchanged, and only the legal score-first TTT eval is updated to PHASED_TTT_PREFIX_DOCS=3500, PHASED_TTT_NUM_PHASES=1, TTT_LORA_LR=8e-5.

Matched deltas vs #1855:

• seed 42: 1.05906444 vs 1.05989454 (-0.00083010)
• seed 0: 1.06059202 vs 1.06124613 (-0.00065411)
• seed 1234: 1.06129561 vs 1.06208695 (-0.00079134)
• mean: 1.06031736 vs 1.06107587 (-0.00075852)

Composite logs are included as train_seed*_ttt_n1_lora8e5.log, with an explicit eval-only continuation marker and no model artifacts.