Non-record: 15L Depth Recurrence + LeakyReLU² — BI-guided weight tying (val_bpb=1.1360)

[aruniyer]

Summary

val_bpb: 1.1360 (seed 1337) | 15.87 MB | 8xH100 SXM

Non-record submission exploring BI-guided depth recurrence: using Block Influence scores (ShortGPT) to identify
which layer positions can share weights, enabling 15 effective layers from 11 unique parameter blocks within the
same 16MB budget as standard 11L.

Result

Seed	Steps	Sliding BPB	Artifact
1337	5173	1.1360	15.87 MB

Key Technique: BI-Guided Weight Tying

1. Train a 15L model, measure Block Influence (angular distance input→output per layer)
2. Layers 9–13 have lowest BI (0.10–0.16) — near-identity transformations
3. Tie those 5 positions to share one physical block → 15 virtual layers, ~27M unique params
4. Deduplicate before quantization: store shared weights once with reconstruction map
5. Int6 + zstd-22 → 15.87 MB ✅

Depth vs Steps Tradeoff

15L runs at 116ms/step (vs 86ms for 11L) due to 4 extra forward/backward layers. In 600s: ~5170 steps vs ~6975 for
11L. The depth advantage doesn't fully compensate for ~1800 fewer steps in this wallclock-limited setting. At equal
step counts, 15L outperforms 11L.

Architecture

15L (10 unique + 1 shared×5), 512d, 8H/4KV GQA, MLP 3x, LeakyReLU(0.5)², XSA4, Partial RoPE 16/64, LN Scale, VE128,
SmearGate, BigramHash(2048), EMA, SWA, Late QAT, int6+zstd-22, FA3.

Reproduce

SEED=1337 NUM_LAYERS=15 TIE_LAYERS=9,10,11,12,13 \
  DIFF_ATTN=0 VRES_ENABLED=0 TTT_EPOCHS=0 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

Base: signalrush (PR #374/#414). LeakyReLU²: PR #493, PR #518. Block Influence: ShortGPT (arXiv:2403.03853).

[MatoTeziTanka]

Community Review — Non-record: 15L Depth Recurrence + LeakyReLU² — BI-guided weight tying (val_bpb=1.1360)

BPB: 1.1360 | Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

What I found in the code (head SHA b31dc4acec07, file records/track_10min_16mb/2026-03-26_15L_DepthRecurrence_LeakyReLU2_TTT/train_gpt.py):

Static code review found no TTT adaptation function, no SLOT optimization loop, no n-gram-cache class, and no pre-quant val-token fine-tune. The eval path uses the standard sliding-window stride-64 pattern. The submission is a pure-neural architecture iteration on the standard SP1024/SP4096/SP8192 baseline.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.06s, dim=512, layers=11, vocab=1024, code=81374 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the classification pass — this looks like a clean pure-neural iteration on the standard baseline.

Auto-classification caveat: this review was drafted by the AST-based classifier. If there's a non-standard eval mechanism (logit postprocessing, hedge mixing, etc.) that I missed because it's factored into a helper file or a non-standard function name, please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.06s, dim=512, layers=11, vocab=1024, code=81374 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.