Non-record: XSA-All + QK Gain 4.0 + LN Scale — 45 Experiments on 1×RTX 5090

[jainpranjal97]

Summary

Non-record submission: 1.1946 BPB on 1×RTX 5090 (60-min, 3699 steps). 45 systematic experiments exploring hyperparameter space and novel architectures.

Key findings for the community:

• XSA on ALL layers > XSA on last 4 (-0.0018 BPB). Every top entry uses XSA only on deepest 3-4 layers — all-layer XSA is consistently better.
• qk_gain_init = 4.0 (-0.0039 BPB vs default 1.5). Sharper attention patterns help small models significantly. Swept 1.5 → 2.0 → 3.0 → 4.0 with monotonic gains.
• Warmdown calibration for wallclock-capped training (-0.0078 BPB). Default warmdown_iters=1200 means the LR never reaches full strength when wallclock-capped at 10 min.
• Pre-quant vs post-quant divergence: XSA Gating (learned per-head gate) achieved 1.1932 pre-quant (better than best) but 1.1961 post int8+zlib (worse). Architectural choices that improve FP loss can degrade quantized loss.

Novel approaches tested (all documented with negative results):

Approach	Result	Why it failed
Progressive Layer Growing (5→11L at 60%)	+0.0057	5L capacity ceiling
Depth Recurrence 4×3 + LoRA16	+0.0753	torch.compile bypass + optimization conflicts
XSA Gating (learned per-head gate)	+0.0015	Quantizes worse despite better FP loss
Cosine warmdown	+0.0039	Linear warmdown already optimal

Stack

11L, MLP 3×, Partial RoPE 16/64, LN Scale 1/√(layer+1), XSA all layers, LeakyReLU(0.5)², Muon WD 0.06, seq 2048, grad_clip 0.3, qk_gain 4.0, logit_softcap 20.

Full experiment log with 45 runs in the README.

Test plan

• Verified val_bpb 1.1946 on 1×RTX 5090 (60-min run)
• All 45 experiments logged with reproducible configurations
• train_gpt.py included and runnable

🤖 Generated with Claude Code

[MatoTeziTanka]

Community Review — Non-record: XSA-All + QK Gain 4.0 + LN Scale — 45 Experiments on 1×RTX 5090

Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

Summary PR #1125 (records/track_non_record_16mb/2026-03-30_XSA-All_QKGain4_LNScale_1x5090/train_gpt.py) is a pure neural submission. Head SHA: 3eff27ce3986cb83e4b50ffc5b474fe45307254a. ## Checks ### N-gram family bug (ILLEGAL: target XOR'd into hash key, NOT BigramHash) NOT PRESENT. No BigramHash, no n-gram table, no hash key, no XOR operation anywhere in the file. The only tgt_ids usage is at lines 270-272 inside eval_val() for byte-counting with base_bytes_lut — this is the standard BPB metric bookkeeping, not a training signal. ### Pre-Quant TTT — multi-epoch on val_tokens without score-first (ILLEGAL) NOT PRESENT. The eval_val() function runs under torch.inference_mode() (line 256) and never calls .backward(). val_tokens is used only for read-only evaluation. The only .backward() calls are at lines 992 and 1061, both operating on train_loader batches (x, y from train_files). model.train() at line 283 is just restoring training mode after evaluation — no gradient flows through val data. ### Score-first TTT / is_last_chunk guard (LEGAL, PR #1413 pattern) NOT PRESENT. No TTT of any kind exists in this submission. No is_last_chunk guard, no score-first logic. ### Scored-region SLOT (HOLD) NOT PRESENT. No scored-region slot indicators found. ## Novel technique The PR introduces XSA (Cross-Self Attention) via the _xsa_efficient method (lines 594-602) enabled on all attention layers (line 725-727). This subtracts the self-value projection from the attention output: y = y - (y·v̂)v̂ in a GQA-aware manner. This is a zero-parameter architectural modification applied during training, not a post-hoc or inference-only trick. It is pure-neural and architecturally clean. Also includes: LNScale (per-layer ln_scale_factor = 1/sqrt(layer_idx+1) at line 671), QKGain4 (q_gain_init=4.0), and...

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the audit — this looks like a clean pure-neural submission.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.