ArjunAutoResearch: MLP 3x + STE int6 QAT + seq4096 + sliding window. val_bpb 1.1632

[arjun-krishna1]

Hey there! This was super fun thanks. I took the approach of building an AutoResearch agent harness that could work towards solving this autonomously. Added as an agent skill in my submission for others to build off of!

Built an auto-research pipeline:

1. Gave an agent the GitHub CLI
2. Asked it to go through all of the open PRs, describe what the approach being taken was and bucketing it by expected impact on BPB (High, Medium or Low)
3. The agent then composed the approaches in the "High" bucket
4. Iteratively trying to figure out which approaches built on top of each other well and led to the best score

ArjunAutoResearch ended up with a final val_bpb of 1.16323 +/- 0.00042 (mean across 3 seeds, p << 0.001).
The artifact size is: 15,265,243 bytes (under 16,000,000).

With more compute (will apply for more) I would scale this AutoResearch agent by trying to compose approaches in the Medium, Low buckets as well as trying to come up with strategies on its own and researching approaches from the internet that people haven't made pull requests for yet.

The approach ArjunAutoResearch came up with composed the following techniques from these pull requests:

1. Wider MLP (MLP_MULT=3.0, hidden=1536) (from Submission: Wider MLP 3x + int6 quant + sliding window eval, val_bpb=1.1659 #70)
   3x MLP expansion enabled by int6 quantization saving ~4MB. 1536 is 64-aligned for optimal H100 matmul tile utilization.

2. Longer training context (TRAIN_SEQ_LEN=4096) (from Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65)
   4x more context per sequence than the baseline's 1024, significantly improving convergence quality per step.

3. Optimizer tuning (from New SOTA attempt (val_bpb=1.2014) #52)

• MUON_MOMENTUM=0.99, learning rates halved, batch 393K, warmdown 3000 steps

4. STE fake int6 quantization-aware training (from Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65)
   During training, all CastedLinear weights get fake int6 quantization via Straight-Through Estimator: forward uses quantized weights, backward passes gradients through originals. Teaches weight distributions that survive int6 post-training quantization.

5. int6 per-row quantization on MLP+attention (from Submission: Wider MLP 3x + int6 quant + sliding window eval, val_bpb=1.1659 #70)
   Mixed precision: int6 on 2D block weights, fp16 passthrough on tied embedding, zstd-22 compression.

6. fp16 tied embedding passthrough (from fp16 tied embedding + warmdown/LR tuning (val_bpb 1.2197) #42)
   The tied embedding doubles as the output head. Keeping it in fp16 eliminates embedding quantization penalty entirely.

7. Sliding window evaluation (EVAL_STRIDE=64, TRAIN_SEQ_LEN=4096) (from Record: Sliding Window Eval (stride=64), val_bpb=1.1925 #50, extended to seq_len=4096)
   Each token scored with up to 4032 tokens of context. Compiled forward_logits for fast eval.

Results

Seed	val_bpb
1337	1.16356083
1338	1.16275343
1339	1.16337225

Mean: 1.16323, std: 0.00042. One-sample t-test against threshold 1.2194: t = 230.34 (df=2). p << 0.001.

Key numbers (seed 1337)

• Stopped at step 9211/20000 (wallclock cap)
• Pre-quant val_bpb: 1.1769
• Post-quant sliding window val_bpb: 1.16356083
• Train time: 600s at 65.15ms/step
• Eval time: 157s (within the separate 10-min eval budget)
• Artifact size: 15,265,243 bytes (under 16,000,000)
• Peak memory: 8521 MiB

[arjun-krishna1]

Time to sleep 😭

[chonchiog]

Awesome!

[arjun-krishna1]

Awesome!

thanks man! @chonchiog feel free to fork and build off of it

[chonchiog]

for sure! @arjun-krishna1 I'm waiting for the runpod credits :)

[MatoTeziTanka]

Community Review — ArjunAutoResearch: MLP 3x + STE int6 QAT + seq4096 + sliding window. val_bpb 1.1632

BPB: 1.1632 | Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

What I found in the code (head SHA 3d1780d86d24, file records/track_10min_16mb/2026-03-19_ArjunAutoResearch/train_gpt.py):

Static code review found no TTT adaptation function, no SLOT optimization loop, no n-gram-cache class, and no pre-quant val-token fine-tune. The eval path uses the standard sliding-window stride-64 pattern. The submission is a pure-neural architecture iteration on the standard SP1024/SP4096/SP8192 baseline.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=9, vocab=1024, code=57201 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the classification pass — this looks like a clean pure-neural iteration on the standard baseline.

Auto-classification caveat: this review was drafted by the AST-based classifier. If there's a non-standard eval mechanism (logit postprocessing, hedge mixing, etc.) that I missed because it's factored into a helper file or a non-standard function name, please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=9, vocab=1024, code=57201 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.