Non-record: sin² activation + causal screening pipeline

[clthuang]

Non-Record Submission: sin² Activation + Causal Screening Pipeline

Summary

• sin² periodic activation replacing relu² in MLP layers (1-line change in train_gpt.py)
• Discovered via an automated causal screening pipeline that tested 24 interventions
• Novel explorations: adaptive multi-token prediction (variable-depth MTP) and Rho-1 selective loss masking

Key Change

# Before (relu²):
x = torch.relu(self.fc(x))
return self.proj(x.square())

# After (sin²):
return self.proj(torch.sin(self.fc(x)).square())

Why sin²?

Motivated by FAN/FANformer (NeurIPS 2025) showing 31% parameter efficiency with periodic activations at 1B scale. sin² preserves relu²'s output range (non-negative) and squaring structure, while eliminating the hard zero cutoff that kills 50% of activations.

Screening Results (24 interventions, Apple Silicon MLX)

Intervention	Train Loss Delta	Notes
softcap=20	-0.447	Strongest signal
dim=640	-0.311	Wider > deeper for short training
sin² activation	-0.017	Competitive with GELU (-0.014) and SiLU (-0.018)
adaptive-K MTP	+0.008	Needs longer training (warmup-dependent)
rho1 selective loss	pending	Threshold calibration needed

Infrastructure

Built a 12-script automated experimentation pipeline (144 tests):

• Causal DAG extraction from leaderboard ablation data
• Discovery-adjust cycle with paired-seed experiments
• In-process MLX training with warmup caching (7× faster than subprocess)
• Per-step loss tracking with curve plotting

Status

• ✅ Local MLX screening complete (24 interventions)
• ⏳ H100 validation pending compute credits
• ⏳ 3-seed statistical significance pending

Files

• records/track_non_record_16mb/2026-03-27_CausalScreening_SinSq_AdaptiveK/train_gpt.py — modified baseline with sin² activation
• records/track_non_record_16mb/2026-03-27_CausalScreening_SinSq_AdaptiveK/README.md — detailed writeup
• records/track_non_record_16mb/2026-03-27_CausalScreening_SinSq_AdaptiveK/submission.json — metadata

[MatoTeziTanka]

Community Review — Non-record: sin² activation + causal screening pipeline

Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

--- ## Analysis ### 1. BigramHash — CLEAN (no target leakage) BigramHashEmbedding.bigram_hash at lines 662–668: python out[..., 1:] = torch.bitwise_xor(36313 * t[..., 1:], 27191 * t[..., :-1]) % mod The XOR mixes t[i] (current token) with t[i-1] (previous token). No target (next-token) is involved in the hash key. This is a standard causal bigram embedding. The ILLEGAL pattern (target XOR'd into hash key) is NOT present. ### 2. No TTT of any kind There is no test-time training anywhere in the file. The model is trained once on train tokens, evaluated with eval_val (lines 217–273) using torch.inference_mode() — read-only forward pass, no gradient updates on val data. No multi-epoch loop over val_tokens, no score-first guard, no is_last_chunk logic. TTT categories do not apply. ### 3. GPTQ calibration on val_tokens — CLEAN Lines 1229–1232 use val_tokens as calibration data for Hessian collection (collect_hessians). This is a post-training quantization step run after training completes, with torch.inference_mode() (line 336). The Hessians inform INT6 GPTQ weight rounding only — no gradient updates, no weight optimization targeting val loss. This is standard GPTQ calibration, permitted under competition rules. ### 4. Scored-region SLOT — not applicable No evidence of scored-region slot injection or look-ahead into the evaluation window. ### 5. Architecture summary 11-layer transformer with BigramHash(3072, 112), XSA on all layers, Value Residual, U-Net skips, Partial RoPE (16 dims), EMA(0.997), 10% corrupted context training (lines 1157–1162), Late QAT, INT6 GPTQ+lzma. All techniques operate on training data only and are causally legal. --- ## Conclusion No...

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the audit — this looks like a clean pure-neural submission.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.