Record: Sliding Window + FP16 Embed + 10L + Muon WD + Overtone Init (val_bpb=1.1748)

records/track_10min_16mb/2026-03-19_SlidingWindow_FP16Emb_10L_MuonWD_OvertoneInit/README.md

@@ -0,0 +1,28 @@
+# Sliding Window + FP16 Embed + 10L + Muon WD + Overtone Init
+
+**Mean val_bpb: 1.1748** (3 seeds, p<0.001)
+
+## Key Techniques
+
+1. **Sliding window evaluation** (stride=64, seq_len=1024): Every token scored with 960+ context instead of 0-1023 average. Compiled `forward_logits` method for efficient batch inference.
+
+2. **FP16 tied embedding export**: Keep `tok_emb.weight` in fp16 — int8 errors compound through both input and output paths.
+
+3. **10 transformer layers** (up from 9): Muon weight decay compresses enough to fit the extra layer.
+
+4. **Decoupled weight decay for Muon optimizer** (0.02): Improves generalization and quantization robustness.
+
+5. **Overtone spectral embedding init**: SVD power-law spectrum shaping (`S_k ~ k^{-0.5}`).
+
+6. **Phase-transition residual mixing**: Sigmoid-scheduled `resid_mix` initialization.
+
+## Results
+
+| Seed | val_loss | val_bpb | Steps | ms/step |
+|------|----------|---------|-------|---------|
+| 1337 | 1.9849 | 1.1756 | 10424 | 57.55 |
+| 42 | 1.9827 | 1.1742 | 10710 | 56.06 |
+| 7 | 1.9830 | 1.1744 | 10498 | 57.18 |
+| **Mean** | **1.9835** | **1.1748** | | |
+
+Artifact: ~14.7 MB | Eval time: ~162s (sliding window)

records/track_10min_16mb/2026-03-19_SlidingWindow_FP16Emb_10L_MuonWD_OvertoneInit/submission.json

@@ -0,0 +1,16 @@
+{
+  "track": "10min_16mb",
+  "date": "2026-03-19",
+  "name": "Sliding Window + FP16 Embed + 10L + Muon WD + Overtone Init",
+  "author": "notapplica",
+  "seed_results": {
+    "1337": {"val_loss": 1.98492632, "val_bpb": 1.17558517, "steps": 10424, "ms_per_step": 57.55},
+    "42":   {"val_loss": 1.98265459, "val_bpb": 1.17423973, "steps": 10710, "ms_per_step": 56.06},
+    "7":    {"val_loss": 1.98298356, "val_bpb": 1.17443456, "steps": 10498, "ms_per_step": 57.18}
+  },
+  "mean_val_loss": 1.98352149,
+  "mean_val_bpb": 1.17475315,
+  "p_value": 0.0001,
+  "artifact_bytes": 15374243,
+  "code_bytes": 50651
+}