Record: 11L + EMA + Tight SWA + QAT0.15 + VE128 + Partial RoPE + LN Scale (val_bpb: 1.1243)#401

Closed

newjordan wants to merge 134 commits intoopenai:mainfrom

newjordan:experiments/pr374-edge

Commits on Mar 18, 2026

docs: fractal transformer research plan — weight sharing + gravity + AttnRes
Octavian
committed

Commits on Mar 19, 2026

results: first local ladder — fractal 3x3 beats baseline by 7.1% BPB, gravity needs more steps
Octavian
committed

Commits on Mar 21, 2026

Add exact clone of PR #254 — best pending SOTA (1.1313 BPB)

Octavian
and
claude
committed
Add XSA last 3 layers to #254 SOTA clone

Octavian
and
claude
committed
Fix XSA GQA broadcast bug — expand KV heads before manual attention

Octavian
and
claude
committed
Add 3 SOTA improvement experiments: MTP, SwiGLU, Vocab1536

Octavian
and
claude
committed
Add FarnsworthEngine v2: full improvement stack on SOTA254 base

Octavian
and
claude
committed
Add FA3→FA2→SDPA fallback chain for pod restart resilience

Octavian
and
claude
committed
Revert FA3 fallback chain — was unauthorized code change to baseline + untested v2

Octavian
and
claude
committed
Fix FA3 NaN: cast qkv to bf16 before FA3 call, disable dynamo DDP opt

Octavian
and
claude
committed
Add 2-seed validation scripts for exp A/B/C

Octavian
and
claude
committed

Commits on Mar 22, 2026

Log exp A/B results: both behind baseline, zlib fallback bug found

Octavian
and
claude
committed
Fix XSA NaN: position 0 has no valid targets when self-mask + causal combine

Octavian
and
claude
committed
Disable XSA in ttt_only run — manual attention too slow vs FA3

Octavian
and
claude
committed
Add run_v2_ttt_noXSA.sh — TTT v2 + temp scaling, all FA3, max speed

Octavian
and
claude
committed
Restore XSA_LAST_N=3 in run_v2_ttt_only.sh (keep existing test intact)

Octavian
and
claude
committed
Log v2 TTT-only + XSA=3 result: 1.1982 BPB (worse than 1.1301 baseline)

Octavian
and
claude
committed
Strip verbose logging from v2 train loop — match baseline format

Octavian
and
claude
committed
Log v2 noXSA result: 1.1538/1.1315 BPB — TTT v2 hurt, no edge over baseline

Octavian
and
claude
committed
Log exp_a/b/c results: all worse than 1.1301 baseline, exp_c never ran

Octavian
and
claude
committed
Add exp D: TTT 8 epochs + stride 32 (eval-only improvement)

Octavian
and
claude
committed
Add SAM (Sharpness-Aware Minimization) option for TTT

Octavian
and
claude
committed
Add baseline reproduction script — verify 1.1303 on current FA3 build

Octavian
and
claude
committed
Add SAM to baseline TTT — test sharpness-aware adaptation on proven code

Octavian
and
claude
committed
Log exp D result: 1.1295 BPB — new best (-0.0008 vs baseline)

Octavian
and
claude
committed
Log exp D seed 42: 1.1307 BPB — confirms improvement (mean 1.1301)

Octavian
and
claude
committed
Add exp_d SAM variant — TTT 8ep + stride 32 + sharpness-aware TTT

Octavian
and
claude
committed
Log exp D seed 7: 1.1313 BPB but 16.18 MB — over size limit

Octavian
and
claude
committed
Add Partial RoPE + LN Scale (from PR #315) to sota254 + run_sam

Octavian
and
claude
committed
Add exp_d/run_sam_clean.sh — pure SAM A/B test, no other changes

Octavian
and
claude
committed
Log exp D seeds 7+137: both over size limit

Octavian
and
claude
committed
Add Sponge Bath experiment: TTT 8ep + stride 32 eval-only improvement
Octavian
committed
Add PR#315 clone with TTT 8ep + run script
Octavian
committed
Log D+SAM+PR315tricks: 1.1274 BPB new best; add SAM to pr315 run script
Octavian
committed
Log PR315+TTT results: 1.1240 BPB (invalidated — TTT now banned)
Octavian
committed
Add PR374 enchilada experiment: 12L/2KV/2.75xMLP + train@1024 + EMA

Octavian
and
claude
committed
Add RunPod 8xH100 setup guide — every gotcha we've hit 3 times

Octavian
and
claude
committed
Add fractal cadence experiment: F/N alternation pattern

Octavian
and
claude
committed
Add PR374-safe: EMA + earlier QAT + longer warmdown only, shape unchanged

Octavian
and
claude
committed
Add PR374-depth: 12L/4KV/2.625xMLP (same params, +1 layer) + EMA + QAT + warmdown

Octavian
and
claude
committed
Add minified train_gpt (30KB vs 74KB) with EMA+SWA edge

Octavian
and
claude
committed
Fix log0 keyword arg bug in minified script

Octavian
and
claude
committed
Fix Block kwarg: layer_idx -> li in minified script

Octavian
and
claude
committed
Fix a.te -> a.tie_embeddings in minified script

Octavian
and
claude
committed
Add pr374_submit: trimmed winner for submission

Octavian
and
claude
committed
Remove duplicate stride=64 eval from submit script

Octavian
and
claude
committed
Revert "Remove duplicate stride=64 eval from submit script"
Octavian
committed
Add fractal cadence H100 script: 4 unique × 3 loops + ortho positions

Octavian
and
claude
committed
Add fractal cadence long run (1.6575 BPB @ 3929 steps) + H100 script

Octavian
and
claude
committed
Add pr374_slim: comment-stripped pr374_safe (67KB vs 74KB, -6.4KB)

Octavian
and
claude
committed
Add fractal cadence auto-research sweep runner

Octavian
and
claude
committed
Fix FA3 head_dim limit: 12 heads / 6 kv_heads for dim=768 (head_dim=64)

Octavian
and
claude
committed
Add pr374_ttt: PR374 + EMA + TTT(20ep,SAM) — rock the house

Octavian
and
claude
committed
Fix early QAT trigger: skip warmdown/QAT for first 100 steps

Octavian
and
claude
committed
Optimize fractal: tuned H100 defaults + focused sweep grid

Octavian
and
claude
committed
Add TTT eval: test-time training on already-graded tokens

Octavian
and
claude
committed
Log depth + TTT results: 1.1223 BPB (12L+TTT20ep+SAM)

Octavian
and
claude
committed
Save pr374_throne: our #1 non-TTT record holder (1.1243 BPB)

Octavian
and
claude
committed
Add autonomous overnight fractal optimizer (Karpathy auto-research)

Octavian
and
claude
committed
Rewrite autoresearch: Qwen-guided fractal optimizer via Ollama

Octavian
and
claude
committed
Add 3 leapfrog variants based on PR#414 SOTA (1.1233 BPB)

Octavian
and
claude
committed
Add pod setup script for leapfrog variants (v1/v2/v3)

Octavian
and
claude
committed
Fix setup_pod.sh: apply FA3 build fixes from RUNPOD_SETUP.md

Octavian
and
claude
committed
Update fractal H100 with overnight Qwen findings

Octavian
and
claude
committed
Fix v1 TTT burst: apply EMA first, then QAT-aware sharpening

Octavian
and
claude
committed
Add inner-TTT to fractal eval: recursive weight improvement per loop

Octavian
and
claude
committed
v1 combo: burst+QAT before EMA + 15 GPTQ percentiles

Octavian
and
claude
committed
Isolate TTT to fractal layers only: blocks + loop_pos + skip_weights

Octavian
and
claude
committed
Add TTT drift gate: snap block weights back toward trained originals

Octavian
and
claude
committed
Add v4: burst + distill + train_seq_len=1024 for more steps

Octavian
and
claude
committed
Add SOTA autoresearch: Qwen-guided edge finding for 1.1233 record

Octavian
and
claude
committed
Fix TTT graph bug + upgrade to 3 blocks x 896d (23M params, ~14MB)

Octavian
and
claude
committed
Save leapfrog experiment results: 6 variants tested, v1 wins at 1.12319

Octavian
and
claude
committed
Fractal v2: dim=960 (25.7M, ~15.8MB), Muon LR back to 0.025

Octavian
and
claude
committed
Disable TTT in v2 (backward graph bug), focus on bigger model

Octavian
and
claude
committed
Add v5: QAT percentile fix + TrigramHash + EMA-SWA blend

Octavian
and
claude
committed
Fix TTT graph bug: fresh vc + detached inputs per loop

Octavian
and
claude
committed
Two v2 scripts: with TTT and without, for A/B comparison

Octavian
and
claude
committed
Fractal v3: MLP 3.0->3.3 fills 16MB budget (27.4M params, ~15.2MB)

Octavian
and
claude
committed
Fractal v4: 1.5x batch (1.18M tokens/step) to use spare VRAM

Octavian
and
claude
committed
Log The Frugendorff: fractal cadence baseline at 1.2113 BPB

Octavian
and
claude
committed
Add v6: fractal 6L×2 loops (12 effective) + 480d/16H/4xMLP

Octavian
and
claude
committed
Frugendorff v5: TTT warmup-then-freeze to capture 1.19 peak

Octavian
and
claude
committed
Log all Frugendorff results: v3 TTT peak 1.1901, v4 batch 1.2186

Octavian
and
claude
committed
Fix v6: 512d/16H/8KV (head_dim=32, FA3 requires multiple of 8)

Octavian
and
claude
committed
Log v5/v6 results + add fractal sweep script

Octavian
and
claude
committed
Frugendorff 8-hour single GPU longrun

Octavian
and
claude
committed

Commits on Mar 23, 2026

Complete Frugendorff documentation: all runs, findings, architecture analysis

Octavian
and
claude
committed
Add v7: legal score-first TTT eval (PR #461 recipe)

Octavian
and
claude
committed
Add v8: mutual learning — flat GPT + fractal GPT co-training

Octavian
and
claude
committed
v7: add TTT early stopping — train only first 60 chunks

Octavian
and
claude
committed
Update v6 to sweep winner: 4×3/512d/8H/4KV/4xMLP

Octavian
and
claude
committed
Log v7 TTT results: early stop at 60 chunks = 1.12312 (best)

Octavian
and
claude
committed
The Frugendorff Squared: 6L x 2 loops, dim=640, MLP 4x, 28.5M params

Octavian
and
claude
committed
Log higher LR (0.030) + MTP results — both worse

Octavian
and
claude
committed
The Frugendorff Squared: 1.1478 BPB — 0.025 from world lead

Octavian
and
claude
committed
Stack distillation onto Frugendorff Squared

Octavian
and
claude
committed
Prep Frugendorff Squared PR submission (draft — needs image)

Octavian
and
claude
committed
Add train script to Frugendorff submission folder

Octavian
and
claude
committed
TTT stabilization: EMA scoring + fixed cosine LR + embed freeze

Octavian
and
claude
committed
Add GPTQ quantization: Hessian-aware error compensation for int6

Octavian
and
claude
committed
Stack low-hanging fruit: fix GPTQ, earlier QAT, matched clipping

Octavian
and
claude
committed
Add post-quant training burst: repair quant damage before eval

Octavian
and
claude
committed
Remove post-quant burst — illegal (uses training data during eval)

Octavian
and
claude
committed
Add selective int8 for sensitive layers (GPTQ-int8 upgrade path)

Octavian
and
claude
committed
Frugendorff V3: v7 full stack + fractal loops (GPTQ + TTT + PQB)

Octavian
and
claude
committed
Add AdamW TTT option — PR #462 shows 5x better TTT gain vs SGD

Octavian
and
claude
committed
Frugendorff V3: SwiGLU + AdamW TTT + 8K bigram (PR#462 techniques)

Octavian
and
claude
committed
Fix DDP unused params: add find_unused_parameters=True

Octavian
and
claude
committed
SwiGLU + GPTQ: PR #462 architecture with our quant improvements

Octavian
and
claude
committed
Add OptRot: Hadamard rotation before GPTQ quantization

Octavian
and
claude
committed
Frugendorff V3: Star-ReLU SwiGLU + VRL + MLP 4.0 (fits 16MB)

Octavian
and
claude
committed
Log session results: 1.0763 SwiGLU (size problem), 1.1215 v7 (submitted)

Octavian
and
claude
committed
Frugendorff V4: GEPA-matched config for 16MB compression

Octavian
and
claude
committed
Frugendorff V4 A/B: 5x2 and 4x2 variants for H100 calibration

Octavian
and
claude
committed
Add short TTT experiment: no-EMA, 50 chunks, SGD

Octavian
and
claude
committed
Add XSA-all(11) + short TTT run script

Octavian
and
claude
committed
Record: v7 GPTQ + Short TTT — val_bpb 1.1207 (seed 1337)

Octavian
and
claude
committed
Frugendorff compression shim on SwiGLU SOTA (1.0763 model)

Octavian
and
claude
committed
Add v7 smooth run: proper warmdown + XSA-all

Octavian
and
claude
committed
Reduce bigram buckets 2048→1792 for XSA-11 size fit

Octavian
and
claude
committed
Frugendorff stacked: LeakyReLU(0.5)² + VRL + decoder_lr_mult=2.0

Octavian
and
claude
committed
Add research batch script + GPTQ env var tuning

Octavian
and
claude
committed
Sanitize: remove all illegal TTT from Frugendorff scripts

Octavian
and
claude
committed
Add Frugendorff SwiGLU sweep: 9 configs across 3 batches

Octavian
and
claude
committed
EMERGENCY: Disable all illegal TTT across entire project

Octavian
and
claude
committed
CRITICAL: Remove illegal TTT from train_gpt_swiglu.py

Octavian
and
claude
committed
Add SwiGLU Frugendorff compression sweep (6 configs)

Octavian
and
claude
committed
SwiGLU F1: VRL + LeakyReLU(0.5)² + legal TTT + int8 attn.proj + seq2048

Octavian
and
claude
committed
Fix DDP: find_unused_parameters=True for VRL block 0

Octavian
and
claude
committed
Add GPTQ re-quantization sweep (no training, ~2 min)

Octavian
and
claude
committed
Save Gold Standard (GS): v7 GPTQ 1.1206 BPB — 3 copies

Octavian
and
claude
committed
Restore original train_gpt_swiglu.py, new F1 variant as separate file

Octavian
and
claude
committed
Log SwiGLU F1 run: 1.1208 BPB (20.6MB, needs compression)

Octavian
and
claude
committed