Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb)

[jorge-asenjo]

Summary

3-seed mean val_bpb 1.07219 on Track A (10min/16MB) using multi-phase global SGD at test-time combined with phased LoRA TTT, SP-8192 tokenization, int7 embeddings, per-layer GPTQ with sigma clipping, Muon optimizer, depth recurrence, VarLen flash attention, and fused triton MLP.

Results

Seed	val_bpb	artifact
42	1.07332	15,930,192 B
0	1.07115	15,939,461 B
1234	1.07211	15,930,004 B
mean	1.07219	all <16MB

Approach

Multi-phase global SGD splits validation into N phases. Within each phase:

1. All chunks are scored under torch.no_grad() (score-first)
2. Base model weights updated with SGD on scored tokens (train on already-scored tokens only)

This cycles for 3 phases, letting the base model progressively adapt to validation distribution while remaining legal under Issue #1017 (causal, normalized softmax, score-before-update, single pass).

Legal compliance

• Causal: standard causal attention
• Normalized softmax: yes
• Score-before-update: each chunk fully scored under torch.no_grad() BEFORE any SGD update
• Single pass: each token scored exactly once

Reproduction

8x H100 SXM, torch 2.9.1+cu128, flash_attn_3 (Hopper wheel). Env vars and seeds documented in records/track_10min_16mb/2026-04-16_SP8192_MultiPhaseGlobalSGD_PhasedTTT/README.md.

Test plan

• 3-seed run completed with full training and evaluation logs
• All artifacts fit under 16MB cap
• Scores consistent across seeds (spread: 1.07115 - 1.07332)

[himanshudongre]

Reading through this PR I am getting three different per-seed mean candidates depending
on which artifact I look at, and I want to make sure I am reading them right.

The README per-seed table gives 1.07294 / 1.07213 / 1.07259 (mean 1.07255). The
committed seed logs give 1.07332 / 1.07115 / 1.07211 (mean 1.07219). The
submission.json reports 1.07219, matching the logs.

Could you confirm which set is canonical? If the logs are authoritative the headline
should probably say 1.07219 to match submission.json; if the README table is right
something needs to be reconciled with the logs.

[jorge-asenjo]

Good catch — the logs and submission.json are authoritative (1.07219). The stale README table was from an earlier 3-seed run whose per-seed logs were lost during a pod restart; I then re-ran all three seeds and that's the run that is committed (train_seed{42,0,1234}.log). I've just pushed 5f54d26 updating the headline and table to 1.07219 / seeds 1.07332 / 1.07115 / 1.07211 so README, logs and submission.json all agree.