Non-record: Depth Recurrence Sweep — Systematic Layer Loop Ablation

[krishs0404]

Summary

Systematic ablation of depth recurrence loop configuration in the current SOTA training stack. Five variants tested against the baseline across three axes: which layers to loop (LOOP_START, LOOP_END) and when to activate the loop (ENABLE_LOOPING_AT). Every variant was worse than the SOTA config.

Best result: 1.4689 post-quant bpb (SOTA baseline, included as reference)
Hardware: RunPod 1×H100 SXM 80GB, 6 runs × 10-min wallclock cap ≈ ~60 GPU-minutes total

Results

Experiment	LOOP_START	LOOP_END	ENABLE_LOOPING_AT	Steps	val_bpb (post-quant)	Δ vs baseline
Baseline (SOTA)	3	5	0.35	568	1.4689	—
A — minimal reuse	5	6	0.35	565	1.4750	+0.006
D — early layers	1	4	0.35	538	1.5072	+0.038
C — late layers	7	10	0.35	541	1.5181	+0.049
E — early activation	3	5	0.15	522	1.5190	+0.050
B — heavy reuse	2	7	0.35	451	1.6321	+0.163

Key Findings

• The SOTA config is genuinely optimal. Moving the loop range by a few layers in either direction costs +0.04–0.05 bpb. Not arbitrary.
• Minimal reuse (layers 5–6 only) is surprisingly competitive at +0.006 bpb — most of the recurrence benefit is concentrated in those two layers.
• Heavy reuse (layers 2–7) is catastrophically worse (+0.163) due to throughput loss: only 451 steps vs 568 for baseline at the 10-min budget.
• Early/late layer shifts are symmetrically bad (~+0.04–0.05 bpb). Middle layers (3–5) are the sweet spot.
• Early loop activation (15%) hurts: 46 fewer steps and +0.050 bpb. Representations need to stabilize before recurrence is useful.

See README.md for full analysis, encoder/decoder index lists, and reproduce commands.