[WIP] Recurrent MQA Transformer — depth recurrence + weight tying (nidhilak-Aquarius)

[nidhilak-Aquarius]

Recurrent MQA Transformer — WIP Submission

My approach draws from two ideas separated by 2,000 years.

The Chakravyuha in the Mahabharata achieves depth through repetition —
one structural unit looping inward, creating power far beyond its apparent
size. Kalaripayattu, Kerala's martial art, teaches that maximum force comes
from finding the exact marma point, not from raw strength.

Core innovation: One shared TransformerBlock looped 12 times instead
of 9 unique blocks. Same computational depth. 12x fewer unique parameters.

The marma insight: weight sharing acts as a regularizer — the same weights
must generalize across ALL depths simultaneously, forcing more robust
representations than unique per-layer weights ever could.

Architecture:

• Depth recurrence: 1 shared block × 12 loops (Universal Transformer style)
• Weight-tied embeddings: zero-parameter output projection
• Multi-Query Attention: 8Q / 1KV heads (43% fewer attention params)
• SwiGLU FFN: outperforms GELU at identical parameter count (Shazeer 2020)
• RoPE: zero learned positional parameters

Results so far:

• Unique parameters: ~3.5M
• Compressed artifact: ~5.2MB (32.5% of 16MB limit)
• Unused budget: 10.8MB
• val_bpb on FineWeb: pending GPU compute grant

Hypothesis: Recurrence depth N=12 outperforms N=8 at identical
parameter count, with diminishing returns beyond N=16. The compute
grant will map this curve empirically.

Phase 2: BitNet ternary weights {-1,0,+1} at log2(3)=1.58 bits vs
16 bits = ~10x more effective parameters within the same 16MB artifact.

[nidhilak-Aquarius]

Recurrent MQA Transformer — Core Logic

This submission focuses on maximizing effective model capacity under a strict artifact constraint through parameter sharing and architectural efficiency:

• Depth Recurrence — a single TransformerBlock is looped 12x, achieving deep computation with ~3.28M unique parameters (~39M effective).
• Weight-Tied Embeddings — input embeddings are reused for output projection, eliminating additional parameters.
• Multi-Query Attention (8Q / 1KV) — shared KV heads reduce attention parameters and memory overhead.
• SwiGLU FFN — improved efficiency over GELU at the same parameter count.
• RoPE — parameter-free positional encoding.

Artifact size (measured): ~2.82MB (int8 + zlib, smoke test) — well under the 16MB constraint, leaving substantial headroom for further optimization.

Hypothesis: Increasing recurrence depth (N=12) improves performance over shallower configurations at fixed parameter count, with diminishing returns beyond N~16.

Local smoke test completed successfully; full GPU evaluation (val_bpb) pending compute grant.