Add Combined Int6 + QAT + Sliding Window submission#149

Draft

pleasedontddosme wants to merge 11 commits intoopenai:mainfrom

pleasedontddosme:combined-int6-qat-slidingwindow

Commits on Mar 20, 2026

Add Combined Int6 + QAT + Sliding Window submission

pleasedontddosme
and
claude
committed

Commits on Mar 21, 2026

Fix GQA compatibility with PyTorch 2.4

pleasedontddosme
and
claude
committed
V2: Remove QAT, 10 layers, eval@2048/stride=128, GQA autodetect

pleasedontddosme
and
claude
committed
Fix PyTorch 2.8 compat: version-based GQA detection, weights_only=False

pleasedontddosme
and
claude
committed
V4: Full rewrite based on thwu1's winning 1.1428 architecture

pleasedontddosme
and
claude
committed
V5: Add TTT LoRA eval + label smoothing + z-loss + dual bigram in TTT

pleasedontddosme
and
claude
committed
V5.1: Fix MTP init bug, add LR floor, strip training-only params from artifact

pleasedontddosme
and
claude
committed
V5.2 Enhance hyperparameters and optimizer: add Partial RoPE, LN Scale, XSA, EMA; update learning rates and SWA settings
pleasedontddosme
committed
Fix PyTorch <2.5 compat: conditional device_id in init_process_group

pleasedontddosme
and
claude
committed
Allow DIST_BACKEND override (gloo fallback for broken NCCL)

pleasedontddosme
and
claude
committed
Fix torch.compile fullgraph bug on PyTorch 2.4 (mul_softmax_pattern crash)

pleasedontddosme
and
claude
committed