Non-record: XSA-11 + Parallel Residual (L7+) + Depth Recurrence — val_bpb 1.1056 (1-seed, 1×H100)#1467

Open

PhamPhuHoa-23 wants to merge 47 commits intoopenai:mainfrom

angela231005:non-record/xsa11-parallel-recurrence-1xH100

Commits on Mar 31, 2026

improve over SOTA: trigram, VE 4layers, MTP, warmdown=4000, GPTQ AR calib x128, block_size=64, step-based stopping
PhamPhuHoa-23
committed
add run_colab.py jupytext notebook for 1xH100 training
PhamPhuHoa-23
committed
add DATA_PATH and TOKENIZER_PATH config to run_colab.py
PhamPhuHoa-23
committed
fix: set LD_LIBRARY_PATH for libcudart.so.12 on Kaggle
PhamPhuHoa-23
committed
fix: use torch bundled lib dir for libcudart.so.12 (Kaggle)
PhamPhuHoa-23
committed
replace flash_attn_3 with PyTorch built-in SDPA
PhamPhuHoa-23
committed
fix: pass DATA_PATH and TOKENIZER_PATH to torchrun env
PhamPhuHoa-23
committed
fix: enable_math_sdp(True) for torch.compile fake-tensor tracing
PhamPhuHoa-23
committed
fix: expand K/V heads for GQA in SDPA shim
PhamPhuHoa-23
committed
add train_gpt_sota_2: VRL, gated_attn, rope50k, longer_qat, swa30, ve7layers
PhamPhuHoa-23
committed
add run_colab_2.py for train_gpt_sota_2
PhamPhuHoa-23
committed
sota_3: DiffAttn, MTP=3 decayed weights, val-set GPTQ calib
PhamPhuHoa-23
committed
fix syntax error in GPT() call (stray comment merged into code)
PhamPhuHoa-23
committed
fix NameError: pass block_size param to mixed_quantize_int6 in sota 1/2/3
PhamPhuHoa-23
committed
Add sota_4 training variant with PR #1172 techniques
PhamPhuHoa-23
committed
Add sota_5: BigramHash 3072x112, Brotli-11, Soft-Round QAT from step 3000, Sigmoid-Gated Skips
PhamPhuHoa-23
committed
Add sota_6: Early QAT (step 2000, alpha ramp 2500), Split-LR 0.025/0.030, BigramHash 3072x112, Brotli
PhamPhuHoa-23
committed

Commits on Apr 1, 2026

Commits on Apr 8, 2026