Record: LongCtx No-QV QK5.25 + AsymLogit + LQER g32/top4 + TTT-local 0.80 — 1.05792 BPB 3-seed mean

[S0urC10ud]

Summary

This PR adds a 10min/16MB record based on a five-knob hyperparameter retune of PR #2007's LongCtx No-QV QK5.25 + AsymLogit configuration.
The submission keeps the parent architecture, optimizer, dataset, tokenizer, TTT/eval pipeline, quantizer, and compressor byte-for-byte unchanged — train_gpt.py is byte-identical to #2007 (md5 2a7e36e29aa5b5811abb6170059aa8d1). Only five env-var scalars are retuned.

Record folder:

records/track_10min_16mb/2026-05-01_LongCtx_NoQV_QK525_AsymLogit_LQERg32top4_TTTlocal080_1.0579/

Results

Seed	Stop step	Train time	Final TTT BPB	Artifact bytes
42	4868	596.142 s	1.05781454	15,971,753
0	4861	595.821 s	1.05798212	15,971,492
1234	4873	595.991 s	1.05796494	15,971,748
Mean			1.05792053
Std			0.00007528

All seeds satisfy the 10-minute / 16 MB rules:

• train_wallclock_s ≤ 600 ✓ (595.8 – 596.1 s)
• TTT phased eval_time_s ≤ 600 ✓ (395.4 – 397.6 s)
• Total submission size ≤ 16,000,000 B ✓ (15,971,492 – 15,971,753 B)
• Artifact slack ≥ 28,247 B on every seed

What changed vs parent #2007

Five env-var deltas only:

Knob	Parent #2007	This PR	Direction
MATRIX_LR	0.026	0.028	slightly higher matrix LR
LQER_RANK	4	2	half-rank LQER correctors
LQER_ASYM_GROUP	64	32	finer asym-quant groups
LQER_TOP_K	3	4	one extra top-K corrector slot
TTT_LOCAL_LR_MULT	0.75	0.80	slightly hotter local TTT step

Comparison vs parent #2007 (paired, same 3 seeds)

Seed	Parent #2007 BPB	This PR BPB	Δ BPB
42	1.05857451	1.05781454	−0.00076
0	1.05915199	1.05798212	−0.00117
1234	1.05924929	1.05796494	−0.00128
Mean	1.05899193	1.05792053	−0.00107

Paired one-sided t-test: mean Δ_loss = −0.00234 nats, t = −6.73, p ≈ 0.011.

Comparison vs currently-merged SOTA #1493 (1.0810)

Δ_BPB ≈ −0.0231, Δ_nats ≈ −0.051. Every individual seed improves by ≥ 0.022 BPB, far above the 0.005-nat record threshold.

Method

The frozen parent recipe (unchanged here):

• CaseOps/SP8192 tokenization with byte-sidecar BPB accounting.
• Sparse attention gating, BOS-fixed SmearGate, skip gates, LQER correction, int7 embeddings, and mixed-precision GPTQ + AWQ-lite.
• 2560-token eval and TTT windows.
• No-QV TTT masking, keeping K/O/MLP adaptation active.
• TTT_LORA_RANK=80, PHASED_TTT_PREFIX_DOCS=3000.
• QK_GAIN_INIT=5.25, WARMDOWN_FRAC=0.85, MIN_LR=0.1.
• Eval-only asymmetric logit rescale.
• Per-group lrzip -L 9 compression.

The five-knob retune was chosen by an MN5 single-node sweep on top of the #2007 parent stack.

Reproduction

Prepare the CaseOps dataset once:

python prepare_caseops_data.py --local-dir /workspace/caseops_data

Run a seed from this folder:

SEED=42 \
CASEOPS_ROOT=/workspace/caseops_data \
RUN_ID=longctx_noqv_qk525_asym_lqer_g32_top4_tttlocal080_seed42 \
./run_current_candidate.sh

The script sets the full environment and runs:

torchrun --standalone --nproc_per_node=8 train_gpt.py

Repeat with SEED=0 and SEED=1234 for the matched 3-seed validation.

Logs

• train_seed42.log — final BPB 1.05781454
• train_seed0.log — final BPB 1.05798212
• train_seed1234.log — final BPB 1.05796494

Hardware / software

• 8 × NVIDIA H100 80GB HBM3 SXM (RunPod)
• PyTorch 2.9.1+cu128, CUDA 12.8
• Same FlashAttention / Triton / runtime stack as parent Record: LongCtx No-QV QK5.25 + AsymLogit — 1.05899 BPB 3-seed mean #2007 record.