Skip to content

Add W104 faithful SP8192 LegalTTT bad-seed probe#9

Merged
teslaeco merged 1 commit intomainfrom
codex/task-title-bduj3v
Apr 20, 2026
Merged

Add W104 faithful SP8192 LegalTTT bad-seed probe#9
teslaeco merged 1 commit intomainfrom
codex/task-title-bduj3v

Conversation

@teslaeco
Copy link
Copy Markdown
Member

Motivation

  • Reduce bad-seed variance by providing a faithful, evaluator-friendly replay candidate of the SP8192 + LegalTTT near‑SOTA stack with key defaults source-visible.
  • Start reproducibility probe by running only seed=314 first and gate additional seeds (42, 999) on its pass to focus effort on the bad-seed failure mode.

Description

  • Add a new working folder records/track_10min_16mb/2026-04-20_SP8192_LegalTTT_W104_FaithfulReplay/ containing the replay artifacts and guidance.
  • Add train_gpt.py that surfaces source-visible defaults at the top (VOCAB_SIZE, TOKENIZER_PATH, DATA_PATH, TRAIN_SHARDS_OVERRIDE, QK_GAIN_INIT, TTT_ENABLED, TTT_LR, TTT_EPOCHS) and exports them via os.environ.setdefault(...), while preserving the original packed SP8192 LegalTTT payload and architecture/compression surface (3-layer recurrence, parallel residuals, QK gain 5.25, legal score-first TTT, quantized+brotli target under 16 MB).
  • Add run_w104_seed314_probe.sh which bootstraps a venv and deps, caches the official FineWeb sp8192 using MATCHED_FINEWEB_REPO_ID="kevclark/parameter-golf", runs only SEED=314, writes logs to /workspace/w104_seed314.log, and prints the final quantized_ttt val_bpb.
  • Add README.md explaining this is not a new submission, documenting intent and pass criteria for seed314 (must beat 1.08168719, strong pass <1.0812) and instructing to run seeds 42/999 only after seed314 passes.

Testing

  • Ran python -m py_compile train_gpt.py in the new folder and it succeeded.
  • Verified source‑visible defaults with grep for the expected lines in train_gpt.py and all expected defaults were found.
  • Confirmed the change summary via a diff-stat showing the three new files and the expected insertions.

Codex Task

@teslaeco teslaeco merged commit e3e1ab6 into main Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant