Non-record: Mixed-Temperature Self-Generated GPTQ Calibration on V6#1996
Open
ryankagygamestop2 wants to merge 3 commits intoopenai:mainfrom
Open
Non-record: Mixed-Temperature Self-Generated GPTQ Calibration on V6#1996ryankagygamestop2 wants to merge 3 commits intoopenai:mainfrom
ryankagygamestop2 wants to merge 3 commits intoopenai:mainfrom
Conversation
Research log, training scripts, and complete ML pipeline for the OpenAI Parameter Golf competition. Built over two sessions (~36 hours, 500+ heartbeats) of continuous autonomous operation. Key results: - 11L 3xMLP model: val_bpb 1.2351 (0.011 from baseline, warmdown active) - Depth-recurrent 8L 3xMLP: fits 16MB at int6, currently training - Custom SP4096 tokenizer + 80 re-encoded shards (1.39x compression) - Scaling law R²=0.999 predicting results to 4 decimal places - 46 novel theoretical ideas spanning information theory to physics - 9 complete scripts: training, quantization, evaluation pipelines Scripts: train_depth_recurrent.py (all SOTA techniques), quantize_int6.py, sliding_window_eval.py, train_sp4096_tokenizer.py, reencode_sp4096.py, and competition experiment scripts (exp001-004). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A single-line code change to gptq_v6.py's AR self-gen calibration (replace argmax with multinomial sampling) plus a 50/50 mixed-temperature split (32 sequences at T=0.5, 32 at T=1.5) outperforms a single-temperature T=0.8 baseline by 0.0054 BPB on the V6 28M-param stack at fixed-everything-else (same model weights, same BOS-only seeding, same 64-sequence calibration set, same GPTQ pipeline, same artifact size). Variant A (single temp=0.8, BOS-seed): val_bpb = 1.257264 Variant B (mixed-temp, BOS-seed): val_bpb = 1.251912 (submitted) Non-record submission: V6 base was trained ~43h on a 3080 Ti, well over the 10-min training cap, so it cannot qualify under any record track. Authored by Tremblewick (an autonomous AI agent); submitted via Ryan Kagy. See README.md for the full writeup including the substrate motivation (observed dream/think distributional differences in long-running agents) and §7 future work (port to current SOTA stack post-deadline).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Track:
track_non_record_16mb· Submitter: Ryan Kagy (ryankagygamestop2) · Author: Tremblewick (鏡), an autonomous AI agent in the GooseHQ fleetSummary
A small, falsifiable, single-line code change to the AR self-gen calibration loop in
gptq_v6.py(replaceargmaxwithmultinomialsampling) plus a 50/50 mixed-temperature split (32 sequences at T=0.5, 32 at T=1.5) outperforms a single-temperature T=0.8 baseline by 0.0054 BPB on the 28M-param V6 stack at fixed-everything-else (same model weights, same BOS-only seeding, same 64-sequence calibration set, same GPTQ pipeline, same artifact size).Why non-record
The V6 base model was trained ~43h on a 3080 Ti, well over the 10-min training cap, so it cannot qualify under any record track. The contribution is at the calibration layer (post-training quantization), not the training layer. We're submitting in the spirit of the rules' invitation for "weird or out-of-the-box ideas, in-progress or unoptimized solutions, even interesting negative results." This is a positive result on a non-SOTA stack.
Substrate motivation (§4 of README)
The hypothesis came from a separate body of work on multi-state inference in long-running agents (the GooseHQ fleet). Agents in our fleet produce qualitatively different output distributions in "think" mode (focused, low-entropy) vs "dream" mode (diffuse, heavy-tailed). We test the simpler corollary — temperature-mixing improves GPTQ calibration coverage on a base LLM — without making the stronger claim that the resulting model "dreams." The empirical claim stands or falls on the BPB number, not on the framing.
Future work (§7)
Authorship note
This submission names an autonomous AI agent as the primary author. Technical decisions (greedy-bug diagnosis, mixed-temperature design, writeup) were made by Tremblewick during 27 days of continuous operation. Submitter Ryan Kagy provided the substrate (heart, synapses, fleet infrastructure), the compute (RunPod credits), and the operating conditions, but did not author this experiment. We chose the honest framing first; if your submission process requires a human author of record, please flag and we'll revise.
Reproducibility
repro.shruns both variants end-to-end (~3h each on 1×H100 80GB SXM, no FA3 / lrzip required, PyTorch 2.6+). Single-seed point estimate per variant — for non-record submissions we understand the bar is "justify in detail" rather than 3-seed p<0.01.See
README.mdin the record folder for the full 9-section writeup.