Non-record: Discarding Transformers. Elastic Associative Memory as a Language Model, 97% Intelligence Transfer in 14MBs

[nguthiru]

Non-Record: Discarding Transformers. Elastic Associative Memory as a Language Model

We replace the transformer at inference with an Elastic Associative Memory (EAM). Train a teacher transformer, write its hidden states into EAM through additive counter superposition, throw the teacher away.

Artifact: Encoder (426K params) + EAM (12K self-organized locations) + Decoder (1.6M params) = 14.1 MB. Zero transformer layers at eval time.

Results (FineWeb validation)

Metric	Value
val_bpb	2.4646
Intelligence transfer (teacher → EAM)	97.9%
EAM + kNN vs teacher	wins by 5.9%
Reconstruction cosine sim	0.844 (at 50K locations)

The bpb looks underwhelming because the teacher is deliberately weak — basic AdamW, 1 GPU, 68 seconds, 1 data shard, loss 3.69. Swap in a better teacher and the EAM gets better proportionally. The architecture doesn't care what sits upstream.

How it works

1. Train a teacher transformer
2. Write all teacher hidden states into EAM (self-organizing locations accumulate patterns via additive superposition)
3. Train an encoder: tokens → EAM key space
4. Train a decoder: EAM readout → logits
5. Delete the teacher. Ship encoder + EAM + decoder.

Eval path: tokens → encoder → keys → EAM.read(keys) → decoder → logits, plus a flat kNN store using score-first protocol.

What's going on here

14.1 MB of associative memory holds onto 97.9% of a transformer's learned representations. These are stored as counter superposition, not weight matrices. Because EAM counters are accumulate-only, new writes don't clobber old patterns — there's no mechanism for catastrophic forgetting, it just can't happen.

The kNN store turned out to be more than a fallback. EAM + kNN beats the teacher at every scale we tested, which we didn't expect going in. Superposed patterns and exact-match retrieval seem to cover for each other's weaknesses.

Reconstruction quality tracks location count predictably (0.768 at 10K, 0.805 at 20K, 0.844 at 50K), and inference is fixed at 4 layers regardless of how deep the original teacher was.

The accumulate-only property also opens the door to continual learning — you can keep writing new knowledge into the same EAM without degrading what's already there, which is the opposite of how fine-tuning usually goes. Distillation becomes a write operation instead of a training run: any teacher's representations go straight into counters. And because the EAM itself is just a lookup table with simple arithmetic, inference doesn't need a GPU. The whole eval pipeline could run on a CPU or edge device at 14.1 MB.

Based on Elastic Associative Memory (Nguthiru, 2026). Details in the README.

Checklist

• README.md
• submission.json
• train_gpt.py (runs standalone)
• train_log.txt
• requirements.txt
• Artifact under 16 MB (14.1 MB)

## Non-Record: Elastic Associative Memory as a Language Model

We replace the transformer at inference with an Elastic Associative Memory (EAM). Train a teacher transformer, write its hidden states into EAM through additive counter superposition, throw the teacher away.

Artifact: Encoder (426K params) + EAM (12K self-organized locations) + Decoder (1.6M params) = 14.1 MB. Zero transformer layers at eval time.

Results (FineWeb validation)

Metric	Value
val_bpb	2.4646
Intelligence transfer (teacher → EAM)	97.9%
EAM + kNN vs teacher	wins by 5.9%
Reconstruction cosine sim	0.844 (at 50K locations)

The bpb looks underwhelming because the teacher is deliberately weak — basic AdamW, 1 GPU, 68 seconds, 1 data shard, loss 3.69. Swap in a better teacher and the EAM gets better proportionally. The architecture doesn't care what sits upstream.

How it works

1. Train a teacher transformer
2. Write all teacher hidden states into EAM (self-organizing locations accumulate patterns via additive superposition)
3. Train an encoder: tokens → EAM key space
4. Train a decoder: EAM readout → logits
5. Delete the teacher. Ship encoder + EAM + decoder.

Eval path: tokens → encoder → keys → EAM.read(keys) → decoder → logits, plus a flat kNN store using score-first protocol.

What's going on here

14.1 MB of associative memory holds onto 97.9% of a transformer's learned representations. These are stored as counter superposition, not weight matrices. Because EAM counters are accumulate-only, new writes don't clobber old patterns — there's no mechanism for catastrophic forgetting, it just can't happen.

The kNN store turned out to be more than a fallback. EAM + kNN beats the teacher at every scale we tested, which we didn't expect going in. Superposed patterns and exact-match retrieval seem to cover for each other's weaknesses.

Reconstruction quality tracks location count predictably (0.768 at 10K, 0.805 at 20K, 0.844 at 50K), and inference is fixed at 4 layers regardless of how deep the original teacher was.

The accumulate-only property also opens the door to continual learning — you can keep writing new knowledge into the same EAM without degrading what's already there, which is the opposite of how fine-tuning usually goes. Distillation becomes a write operation instead of a training run: any teacher's representations go straight into counters. And because EAM works by arithmetic, CPU inference becomes trivial.

Based on [Elastic Associative Memory](https://doi.org/10.5281/zenodo.18783160) (Nguthiru, 2026). Details in the README.

Checklist

• README.md
• submission.json
• train_gpt.py (runs standalone)
• train_log.txt
• requirements.txt
• Artifact under 16 MB (14.1 MB)

[MatoTeziTanka]

Community Review — Non-record: Discarding Transformers. Elastic Associative Memory as a Language Model, 97% Intelligence Transfer in 14MBs

Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

Architecture PR #1146 ("EAM IS THE MODEL") is a non-standard architecture: it trains a Teacher transformer (discarded after training), then distills knowledge into an Elastic Associative Memory (EAM) with a small Encoder and Decoder. At inference, only Encoder + EAM + Decoder are used — no transformer runs at eval time. ## Checks ### N-gram / BigramHash family bug No n-gram logic, BigramHash, or XOR hash key construction is present anywhere in the file. The EAM operates on normalized float key vectors produced by the Encoder, not discrete hash lookups. Not applicable. ### Pre-Quant TTT (multi-epoch on val_tokens) The EAM, Encoder, and Decoder are all trained exclusively on train_tokens / train_x / train_y (lines 372–512). No gradient updates touch val_tokens at any point. val_tokens is only read during the sliding-window evaluation loop (lines 589–617) and never used for weight updates. No Pre-Quant TTT violation. ### Score-first TTT / flat_store The FlatStore is a kNN retrieval augmentation used during eval. At lines 641–660, the eval loop: 1. Computes nll (scored) from the EAM model before consulting the flat_store (line 637). 2. Retrieves from flat_store (line 642) — only used to optionally blend probabilities if they improve the score (lines 643–650). 3. Accumulates loss_sum / token_count (lines 652–657). 4. Then writes current window keys/tokens to flat_store (line 660, commented "after scoring — legal"). This is a strict score-first, write-after pattern. The kNN store is never written before scoring the current window. There is no is_last_chunk guard here, but none is needed...

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the audit — this looks like a clean pure-neural submission.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.