Skip to content

Commit 4adcdc6

Browse files
Octavianclaude
andcommitted
X-WING Brown: shared tables + per-order entropy gating, no cubric
PR openai#798's approach on our engine: per-order entropy centers (7:3.0, 6:3.2, 5:3.5, 4:3.8, 3:4.2, 2:4.5) without cubric. Testing if cubric was hurting when combined with per-order gating. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 475a62d commit 4adcdc6

2 files changed

Lines changed: 2101 additions & 0 deletions

File tree

concepts/xwing_brown/run.sh

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
# X-WING BROWN: shared tables + per-order entropy gating (no cubric)
4+
# PR#798 per-order entropy centers on our podracer engine + shared tables
5+
6+
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
7+
REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." && pwd)"
8+
cd "${REPO_ROOT}"
9+
export PYTHONPATH="${REPO_ROOT}/flash-attention/hopper:${PYTHONPATH:-}"
10+
11+
SEED="${SEED:-2045}"
12+
NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
13+
14+
echo "============================================"
15+
echo " X-WING BROWN (shared tables + per-order entropy gating)"
16+
echo " Seed: ${SEED}"
17+
echo " No cubric — per-order gating only"
18+
echo " Chunk tokens: ${NGRAM_CHUNK_TOKENS:-1048576}"
19+
echo "============================================"
20+
21+
SEED="$SEED" \
22+
F1_CORR_RANK=0 \
23+
DISTILL_ENABLED=0 \
24+
MLP_ACT=leaky_relu_sq \
25+
MLP_LEAKY_SLOPE=0.5 \
26+
XSA_LAST_N=4 \
27+
BIGRAM_VOCAB_SIZE=1536 \
28+
TTT_EVAL_ENABLED=0 \
29+
ROPE_DIMS=24 \
30+
NGRAM_EVAL_ORDER=7 \
31+
NGRAM_EVAL_MIN_ORDER=2 \
32+
NGRAM_EVAL_ADAPTIVE=1 \
33+
NGRAM_EVAL_ALPHA=0.30 \
34+
NGRAM_EVAL_ALPHA_MIN=0.05 \
35+
NGRAM_EVAL_ALPHA_MAX=0.70 \
36+
NGRAM_EVAL_ENTROPY_CENTER=3.0 \
37+
NGRAM_EVAL_ENTROPY_SCALE=2.0 \
38+
NGRAM_EVAL_MIN_COUNT=2 \
39+
NGRAM_EVAL_BUCKETS=8388608 \
40+
NGRAM_EVAL_MAX_SECONDS=300 \
41+
CUBRIC_CADENCE=0 \
42+
COMPILE_FULLGRAPH=0 \
43+
torchrun --standalone --nproc_per_node="${NPROC_PER_NODE}" \
44+
"${SCRIPT_DIR}/train_gpt.py" \
45+
2>&1 | tee "logs/xwing_brown_s${SEED}_$(date +%Y%m%d_%H%M%S).log"
46+
47+
echo "============================================"
48+
echo " DONE"
49+
echo "============================================"

0 commit comments

Comments
 (0)