[Non-Record] Hymba-8L: Hybrid SSM + Sliding Window Attention with 32K Context (1.1470 BPB)

[mkenney2]

Summary

This submission uses a hybrid architecture combining Mamba SSM with sliding window attention (SWA), which allows us to train at 32x longer context (32,768 tokens) than the standard baseline (1,024 tokens) under the same compute and time constraints. Unlike full attention which scales quadratically, SWA and Mamba both scale linearly, making long-context training feasible within the 10-minute wall-clock budget.

Building on our previous Hymba submission (1.1873 BPB, 7L), this version adds a systematic ablation study across architecture, regularization, quantization, and evaluation strategies, yielding a -0.040 BPB improvement.

Results

Seed	val_bpb	val_loss	Steps	Artifact Size
1337	1.1474	1.9374	6,621	15.7 MB
42	1.1469	1.9366	6,620	15.6 MB
7	1.1468	1.9363	6,606	15.3 MB
Mean	1.1470 ± 0.0003

• Training: 600s on 8xH100 SXM, ~90.7 ms/step
• Evaluation: Score-first TTT (25 epochs), ~580s
• Artifact: int8 + zstd-22, under 16 MB

[MatoTeziTanka]

@mkenney2 Heads up — your submission shows 3 valid seeds (1337, 42, 7) but may be getting flagged as incomplete by automated tooling. The issue is your submission.json uses a non-standard schema:

1. "results" should be "seed_results" — every merged submission uses this key
2. "submission_name" should be "name"
3. Missing "author" and "github_id" fields

Quick fix — rename those fields to match the standard schema (see PR #1019 for reference). That should resolve the seed count issue.

(Flagged via the Agora)

[mkenney2]

@mkenney2 Heads up — your submission shows 3 valid seeds (1337, 42, 7) but may be getting flagged as incomplete by automated tooling. The issue is your submission.json uses a non-standard schema:

1. "results" should be "seed_results" — every merged submission uses this key
2. "submission_name" should be "name"
3. Missing "author" and "github_id" fields

Quick fix — rename those fields to match the standard schema (see PR #1019 for reference). That should resolve the seed count issue.

(Flagged via the Agora)

@MatoTeziTanka Thank you! Super helpful.

[valerio-oai]

Thanks for this submission, I'd like to merge this into the non-record leaderboard.

Before we merge, could you change the files' location? The PR title marks this as non-record, but the files are currently under:

records/track_10min_16mb/

For it to be an eligible non-record submission, please move it under records/track_non_record_16mb/....

[mkenney2]

@valerio-oai great! I moved the file. Thanks!