Five strategies, each at a different point in the memory–compute trade-off:
Local
Sliding Window
Constant KV cache size. Old tokens are evicted — no long-range memory.
Global Dense
Full Attention
Large memory but KV cache and compute grow with sequence length.
Global Sparse
DSA
Sparse reading, but KV cache still grows with sequence length.
Dense Recurrent
Gated DeltaNet
Constant cache size but dense update — limited memory capacity.
Sparse Recurrent
SDM
Constant cache size and large memory via sparse top-k access.
Background
Sequence models face a fundamental trade-off between memory capacity and computational cost.
Sliding window attention (SWA) restricts context to a fixed window. Full attention retains
the entire history but scales quadratically. Recurrent models like Gated DeltaNet (GDN)
and Mamba compress all past context into a fixed-size dense state — typically on the order
of 100K parameters — which bounds both cost and capacity.
Sparse Deltanet Memory (SDM) extends GDN by replacing its dense state matrix with a large
explicit memory bank of N slots (230K at 1.4B scale). Each token reads from and writes to only W=64
of the N slots, selected via product key hashing. This yields a state roughly
1,000× larger than GDN with identical FLOPs per token.
Unlocking linear memory states through sparsity
Each rectangle represents the memory state one global layer maintains, shown for an 8B model (d=3840) in bf16.
Proportions are up to scale. The attention KV-cache size assumes GQA with group size 2 (i.e. n_kv_heads = n_heads/2).
How does SDM work?
Each SDM layer maintains a memory bank M ∈ Rd×N —
a matrix of N column vectors, each of dimension d.
Processing a token involves three steps: select slots via product keys, write to selected
slots with a gated delta rule, and read from (possibly different) selected slots.
Write & Read: slot selection, gated update, and retrieval
The token embedding is projected to write keys via product key hashing, selecting top-k
memory slots. The memory is updated: selected columns are first decayed, then written with the
delta rule. A separate query projection then selects (possibly different) slots for reading.
The output is a weighted sum of the retrieved memory column vectors.
The SDM mechanism, end to end. The memory is a d×N matrix of slot-vectors. Product keys (two sub-keys → outer sum → top-k) select W=64 whole slot-vectors; a gated delta rule writes them (decay, then Δv); a softmax-weighted read gathers them into the output. Those 64 active vectors are exactly the size of GDN's entire dense state — drawn from N ≈ 230,400 slots, whose fixed footprint rivals a 256k-token KV-cache.
Memory Access Patterns on a Needle-in-a-Haystack Task
To see how SDM uses its memory in practice, we run the 1.4B model on a needle-in-a-haystack
task: a key fact is inserted into filler text, and a query about it appears at the end.
The visualization shows which memory slots are accessed at each token position.
Token 0
Filler writes
Needle writes
Query reads
Token sequence
Write Memory
— cumulative writes
Read Memory
— current token reads
Write–Read Bucket Overlap
— do the same slots get written at the needle and read at the query?
Real model data. These heatmaps show actual memory access patterns from the
SDM 1.4B model (128k post-trained) on a 256-token needle-in-a-haystack task.
The needle (“The secret code for Project Alpha is 7392”) is inserted at token 61.
The query appears at token 239. Try layer 7: the model concentrates writes into specific
memory slots when it sees the code “7392”, and reads from the same slots when the query arrives —
the addressing behavior emerges entirely from next-token prediction training.
SDM matches full attention scaling while outperforming GDN
SDM and baselines are trained across seven scales (160M–1.4B parameters) at 160 tokens per
parameter using SWA (w=128) as the local layer in a 3:1 hybrid. SDM matches full attention
training loss at 1.4B (2.055 vs. 2.048) while outperforming GDN at every scale.
Training loss vs. compute (scaling ladder)
Short-Context Benchmark Results
Downstream task accuracy across 15 benchmarks. All models use SWA as the local layer.
Benchmark
FullAttn
GDN
SDM
Δ vs GDN
Val NLL ↓
2.285
2.298
2.253
-0.040
HellaSwag
79.33
79.10
80.02
+0.92
WinoGrande
73.64
73.24
75.30
+2.06
ARC easy
78.10
79.15
78.52
-0.63
ARC challenge
50.82
52.27
53.05
+0.78
PIQA
79.98
81.01
80.47
-0.54
OpenBookQA
43.80
43.00
44.60
+1.60
RACE mid
64.83
60.17
62.05
+1.88
RACE high
47.94
43.40
44.97
+1.57
CommonsenseQA
68.88
66.83
70.60
+3.77
BoolQ
71.83
74.98
67.13
-7.85
TriviaQA
55.23
55.36
59.11
+3.75
HumanEval+
24.39
18.29
24.39
+6.10
NaturalQuestions
22.94
22.60
25.93
+3.33
MMLU
58.73
57.24
57.81
+0.57
GSM8K
29.34
28.81
28.66
-0.15
Average accuracy
56.65
55.70
56.84
+1.14
Benchmark
FullAttn
Mamba2
GDN
SDM
Δ vs GDN
Val NLL ↓
2.660
2.686
2.683
2.654
-0.029
HellaSwag
61.47
61.31
61.27
62.04
+0.77
WinoGrande
60.54
62.67
61.56
62.12
+0.55
ARC easy
61.56
62.88
62.49
63.00
+0.51
ARC challenge
33.56
34.08
34.59
34.76
+0.17
PIQA
73.83
73.67
73.83
74.27
+0.44
OpenBookQA
35.80
36.40
36.80
36.00
-0.80
RACE mid
53.55
50.28
50.42
49.65
-0.77
RACE high
39.62
36.25
36.68
37.76
+1.09
CommonsenseQA
19.57
20.31
19.33
20.72
+1.39
BoolQ
62.42
62.75
63.03
62.78
-0.24
TriviaQA
24.24
23.48
23.68
26.61
+2.93
HumanEval+
8.54
9.76
8.54
9.76
+1.22
NaturalQuestions
7.70
8.09
7.76
8.23
+0.47
MMLU
24.54
25.03
26.02
27.24
+1.22
GSM8K
2.81
3.03
2.96
2.73
-0.23
Average accuracy
37.98
38.00
37.93
38.51
+0.58
SDM excels at long-context evaluation
After 128k post-training, SDM maintains low perplexity from 32K to 1M tokens,
while full attention degrades beyond 128K (ppl rises from 1.98 to 4.17).
On RULER (exact-match, avg over 4k–131k, all 13 tasks), SDM reaches 31.2% at 1.4B and
50.2% at 8B — far above the iso-FLOP GDN (20.0% and 34.2%), and at 1.4B on par with
unbounded-cache full attention (32.5%).
Perplexity by token position (code, 1.4B)
RULER accuracy by task (avg over 4k–131k)
RULER Task
FullAttn
GDN
SDM
Δ vs GDN
single_1
99.3
100.0
100.0
+0.0
single_2
89.4
45.1
71.5
+26.4
single_3
70.1
32.6
74.9
+42.3
multikey_1
75.3
32.9
59.3
+26.4
multikey_2
58.0
1.0
4.7
+3.7
multikey_3
29.8
0.1
1.2
+1.1
multivalue
74.3
31.1
66.1
+35.0
multiquery
73.0
31.2
68.6
+37.4
vt
67.2
46.2
72.3
+26.1
cwe
5.5
11.4
12.8
+1.5
fwe
77.8
62.7
65.0
+2.4
qa_1
39.0
23.6
27.2
+3.6
qa_2
38.4
28.2
30.4
+2.3
Average
61.2
34.2
50.2
+16.0
RULER Task
FullAttn
Mamba2
GDN
SDM
Δ vs GDN
single_1
64.2
53.8
99.9
100.0
+0.1
single_2
53.1
19.2
20.7
70.8
+50.1
single_3
41.1
8.0
12.1
46.3
+34.2
multikey_1
44.0
14.7
13.6
35.0
+21.4
multikey_2
25.3
1.0
0.7
0.8
+0.1
multikey_3
8.6
0.2
0.1
0.2
+0.1
multivalue
39.8
17.5
11.7
37.4
+25.7
multiquery
39.0
10.5
11.8
41.1
+29.3
vt
32.1
8.6
16.6
23.7
+7.1
cwe
5.8
3.6
6.6
8.7
+2.1
fwe
30.9
37.5
35.6
12.3
-23.2
qa_1
19.3
11.5
13.3
13.3
+0.0
qa_2
20.9
18.5
18.3
18.0
-0.3
Average
32.5
17.9
20.0
31.2
+11.2
Discussion
The memory access patterns and scaling results suggest that sparse, learned
memory access is a viable path for long-range sequence modeling. The consistent
slot assignment for storage and retrieval across hundreds of thousands of tokens
emerges entirely from the next-token prediction objective.
The finite state is not a limitation. It is a feature. Just as humans cannot
hold an entire book in working memory but can choose to re-read relevant passages,
a model with finite state can benefit from agentic context engineering: selectively
re-reading or searching through previous conversation to refresh its memory.
A fixed-size state makes this practical at any context length, whereas full
attention keeps a KV cache that grows with the sequence.
Open questions remain. SDM still trails full attention on some multi-key retrieval
tasks. The Triton kernels reach lower MFU than GDN because slot access is
memory-bound. And the interplay between state capacity and retrieval accuracy at
very long contexts deserves further study.
Summary
~1,000× larger recurrent state than GDN at iso-FLOP (state-to-parameter ratio near 1:1)
Predictable power-law scaling (R² = 0.999): SDM beats GDN at every scale and surpasses full attention at 8B
Much better perplexity scaling than GDN and Mamba2 beyond 128K
(at 1M tokens: SDM 2.01 vs. GDN 2.22, Δ=0.21; vs. Mamba2 2.29, Δ=0.28)
RULER (13-task average): 31.2% at 1.4B and 50.2% at 8B, far above the iso-FLOP GDN (20.0% and 34.2%)
