← Mingquan Liu
GMSWA · Gated-Memory Sliding-Window Attention

Long context with a constant cache.

GMSWA fuses exact sliding-window attention with a recurrent matrix memory. The result is a controlled long-context study: softmax-like base quality, real semantic recall, and a fixed inference cache — plus a clear warning about where hybrid memory still fails.

The problem Architecture Benchmarks Negative result Blog post
~790× smaller KV cache at 128K real recall comparable to recurrent baseline synthetic needle recall remains hard
KV cache @128K
16.3MB
vs 12.9GB full attention
Total memory @128K
~11×
smaller end-to-end inference memory
Zero-shot quality
0.499
on par with softmax baselines
Real recall
2/3
wins vs recurrent baseline on SQuAD + FDA

The paper-shaped claim

GMSWA is not sold as “the best recall model.” Its value is a clean map of the tradeoff: local attention preserves language quality, memory recovers semantic recall, and exact single-needle retrieval exposes the limit.

Basesoftmax-like
Recallreal tasks
Cacheconstant
Problem

The long-context tradeoff

A paper has to start with the pain point. Full attention has quality but its cache grows with context; local windows are cheap but blind beyond the window; recurrent memory is compact but can lose base language quality. GMSWA asks whether a hybrid can keep the best parts.

01

Full attention scales poorly

At 128K context, the full-attention KV cache reaches 12.9GB in this 340M setup. The memory wall, not parameter count, dominates deployment.

→
02

Sliding windows forget

SWA keeps the cache bounded, but information outside the window is gone. It is strong locally, but cannot recover dropped evidence.

→
03

Recurrent memory is not enough

Pure recurrent models keep constant state and recall well, but trail softmax attention on base language quality in the controlled comparison.

Research question: can a fixed-cache window–memory hybrid keep softmax-level base quality while recovering useful long-range recall?
Architecture

Exact local attention, backed by recurrent memory

The new diagram shows the whole layer: a local SWA branch, an evicted-token memory branch, and a learned gate that mixes local evidence with long-range state.

GMSWA architecture: sliding-window attention branch, recurrent memory branch, and learned gate
GMSWA layer. The local branch keeps exact recent context; the memory branch keeps a fixed-size recurrent state; the gate chooses how much each head trusts local evidence vs memory.constant-cache layer
The intuition

Three axes, one tradeoff map

The latest draft is stronger because it does not overclaim. It separates base modeling, real recall, and synthetic recall.

A
Base quality

Keep the softmax behavior

GMSWA stays with the softmax family on zero-shot quality and length-stable NLL, unlike a pure recurrent baseline.

R
Real recall

Recover semantic evidence

On SQuAD and FDA, GMSWA beats the recurrent baseline; on SWDE it trails. Honest read: recall parity, not dominance.

!
Synthetic needle

Sharp lookup is still hard

On NIAH, GMSWA tracks SWA and trails the recurrent model. This becomes the main design caution.

Takeaway: window–memory hybrids can be excellent constant-cache language models, but they need sharper addressing if exact needle retrieval is the target.
Results from latest draft

Quality, recall, and cache efficiency

These are the headline numbers now worth showing on the project page.

790×
smaller KV cache than full attention at 128K
16.3MB
constant GMSWA cache across context length
0.499
10-task zero-shot avg, above recurrent baseline
0.284
SQuAD recall, slightly above recurrent baseline

Base quality: zero-shot + length stability

GMSWA stays in the softmax cluster on standard zero-shot evaluation, and is the strongest length-stable model on token NLL buckets.

ModelZero-shot avg ↑NLL 1–2K ↓NLL 2–4K ↓NLL 4–8K ↓
Transformer0.5003.625.747.21
SWA0.4983.874.144.21
GMSWA0.4993.784.024.08
GDN0.4863.894.154.24
Reading: GMSWA preserves softmax-like base quality while avoiding the Transformer's length extrapolation cliff.

Efficiency: cache stays flat

The deployment win is constant cache. Full attention grows linearly; GMSWA remains fixed as context length increases.

ModelKV @8KKV @128KShape
Transformer805MB12.9GBlinear
SWA50MB50MBconstant
GDN3.4MB3.4MBconstant
GMSWA16.3MB16.3MBconstant
Reading: at 128K, GMSWA's KV cache is roughly 790× smaller than full attention.

Full zero-shot suite: task-by-task scores

The controlled suite in the current draft reports LAMBADA, PIQA, HellaSwag, WinoGrande, ARC, BoolQ, COPA, OpenBookQA, and SciQ. MMLU was not run in this controlled comparison yet, so I do not show a fabricated number.

ModelAvg ↑LAMBADAPIQAHellaSwagWinoGrandeARC-eARC-cBoolQCOPAOBQASciQ
Transformer0.5000.3170.6600.3300.5230.5690.2530.6150.7000.2060.822
SWA0.4980.3150.6580.3260.5100.5770.2410.6080.7000.2400.803
GMSWA0.4990.3240.6550.3300.5300.5660.2580.6110.6700.2020.849
GDN0.4860.2910.6560.3240.5090.5610.2410.5880.6500.2180.819
Reading: GMSWA is not winning every task, but it stays in the softmax band and beats the pure recurrent baseline on the average. This is exactly the point: memory does not cost base quality.

Real recall tasks

contains accuracy
SWA · SWDE
.072
GMSWA · SWDE
.088
GDN · SWDE
.110
SWA · FDA
.040
GMSWA · FDA
.094
GDN · FDA
.026
SWA · SQuAD
.072
GMSWA · SQuAD
.284
GDN · SQuAD
.274

Reading: GMSWA is comparable to the recurrent baseline on real recall and clearly above local-only SWA.

NIAH synthetic needle

the caution
1.00
.65
.29
.18
.08
.99
.57
.27
.16
.08
.88
.75
.57
.40
.17
1.00
.86
.74
.00
.00
5121K2K4K8K
SWA localGMSWA hybridGDN recurrentFull attention

Reading: GMSWA tracks SWA on synthetic single-needle retrieval; the recurrent baseline is better. This is the paper's honest negative result.

Negative result

Forcing memory did not solve sharp retrieval

The important analysis in the latest draft: five memory-directed controls all hurt or collapse recall. The issue is not simply “the model ignored memory.”

Output normalization
~0
best NLL, collapsed recall
Aggressive pathway dropout
0.00
destabilized training
Gentle pathway dropout
0.11
worse NIAH and NLL
Memory-first curriculum
~0
competent LM, no needle recall
Memory-only training
0.011
NIAH at 2K collapses
Design caution: this memory branch helps smooth semantic recall, but exact synthetic retrieval needs sharper addressing or a retrieval-explicit objective.
A practical constant-cache model — and a clean map of where hybrid recall stops.

GMSWA's strongest story is not overclaiming. It shows what a window–memory hybrid gives you, and exactly which kind of recall still requires a better memory interface.