Jekyll2026-09-09T08:48:55+00:00https://wxqnl.github.io/feed.xmlMingquan Liupersonal descriptionMingquan Liuwxqnl@cug.edu.cnFuture Blog Post2199-01-01T00:00:00+00:002199-01-01T00:00:00+00:00https://wxqnl.github.io/posts/2012/08/future-postThis post will show up by default. To disable scheduling of future posts, edit config.yml and set future: false.

]]>Mingquan Liuwxqnl@cug.edu.cnGMSWA: Constant-Cache Long-Context Memory2026-06-25T00:00:00+00:002026-06-25T00:00:00+00:00https://wxqnl.github.io/posts/2026/06/gmswa-state-trackingGMSWA (Gated-Memory Sliding-Window Attention) studies a practical long-context question:

Can a model keep the language quality of local/softmax attention, recover useful long-range recall, and avoid a growing KV cache?

The problem is the core of the story: full attention has strong quality but its KV cache grows linearly with context; sliding-window attention keeps cache bounded but forgets beyond the window; recurrent memory is compact and recall-friendly but can trail softmax attention on base language quality. GMSWA tests whether a window-memory hybrid can keep the best parts.

The layer runs two branches: exact sliding-window attention for recent local evidence, and a gated-delta recurrent matrix memory for long-range information. A learned per-head gate mixes the two.

GMSWA architecture

The latest paper draft frames the contribution as a controlled characterization rather than an overclaim. The headline findings are:

  • Base quality is preserved. GMSWA is on par with softmax baselines on a 10-task zero-shot suite and above a pure recurrent baseline.
  • Real semantic recall improves over local-only attention. On real recall-intensive tasks, GMSWA is comparable to the recurrent baseline: it wins on SQuAD and FDA, while trailing on SWDE.
  • The cache is constant. At 128K context, GMSWA’s KV cache is about 16.3MB, versus about 12.9GB for full attention — roughly 790× smaller cache.
  • Synthetic needle retrieval is the limitation. On NIAH, GMSWA tracks local-window attention and trails the pure recurrent model.
  • The negative result is informative. Memory-directed controls — normalization, pathway dropout, memory-first curriculum, and memory-only training — all hurt or collapse recall, suggesting the issue is limited sharp addressing rather than simply “the model ignored memory.”

Core benchmark snapshot:

Model Zero-shot avg ↑ NLL 1–2K ↓ NLL 2–4K ↓ NLL 4–8K ↓
Transformer 0.500 3.62 5.74 7.21
SWA 0.498 3.87 4.14 4.21
GMSWA 0.499 3.78 4.02 4.08
GDN 0.486 3.89 4.15 4.24

Detailed zero-shot suite:

Model Avg ↑ LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c BoolQ COPA OBQA SciQ
Transformer 0.500 0.317 0.660 0.330 0.523 0.569 0.253 0.615 0.700 0.206 0.822
SWA 0.498 0.315 0.658 0.326 0.510 0.577 0.241 0.608 0.700 0.240 0.803
GMSWA 0.499 0.324 0.655 0.330 0.530 0.566 0.258 0.611 0.670 0.202 0.849
GDN 0.486 0.291 0.656 0.324 0.509 0.561 0.241 0.588 0.650 0.218 0.819

MMLU is not shown because it was not part of this controlled run’s completed result set; the available standard suite includes HellaSwag, PIQA, ARC, BoolQ, COPA, OpenBookQA, SciQ, WinoGrande, and LAMBADA.

Model SWDE ↑ FDA ↑ SQuAD ↑ KV cache @128K
SWA 0.072 0.040 0.072 50MB
GMSWA 0.088 0.094 0.284 16.3MB
GDN 0.110 0.026 0.274 3.4MB
Transformer 0.438 0.156 0.066 12.9GB

The short version: GMSWA is a practical constant-cache long-context model and a clean map of where window–memory hybrid recall works — and where it stops.

View the visual project showcase →

]]>Mingquan Liuwxqnl@cug.edu.cnWM3D: Native-3D World Modeling for Robot Action2026-06-08T00:00:00+00:002026-06-08T00:00:00+00:00https://wxqnl.github.io/posts/2026/06/wm3d-native-3d-world-modelMost world models predict the future in pixel or video-latent space. WM3D takes a different route: for robot manipulation, the state that matters is the 3D world — depth, point geometry, camera/robot pose, task text, and the action that changes the scene.

Can a robot world model keep native 3D as its core state space, while still producing action-aware, high-quality RGB rollouts?

That is the current WM3D story. Pixels are useful as observations and demos, but they are not the physical state a robot acts on. A plausible video can still hide the wrong depth, contact, or object motion. WM3D therefore trains action-conditioned dynamics directly over native-3D supervision, and treats RGB video as a rendered view of the imagined state.

WM3D architecture comparison

Why this is different

Approach Main state Strength Limitation for robotics
Video-latent world models future frames / video tokens strong visual prior and photorealism action→geometry causality is implicit
Compact dynamics / policy models low-dimensional control state efficient and stable for policies can lose rich visuospatial structure
WM3D native 3D + action-conditioned dynamics explicit geometry, depth, pose, progress, and policy interfaces RGB must be rendered without taking over the state

The latest architecture makes this boundary explicit. WM3D owns the world state and action reasoning. A stronger action-aware video branch renders RGB, but the native-3D core remains central: depth, point/geometry, pose, action, proposer, progress, and policy objectives are still trained as first-class targets.

Benchmark snapshot

Open-loop future prediction on a balanced OXE/DROID protocol shows WM3D beating a last-frame baseline across the main RGB metrics:

Metric WM3D Last-frame Delta
PSNR ↑ 28.374 23.285 +21.9%
SSIM ↑ 0.9135 0.8744 +4.5%
LPIPS ↓ 0.0985 0.1199 −17.9%
RGB L1 ↓ 0.0199 0.0321 −38.1%
Motion RGB L1 ↓ 0.0648 0.1409 −54.0%
R3D18-FVD ↓ 22.037 28.157 better

The more important evidence is native-3D action sensitivity:

Metric Value Reading
Depth future L1 ↓ 0.02659 future depth is directly supervised
Depth-change cosine ↑ 0.6735 captures how geometry changes
Depth-change sign accuracy ↑ 0.8193 predicts the direction of depth change
Real-action token win-rate ↑ 0.8650 real actions beat counterfactual alternatives
Real-action depth win-rate ↑ 0.7825 action affects geometry, not just texture
Motion-region depth win-rate ↑ 0.7708 the effect remains in moving regions

For downstream policy evaluation, the current best LIBERO spatial policy chain reaches 384/500 = 76.8% over 500 episodes. I keep this separate from reconstruction metrics because lower offline loss did not always produce higher rollout success.

Current takeaway

WM3D is not just a video demo. It is a state-space bet:

  • Predict the future in native 3D.
  • Treat actions as interventions on geometry.
  • Render RGB as a view of the imagined world, not as the world state itself.
  • Use the same state for prediction, counterfactual testing, progress scoring, action proposal, and policy evaluation.

View the visual project showcase →

]]>Mingquan Liuwxqnl@cug.edu.cnBlog Post number 42015-08-14T00:00:00+00:002015-08-14T00:00:00+00:00https://wxqnl.github.io/posts/2012/08/blog-post-4This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.

Headings are cool

You can have many headings

Aren’t headings cool?

]]>Mingquan Liuwxqnl@cug.edu.cnBlog Post number 32014-08-14T00:00:00+00:002014-08-14T00:00:00+00:00https://wxqnl.github.io/posts/2014/08/blog-post-3This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.

Headings are cool

You can have many headings

Aren’t headings cool?

]]>Mingquan Liuwxqnl@cug.edu.cnBlog Post number 22013-08-14T00:00:00+00:002013-08-14T00:00:00+00:00https://wxqnl.github.io/posts/2013/08/blog-post-2This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.

Headings are cool

You can have many headings

Aren’t headings cool?

]]>Mingquan Liuwxqnl@cug.edu.cnBlog Post number 12012-08-14T00:00:00+00:002012-08-14T00:00:00+00:00https://wxqnl.github.io/posts/2012/08/blog-post-1This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.

Headings are cool

You can have many headings

Aren’t headings cool?

]]>Mingquan Liuwxqnl@cug.edu.cn