Introduction
Imagine you are watching a tennis ball mid flight, you don’t need to see every frame of the match to know where the match is heading at any given moment. Your brain stitches together a few observations, build an internal model, and projects forward. You’re not reacting, you’re anticipating.
This is exactly what we want sequence models to do. But the challenge has three parts:
- You never see the full state of the world
- The future is genuinely uncertain
- Context from past matters
No single classical architecture handles all three cleanly. Recurrent Neural Networks (RNNs) remember but can’t express uncertainty, Variational Autoencoders (VAEs) capture uncertainty but have no memory.
The Recurrent State Space Model (RSSM) is an elegant answer to all the three problems. By running two streams of state in parallel, one deterministic, one stochastic, it does what neither could do alone.
In this post, we’ll build up RSSM from first principles: why it was needed, how it works, what makes it tick.
The Sequence Modeling Problem
At its core, sequence modeling is about one thing: given what you’ve seen so far, what comes next ?
Simple in theory, hard in practice, there are three reasons why:
- The world is partially observable: You never get the full picture, only a noisy, incomplete observation at each timestep. A camera sees pixels, not velocities. A sensor reads pressure, not the full system size. The true underlying state is always hidden.
- The future is uncertain: Even with a perfect memory of the past multiple futures are plausible. A door that hasn’t opened in ten frames could open in the next one. A model that commits to a single prediction is overconfident by design.
- Context has a long reach. What happened twenty steps ago can matter enormously right now. A model that only looks at recent history will miss slow-building patterns and long-range dependencies.
These three challenges don’t just make sequence modeling harder. They make it fundamentally different from standard supervised learning. And they each demand something specific from the architecture you choose. RSSM was designed with all three in mind.
A Brief Lineage: From RNNs to SSMs to RSSM
State space models have a long history. The core idea that observations are generated by a hidden state evolving over time goes back decades.
Hidden Markov Models (HMMs) were among the first to formalize this. They modeled discrete latent states with probabilistic transitions, finding early success in speech recognition. But discrete states are limited to expressiveness, and inference doesn’t scale well.
Kalman Filters took a different route, continuous latent states with linear Gaussian dynamics