NASA used it to navigate to the moon. Quant funds use it to navigate markets. Here is exactly how it works and how to build it.
The problem was simple to state and hard to solve. You are tracking something that moves a spacecraft, a signal, a hidden state in a dynamic system. Your measurements of that thing are noisy. Every observation contains random error. How do you extract the true state from a stream of noisy measurements, in real time, using every new data point as it arrives?
NASA read the paper and immediately put it to work. The Kalman filter navigated the Apollo missions to the moon. It processed noisy radar readings, estimated the spacecraft's true position and velocity, and updated those estimates continuously as new sensor data arrived. No batch reprocessing. No historical lookback windows. Pure recursive updating every new observation made the estimate more precise.
Markets have exactly the same problem.
The true hedge ratio between two assets is hidden. The genuine trend beneath noisy price data is hidden. The relationship between two cointegrated assets evolves slowly over time and cannot be directly measured. What you observe is the noisy market price at each moment. The Kalman filter is the mathematically optimal tool for estimating hidden states from noisy observations and it updates continuously as new data arrives.
If you have not read the pairs trading article yet then you should read it, this one builds directly on top of it. The Kalman filter is what takes that framework from good to production grade. I would suggest reading it first to understand pairs trading as well.
The Portfolio Optimization Book, published by Cambridge University Press in 2025, reaches one explicit conclusion after testing multiple estimation approaches on real pairs data: "Kalman filtering is a must in pairs trading." Rolling OLS produces hedge ratio estimates varying between 0.6 and 1.2 on the same pairs where the Kalman filter stays stable between 0.55 and 0.65. That difference is not cosmetic. It determines whether the spread is stationary and tradeable or noisy and unreliable.
>NOTE: This article builds the complete framework. What the filter is actually doing, the full mathematics, and three concrete production-grade applications. Read it in order every Chapter connects to the next.
---
# Chapter 1: The Idea Before Any Equations
Before the equations, the idea. Because the idea is more important than the equations and most explanations get it backwards.
You want to know the true state of a system. But you cannot measure it directly. You can only observe measurements that are related to the true state and those measurements contain noise.
At every moment you have two sources of information:
Your model: what you predict the state should be based on how the system evolves. This prediction has uncertainty because the system changes in ways you cannot perfectly anticipate.
Your measurement: what your sensor tells you the state is right now. This also has uncertainty because measurements are noisy.
The question is: given both of these imperfect sources, what is the best possible estimate of the true state?
The Kalman filter answers this question optimally. It combines the prediction and the measurement, weighting each one by its relative reliability. When the measurement is noisy (high measurement variance R), trust the model prediction more. When the model is uncertain (high process variance Q), trust the new measurement more. The weight assigned to the new measurement the Kalman gain K updates automatically at every step based on the current uncertainty levels.
This is not an approximation. For linear systems with Gaussian noise it is the provably optimal estimator. No other method extracts more information from the available data.
In trading, the hidden states you care about are things like: the true hedge ratio between two assets right now, the genuine trend beneath noisy daily prices, the time-varying relationship between two correlated instruments. The measurements are the market prices you observe.
Generated by Thread Navigator
Press ⌘ + S to quick-export