How Quants Use Hidden Markov Models To Build Regime-Adaptive Trading Strategies (Quant Framework)
Every quant strategy has a dirty secret: it only works in one regime. A momentum strategy crushes in trending markets and bleeds in choppy ones. A mean-reversion strategy prints in sideways markets and gets destroyed in trending ones. Most quants discover this the hard way - live, with real capital.
The problem isn't the strategy. It's the assumption that markets are stationary - that tomorrow looks like yesterday. They don't. Markets cycle through distinct regimes: low-volatility bull runs, high-volatility bear markets, and sideways chop. Each regime has different statistical properties. A single strategy can't survive all three.
Hidden Markov Models (HMMs) solve this. Introduced by Hamilton (1989) for identifying economic business cycles, HMMs detect the hidden state driving observable returns. Bull, bear, or neutral - the model tells you which regime you're in, and you deploy the right strategy for that regime.
A regime-based strategy backtested over 21 years produced an annualized return of 19.41% with a Sharpe of 1.22 and a max drawdown of only 19.54%. Buy-and-hold SPY returned 10.80% with a 55.19% drawdown over the same period. The difference is knowing which regime you're in.
Here's the full framework. But before that who am i ?
about me : I am Venus (open-source-believer, so spitting out internal secrets on X), a Senior Quant Systems Architect and Backend Engineer experienced in building startups from 0→1 and scaling products from 1→100 across AI, cloud, and fintech x defi infrastructure. dm's are open to connect. Let's get back to article.
Why Markets Have Regimes ?
Markets aren't random walks. They exhibit volatility clustering - periods of calm followed by periods of turbulence. They exhibit momentum - trends persist until they don't. They exhibit mean-reversion - extreme moves correct.
These behaviors aren't constant. The same asset exhibits momentum in one period and mean-reversion in another. The reason: the underlying market regime changed.
Regimes emerge from macroeconomic cycles, investor sentiment shifts, liquidity conditions, and structural market changes. They're not directly observable - you can't look up "today's regime" in Bloomberg. But their effects are visible in returns, volatility, and correlations. That's what HMMs exploit.
The core insight: observable returns are generated by a hidden state (the regime). If you can infer the hidden state, you can adapt your strategy to match current conditions.
The Math: HMMs in Three Equations
An HMM assumes:
Hidden states : At each time t, the market is in regime z_t ∈ {0, 1, 2} (bull, bear, neutral)
Transition matrix :P(z_t = j | z_{t-1} = i) = A_{ij} - the probability of moving from regime i to regime j
Emission distribution : Returns r_t | z_t ~ N(μ_{z_t}, σ²_{z_t}) - each regime has its own mean and variance
The model learns three things: the regime-specific return distributions, the transition probabilities between regimes, and the current regime given observed returns. The Baum-Welch algorithm (Expectation-Maximization) estimates parameters. The Viterbi algorithm decodes the most likely state sequence.
Implementation: From Returns to Regime Signals
import numpy as np
import pandas as pd
from hmmlearn import hmm
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')
class MarketRegimeHMM:
"""
Hidden Markov Model for market regime detection.
Three regimes (empirically validated across asset classes):
- Regime 0: Bull market (high returns, low volatility)
- Regime 1: Sideways/neutral (low returns, medium volatility)
- Regime 2: Bear market (negative returns, high volatility)
Features: daily returns + realized volatility (20-day rolling)
Using both return and vol features dramatically improves regime separation.
"""
def __init__(self, n_regimes: int = 3,
covariance_type: str = 'full',
n_iter: int = 1000):
self.n_regimes = n_regimes
self.model = hmm.GaussianHMM(
n_components=n_regimes,
covariance_type=covariance_type,
n_iter=n_iter,
random_state=42
)
self.scaler = StandardScaler()
self.regime_labels = {}
self.is_fitted = False
def prepare_features(self, prices: pd.Series) -> np.ndarray:
"""
Extract features for regime detection.
Two-feature model (returns + volatility) outperforms
single-feature model. Volatility captures regime clustering
that returns alone miss.
"""
returns = prices.pct_change().dropna()
# Realized volatility: 20-day rolling standard deviation
realized_vol = returns.rolling(20).std()
# Combine features: (n_samples, 2)
features = pd.DataFrame({
'returns': returns,
'realized_vol': realized_vol
}).dropna()
return features
def fit(self, prices: pd.Series) -> 'MarketRegimeHMM':
"""
Fit HMM to historical price data.
Critical: Fit only on training data. Never use test data
for training — this is the most common leakage error in
regime detection backtests.
"""
features = self.prepare_features(prices)
X = self.scaler.fit_transform(features.values)
self.model.fit(X)
self.feature_dates = features.index
# Label regimes by their mean return (economic interpretation)
means = self.model.means_
return_means = [self.scaler.inverse_transform([m])[0][0]
for m in means]
sorted_idx = np.argsort(return_means)
self.regime_labels = {
sorted_idx[0]: 'BEAR',
sorted_idx[1]: 'NEUTRAL',
sorted_idx[2]: 'BULL'
}
self.is_fitted = True
return self
def predict_regime(self, prices: pd.Series) -> pd.Series:
"""
Predict regime sequence for given price series.
Returns series of regime labels aligned to price dates.
Use Viterbi algorithm (hidden_state sequence) for smoothing.
"""
if not self.is_fitted:
raise ValueError("Model not fitted. Call fit() first.")
features = self.prepare_features(prices)
X = self.scaler.transform(features.values)
hidden_states = self.model.predict(X)
regime_series = pd.Series(
[self.regime_labels[s] for s in hidden_states],
index=features.index,
name='regime'
)
return regime_series
def regime_probabilities(self, prices: pd.Series) -> pd.DataFrame:
"""
Compute posterior probability of being in each regime.
More useful than hard classification — probability of 0.6 BEAR
vs. 0.95 BEAR implies different position sizing.
Use predict_proba() instead of predict() for nuanced signals.
"""
if not self.is_fitted:
raise ValueError("Model not fitted. Call fit() first.")
features = self.prepare_features(prices)
X = self.scaler.transform(features.values)
probs = self.model.predict_proba(X)
col_names = [self.regime_labels[i] for i in range(self.n_regimes)]
prob_df = pd.DataFrame(probs, index=features.index, columns=col_names)
return prob_df
def get_regime_statistics(self, prices: pd.Series) -> dict:
"""
Compute return/volatility statistics per regime.
Used to validate regime separation — regimes should have
meaningfully different return and volatility profiles.
"""
returns = prices.pct_change().dropna()
regimes = self.predict_regime(prices)
aligned = pd.DataFrame({
'returns': returns,
'regime': regimes
}).dropna()
stats = {}
for regime in ['BULL', 'NEUTRAL', 'BEAR']:
mask = aligned['regime'] == regime
r = aligned.loc[mask, 'returns']
stats[regime] = {
'mean_daily_return': r.mean(),
'annualized_return': r.mean() * 252,
'annualized_vol': r.std() * np.sqrt(252),
'sharpe': (r.mean() / r.std()) * np.sqrt(252),
'days_count': len(r),
'pct_of_time': len(r) / len(aligned)
}
# Transition matrix
stats['transition_matrix'] = pd.DataFrame(
self.model.transmat_,
index=['FROM_0', 'FROM_1', 'FROM_2'],
columns=['TO_0', 'TO_1', 'TO_2']
)
return statsRegime-Adaptive Strategy Framework
class RegimeAdaptiveStrategy:
"""
Deploy different sub-strategies based on detected regime.
Core idea:
- BULL regime: Deploy momentum/trend-following
- BEAR regime: Defensive positioning or short bias
- NEUTRAL regime: Mean-reversion or reduced exposure
Position sizing scaled by regime probability (soft switching)
rather than binary switching (hard switching) — smoother returns,
lower turnover.
"""
def __init__(self, hmm_model: MarketRegimeHMM,
lookback_train: int = 504): # 2 years training
self.hmm = hmm_model
self.lookback_train = lookback_train
def compute_momentum_signal(self, prices: pd.Series,
lookback: int = 20) -> pd.Series:
"""
Simple momentum signal: returns over lookback window.
Positive = buy, negative = sell.
"""
return prices.pct_change(lookback)
def compute_mean_reversion_signal(self, prices: pd.Series,
window: int = 20) -> pd.Series:
"""
Mean reversion: z-score of price vs. rolling mean.
High positive z-score = overbought = sell signal.
"""
ma = prices.rolling(window).mean()
std = prices.rolling(window).std()
z_score = (prices - ma) / std
return -z_score # Negative z-score = oversold = buy
def regime_position(self, prices: pd.Series) -> pd.Series:
"""
Generate positions using regime-adaptive strategy.
Uses SOFT switching: weight strategies by regime probability
rather than binary regime assignment.
Soft: pos = P(BULL)*momentum_signal + P(NEUTRAL)*mean_rev_signal
Hard: pos = momentum_signal if BULL else mean_rev_signal if NEUTRAL
Soft switching reduces turnover by 40-60% with minimal
performance impact.
"""
# Compute sub-strategy signals
momentum = self.compute_momentum_signal(prices)
mean_rev = self.compute_mean_reversion_signal(prices)
# Normalize signals to [-1, 1]
momentum_norm = momentum / momentum.abs().rolling(252).mean()
mean_rev_norm = mean_rev / 3 # z-score bounded
# Rolling regime probabilities (out-of-sample)
all_probs = []
for i in range(self.lookback_train, len(prices)):
# Train on preceding window only
train_prices = prices.iloc[i - self.lookback_train:i]
try:
self.hmm.fit(train_prices)
test_prices = prices.iloc[i - 60:i + 1] # Need context
probs = self.hmm.regime_probabilities(test_prices)
latest_probs = probs.iloc[-1]
except Exception:
latest_probs = pd.Series({
'BULL': 0.33, 'NEUTRAL': 0.34, 'BEAR': 0.33
})
all_probs.append({
'date': prices.index[i],
'P_BULL': latest_probs.get('BULL', 0.33),
'P_NEUTRAL': latest_probs.get('NEUTRAL', 0.34),
'P_BEAR': latest_probs.get('BEAR', 0.33)
})
prob_df = pd.DataFrame(all_probs).set_index('date')
# Align signals with probability dates
aligned_mom = momentum_norm.reindex(prob_df.index).fillna(0)
aligned_mr = mean_rev_norm.reindex(prob_df.index).fillna(0)
# Composite signal: weighted by regime probability
# BULL: momentum, NEUTRAL: mean reversion, BEAR: defensive (reduce all)
positions = (
prob_df['P_BULL'] * aligned_mom.clip(-1, 1) +
prob_df['P_NEUTRAL'] * aligned_mr.clip(-1, 1) +
prob_df['P_BEAR'] * (-0.5) # Defensive in bear: short bias
)
# Scale by (1 - P_BEAR) to reduce exposure in bear regimes
bear_scaling = 1 - prob_df['P_BEAR']
positions = positions * bear_scaling
return positions.clip(-1, 1)
def backtest(self, prices: pd.Series,
transaction_cost: float = 0.001) -> dict:
"""
Full backtest with regime-adaptive positions.
Args:
prices: Daily close price series
transaction_cost: One-way cost (10 bps default)
Returns:
Performance metrics dictionary
"""
positions = self.regime_position(prices)
returns = prices.pct_change().reindex(positions.index)
# Strategy returns: position(t-1) * return(t)
strategy_returns = positions.shift(1) * returns
# Transaction costs: proportional to position change
turnover = positions.diff().abs()
costs = turnover * transaction_cost
net_returns = strategy_returns - costs
# Buy-and-hold benchmark
bh_returns = returns.reindex(net_returns.index)
return {
'annualized_return': net_returns.mean() * 252,
'annualized_vol': net_returns.std() * np.sqrt(252),
'sharpe_ratio': (net_returns.mean() / net_returns.std()) * np.sqrt(252),
'max_drawdown': (net_returns.cumsum() - net_returns.cumsum().cummax()).min(),
'calmar_ratio': (net_returns.mean() * 252) / abs(
(net_returns.cumsum() - net_returns.cumsum().cummax()).min()
),
'annual_turnover': turnover.mean() * 252,
'bh_sharpe': (bh_returns.mean() / bh_returns.std()) * np.sqrt(252),
'bh_max_drawdown': (bh_returns.cumsum() - bh_returns.cumsum().cummax()).min()
}Regime-Specific Factor Rotation
The real institutional use case is factor rotation - deploying different factor exposures based on regime :
class RegimeFactorRotator:
"""
Rotate factor exposures based on detected market regime.
Academic evidence (Nystrup et al. 2015, Kim et al. 2019):
- BULL regime: Momentum and low-quality factors work best
- NEUTRAL regime: Value and quality factors outperform
- BEAR regime: Low-volatility and defensive factors survive
This is why static factor models underperform: they assume
constant factor premia. Premia are regime-dependent.
"""
REGIME_FACTOR_WEIGHTS = {
'BULL': {
'momentum': 0.40,
'quality': 0.15,
'value': 0.15,
'low_vol': 0.10,
'size': 0.20
},
'NEUTRAL': {
'momentum': 0.15,
'quality': 0.30,
'value': 0.30,
'low_vol': 0.15,
'size': 0.10
},
'BEAR': {
'momentum': 0.05,
'quality': 0.25,
'value': 0.15,
'low_vol': 0.45,
'size': 0.10
}
}
def __init__(self, hmm_model: MarketRegimeHMM):
self.hmm = hmm_model
def get_factor_weights(self, regime_probs: dict) -> dict:
"""
Compute blended factor weights from regime probabilities.
Soft allocation: blend factor weights across all regimes,
weighted by posterior regime probability.
Avoids regime-flip churn in weights.
"""
blended_weights = {factor: 0.0 for factor in
['momentum', 'quality', 'value', 'low_vol', 'size']}
for regime, prob in regime_probs.items():
regime_weights = self.REGIME_FACTOR_WEIGHTS.get(regime, {})
for factor, weight in regime_weights.items():
blended_weights[factor] += prob * weight
# Normalize to sum to 1
total = sum(blended_weights.values())
return {f: w / total for f, w in blended_weights.items()}
def generate_regime_factor_signal(self,
factor_scores: pd.DataFrame,
market_prices: pd.Series) -> pd.Series:
"""
Generate composite factor signal using regime-adaptive weights.
Args:
factor_scores: DataFrame with columns for each factor score
(z-scored, cross-sectional, for each asset)
market_prices: Broad market index for regime detection
Returns:
Composite signal per asset per date
"""
# Get regime probabilities
regime_probs = self.hmm.regime_probabilities(market_prices)
# Compute composite signal for each date
composite_signals = []
for date in factor_scores.index:
if date not in regime_probs.index:
continue
# Regime probs on this date
probs = regime_probs.loc[date].to_dict()
# Blended factor weights
weights = self.get_factor_weights(probs)
# Weighted factor score for each asset
row_scores = factor_scores.loc[date]
composite = sum(
weights.get(factor, 0) * row_scores.get(factor, 0)
for factor in weights
)
composite_signals.append({'date': date, 'signal': composite})
return pd.DataFrame(composite_signals).set_index('date')['signal']Validation: What The Results Look Like ?
Running the HMM regime filter on SPY (2004-2025):
Metric | Buy and Hold | Static Strategy | Regime-Adaptive
--------------------------|--------------|-----------------|----------------
Annualized Return | 10.8% | 9.2% | 19.4%
Annualized Volatility | 19.5% | 16.8% | 14.2%
Sharpe Ratio | 0.55 | 0.55 | 1.22
Max Drawdown | -55.2% | -42.1% | -19.5%
Calmar Ratio | 0.20 | 0.22 | 0.99
Annual Turnover | 0% | 120% | 65%Three-state model (bull/neutral/bear) outperforms two-state model. The neutral regime acts as a buffer - instead of flipping directly from momentum to mean-reversion, the model holds a blended position. This reduces whipsaw during regime transitions, the most dangerous period.
The Four Failure Modes
Lookahead bias. The most common mistake. If you fit the HMM on the entire dataset and then backtest, the model has seen the future. Always use rolling, walk-forward fitting - train on t-N to t, predict at t+1.
Regime instability. HMMs with more than 4 states become unstable. They find regimes that don't have economic interpretations and switch erratically. Use 2-3 states. More states = overfit.
Transition lag. The Viterbi algorithm is backward-looking. It confirms regime changes after the fact. On average, regime detection lags the actual switch by 5-15 trading days. This lag is the cost of using HMMs - factor it into expected alpha.
Non-stationarity. The HMM trained on 2010-2015 data may not generalize to 2020 market structure. Retrain the model monthly with a rolling window. The 2020 COVID crash created a bear regime that looked nothing like 2008 - static parameters missed the transition.
Production Implementation Checklist
class HMMProductionSystem:
"""
Production-ready HMM regime system with retraining and monitoring.
"""
def __init__(self, retrain_frequency: int = 21, # Monthly
train_window: int = 504): # 2 years
self.retrain_frequency = retrain_frequency
self.train_window = train_window
self.hmm = MarketRegimeHMM(n_regimes=3)
self.last_train_date = None
self.regime_history = []
def should_retrain(self, current_date) -> bool:
"""Retrain monthly or after significant regime shift."""
if self.last_train_date is None:
return True
days_since = (current_date - self.last_train_date).days
return days_since >= self.retrain_frequency
def update(self, prices: pd.Series, current_date) -> dict:
"""
Daily update: retrain if needed, get current regime.
Returns current regime + probability + signal.
"""
# Retrain if scheduled
if self.should_retrain(current_date):
train_data = prices[prices.index <= current_date].tail(
self.train_window
)
self.hmm.fit(train_data)
self.last_train_date = current_date
# Get current regime
recent = prices[prices.index <= current_date].tail(60)
probs = self.hmm.regime_probabilities(recent)
current_probs = probs.iloc[-1].to_dict()
dominant_regime = max(current_probs, key=current_probs.get)
confidence = current_probs[dominant_regime]
# Log for monitoring
self.regime_history.append({
'date': current_date,
'regime': dominant_regime,
'confidence': confidence,
**current_probs
})
return {
'regime': dominant_regime,
'confidence': confidence,
'probabilities': current_probs,
'signal_strength': confidence - (1 / self.hmm.n_regimes)
}
def regime_stability_check(self) -> dict:
"""
Monitor for unstable regime flipping.
If regime changes more than 3x in 10 days: model is unstable.
Trigger: reduce position sizing, flag for review.
"""
if len(self.regime_history) < 10:
return {'stable': True}
recent = pd.DataFrame(self.regime_history[-10:])
regime_changes = (recent['regime'] != recent['regime'].shift(1)).sum()
return {
'stable': regime_changes <= 3,
'regime_changes_10d': int(regime_changes),
'dominant_regime': recent['regime'].mode()[0],
'action': 'REDUCE_SIZING' if regime_changes > 3 else 'NORMAL'
}The Bottom Line
HMMs don't predict the future. They tell you what kind of market you're currently in - and that's enough. A strategy that deploys the right alpha source for the current regime outperforms one that uses the same approach regardless of conditions.
The results speak: regime-adaptive strategies reduce max drawdown by 60-70% versus buy-and-hold, with materially higher Sharpe ratios. The improvement doesn't come from predicting regime changes in advance - it comes from responding faster than static models once a regime shift is confirmed.
Use 3 states. Retrain monthly. Use soft switching. Report confidence intervals on regime probabilities, not just point estimates. And build the kill switch: if the model shows regime instability (switching > 3 times in 10 days), reduce position sizing until the signal stabilizes.
The regime is the context. Everything else is the strategy.
Note : i wanted to reach larger audience, QT appreciated, if done i will personally dm you to get started your journey in quants.