The model worked.
The strategy did not. A Hidden Markov Model regime overlay for tactical asset allocation, run leak-proof over eight years of Indian and US markets.

What was built

Three hidden states inferred from causal features, each state routed to its own convex program. The interesting engineering is not the model, it is the two places the pipeline refuses to look at the future.

The stance map

Each regime gets a distinct convex program. Long-only, fully invested, weight-capped.

The states are not noise. The transition matrix diagonal runs 0.97 to 0.98, so the regimes persist rather than flickering. The corner entries are zero: Bull never jumps straight to Crisis and Crisis never jumps straight to Bull, so the market always passes through the middle state. Nothing in the fit asked for that.

The first result, and the whole problem

Measured at the lag the strategy actually trades: the label is known at the close of day t, the return is earned on day t plus one.

Next-day annualised return and volatility by regime label, both universes.

Volatility orders perfectly with the label. Return orders backwards, on both universes. That single table explains the entire result. De-risking on the Crisis label means selling the highest-returning days, because the rebounds of April 2020 and late 2022 are as violent as the crashes that preceded them.

A state variable has to predict direction before a directional bet on it can pay. Realised volatility is symmetric in sign by construction, so it cannot tell a crash from a rebound. The model was asked for regimes and it delivered regimes. They are the wrong kind.

Four rescues, each judged against a criterion written down first

The obvious response to a broken result is to fix it. Every attempt was given a success criterion before it was run, so that the outcome could not be reinterpreted afterwards.

The comparison was wrong too

An earlier draft compared strategies by checking whether their separate confidence intervals overlapped. That is not a comparison, and correcting it changed what could honestly be claimed.

Each marginal interval asks whether one book's Sharpe differs from zero. The question that matters is whether the gap between two books differs from zero, which needs the interval of the difference, resampled on the same dates for both books. Since every book holds the same assets on the same days, differencing cancels the shared market move and the paired interval comes out roughly three times tighter.

India. Sharpe difference against each benchmark, with its paired bootstrap interval.

Every interval spans zero, on both universes. The "it is all noise" reading survives, and it is now earned rather than assumed. It also sharpens one claim that used to be overstated: on the US the volatility rule's marginal interval excludes zero, which says only that its Sharpe beats zero. Paired against 60/40 it is +0.268 (-0.029, +0.559), so even the ablation that wins the US table cannot be said to beat the benchmarks.

What held up

A negative result is only worth reading if the machinery that produced it is trustworthy. These are the checks that survived their own scrutiny.

heldLeak-proofing, asserted by tests

Lookahead bias is the easiest way to produce a beautiful, worthless backtest. Five defences, each pinned by a unit test that fails if the defence is removed: causal right-aligned features, standardisation fit on training rows alone, a per-fold model refit, causal forward-filter decoding instead of whole-sequence Viterbi, and a one-day execution lag enforced in a single shared code path so every strategy and benchmark inherits it.

heldDeflation at an honest trial count

A Sharpe reported once, from one strategy out of several tried, is a biased number. The deflated Sharpe here charges for 7 trials, stated openly. The volatility rule read 0.928 at 4 trials, and searching three more ideas lowered every number in that column. That is what deflation is for. Intervals use a Politis-Romano stationary bootstrap with the block length read off the data rather than guessed.

heldSensitivity reported as a surface, adopting nothing

The sweep reports the whole surface and adopts nothing, because picking the best cell would be a search and would owe the deflated Sharpe another trial. Two knobs, weight_cap and rebalance_confirm_days, are not flat, and the shipped defaults sit at local Sharpe optima. Both were fixed before any result was computed and neither is re-chosen, but a reader is entitled to be told.

heldVendor data guarded, not trusted

GOLDBEES.NS on Yahoo prints a 100x round trip over 2019-12-19 to 2019-12-23: log returns of -4.61 then +4.61. Two bad prints in 2,193 rows. Left alone they inflate gold's return standard deviation from 0.011 to 0.139 and poison every Indian covariance, regime fit and Sharpe. The guard rejects any daily absolute log return above 0.5 with a loud warning, and a test pins that a genuine -13% crash day still survives it.

heldA no-model ablation that can beat the model

The strongest control here is a two-line volatility-threshold rule: same optimiser, same costs, same walk-forward, but the regimes come from a trailing-volatility quantile instead of the HMM. If the HMM cannot beat that, the HMM is decoration. On the US the rule wins outright, 0.958 against 0.542.

The scorecards

Every book, strategy and benchmark alike, runs through the same cost engine at 7.5 basis points per unit of turnover with a one-day execution lag. A benchmark costed on different terms is not a benchmark.

India, the primary universe. Out-of-sample 2016-07-22 to 2023-12-29, n = 1,814. Sharpe is measured against the 3.79% risk-free rate the cash sleeve actually paid, because scoring against zero would hand every defensive book a free Sharpe for holding cash. Benchmark rows are set in italic.

Read the Sharpe column and the strategy is unremarkable. The overlay's point estimate sits above both benchmarks, and entry 05 is the reason that means nothing: no book here is statistically distinguishable from either one. Read max drawdown and Calmar, the other two metrics the brief names, and the picture inverts: -6.2% worst drawdown against -15.2% and -23.7%, with Calmar roughly 1.9x and 2.8x the benchmarks. That gap is not noise, it is the mechanical consequence of routing to minimum variance whenever the label is not calm. The overlay buys drawdown protection and pays for it in return. Whether that trade is worth making is a mandate question, not a statistical one.

US, the robustness universe. Out-of-sample 2016-07-05 to 2023-12-29, n = 1,886, risk-free rate zero, with a real bond sleeve. Benchmark rows are set in italic.

The US does not reproduce India, and saying so is the point of running it. No HMM book improves on either benchmark on Sharpe or Calmar, and the drawdown protection that vindicated the overlay on India does not reappear: the best HMM drawdown here is level with 60/40 and well behind equal weight at -23.0%. Minimum variance had nowhere to hide in 2022, when bonds fell alongside equities. A result that held on one market and was quietly assumed to hold on the other would be the more comfortable story. It is not the one the data tells.

One more thing the sweep decided rather than assumed. Three regimes were chosen to match the economic states being allocated against, then checked with a BIC sweep. It supports three on the primary Indian universe, where the step from three states to four buys only 392 of fit against 6,256 for the step from two to three. It does not support three on the US. Both are reported.

The verdict

Hidden Markov states on returns and volatility are volatility states, and volatility carries no directional information. That is the finding, and it outlives this codebase.

Realised volatility and VIX are symmetric in sign by construction. Any allocator built on them alone will de-risk into rebounds as reliably as into crashes, because it cannot tell the two apart. Finding a directional state needs a signed state variable: credit spreads, market breadth, earnings revisions, positioning, realised skew. Not another estimator on the same features.

The one thing the overlay does buy is drawdown, and only on one of the two universes tested. Two of the three metrics the brief names favour it there. That is reported as exactly what it is, on one universe, rather than promoted into a headline.