MODEL GOVERNANCE · CASE STUDY · 24 JULY 2026
One factor model, one day, six covariance versions. What failed, how it was diagnosed, what was rebuilt — and what the successful test then revealed. Published because a standard that only ever produces clean results is not being tested.
Our institutional engine decomposes portfolio risk across seven factors — equity market,
rates duration, credit spread, FX, size, value and quality. Portfolio variance is
b′Σb + Σ(w²σ²_spec), where Σ is the factor covariance matrix.
Σ was not calibrated. It was a prior: variances set from published volatility estimates, correlations set by judgement. Our internal model documentation flagged this as the highest-priority open item and predicted that an independent validator would raise it first. That assessment was correct, and it had been outstanding for two months.
Starting from literature priors is normal when a client has no return history. Leaving them in place once history exists is not. The distinction that matters is whether the state is disclosed and scheduled — ours was documented internally, but it had no deadline.
We ingested three to five years of daily factor returns, computed an empirical covariance matrix, and backtested it: apply the model's 99% Value-at-Risk threshold to the historical return series and count how often losses exceed it. Over 504 days at 99% confidence, roughly five exceptions are expected. Kupiec's proportion-of-failures test then asks whether the observed count is statistically consistent with the stated confidence.
| Version | Estimator | Exceptions / 504d | Verdict |
|---|---|---|---|
| prior_v0 | literature prior | not tested | — |
| empirical_1254d | equal-weighted, 5y | 39 | FAIL |
| ewma_504d | EWMA λ=0.94 | 40 (LR 98.3, p ≈ 0) | FAIL |
Eight times the expected exception count. The model was understating risk by roughly 65% in volatility terms. Two observations narrowed the cause quickly. First, because the backtest uses the same exposure vector on both sides of the comparison, any uniform error in exposures or specific volatilities cancels — so the fault had to lie in Σ itself. Second, EWMA at λ=0.94 has an effective memory of about seventeen days; freezing one such snapshot and applying it across 504 days of history is an estimator/test mismatch, not a model failure.
Recalibrating on fresh data with the equal-weighted estimator passed — 8 exceptions, p ≈ 0.22. It also revealed how wrong the priors had been:
| Factor | Prior σ | Calibrated σ |
|---|---|---|
| rates duration | 6.0% | 15.8% |
| size | 10.0% | 22.5% |
| quality | 7.0% | 17.4% |
| value | 9.0% | 14.8% |
| equity market | 16.0% | 16.3% |
The equity prior was sound; the others were understated two- to three-fold. That accounted for the exceptions. But the calibrated matrix contained a second, larger problem.
The calibrated correlations were extreme: equity/quality 0.96, equity/value 0.91, equity/size 0.86, rates/credit 0.88. Principal component analysis of the correlation matrix showed three components explaining 90% of the structure, with a condition number of 131.
The cause was in the proxy definitions. Each factor was represented by a long-only index fund — small-cap, value, quality — and every equity index is roughly 90% market beta. The credit factor used an investment-grade corporate bond fund whose returns are dominated by its eight-year duration, which is why it tracked the Treasury factor at 0.88.
Portfolio-level volatility and VaR were unaffected — b′Σb is valid however
correlated the factors are. What collinearity destroys is attribution: when two factors
move together 96% of the time, the split of risk between them is arbitrary, and a single-factor
stress that moves one while holding the other fixed describes an impossible event.
We rebuilt the style and credit factors as long-short spreads — small minus large, value minus growth, quality minus market, and credit hedged against duration-matched Treasuries — so that each factor isolates its own effect rather than re-expressing the market.
| Diagnostic | Long-only proxies | Long-short spreads |
|---|---|---|
| Highest pairwise correlation | 0.962 | 0.635 |
| First component share | 58.7% | 30.6% |
| Components for 90% of variance | 3 of 7 | 6 of 7 |
| Condition number | 131 | 9.5 |
| Kupiec backtest | — | PASS · 7 exc. · p = 0.41 |
The correlations that survive are economically real rather than artefacts: equity/credit at +0.64 (spreads widen when equities fall) and equity/value at −0.48 (value lags growth-led rallies).
Between those states, the backtest returned zero exceptions — and reported a failure, correctly, because Kupiec rejects in both directions. The cause was that the return history had been replaced with the new spread definitions while the previously calibrated matrix was still active. The model and the data described different things, so the threshold sat far beyond any observed move.
The backtest read the live matrix and the stored history without checking that they came from the same factor definitions, and returned a confident verdict either way. A validator would flag this. The fix is provenance: each stored version carries the definitions it was built from, and the interface now reports the active version, its backtest verdict, and warns when a verdict predates the matrix it is attached to.
The second failure was procedural: portfolio-level risk metrics were not captured before the change was activated, so the before/after impact at portfolio level cannot be stated. The lesson is ordinary and worth writing down — measure what you are about to change, before you change it.
With a validated covariance matrix in place, the factor decomposition became readable — and was visibly wrong. One holding, the largest listed company in its market, decomposed to 70% value and 28% size, with 2.8% equity market. Its size exposure was positive against a small-minus-large factor; its value exposure was 2.25, where such loadings typically fall between −1 and +1. Several exposures were constants shared across every equity, and specific volatility was the system default for every position.
The explanation is straightforward: exposures were derived under the old long-only definitions. Redefining the factors changed what an exposure means, and nothing re-derived them.
The backtest compares b′Σb against returns computed as b·F, using the
same exposure vector on both sides. It therefore validates the covariance matrix
conditional on the exposures — it cannot validate the exposures themselves, which would
require realised portfolio returns. So: the covariance model is validated given the exposures;
the exposures are not yet validated; factor attribution is not identifiable until they are.
Portfolio-level risk and VaR are supported. Attribution is not, and is labelled accordingly.
Every risk model has a history of being wrong in specific, documented ways. Vendors rarely publish theirs, which leaves buyers to assume either that no such history exists or that it is being withheld. We prefer the third option: publish the sequence, including the two failures and the flaw the failures exposed, and let the work be judged on how it was handled. Every version described here remains in the archive and can be reactivated.