Chapter 09 of 10
Adversarial validation
A scorecard that tries to break the fit: placebo channels, perturbed inputs, and a holdout graded against a benchmark that cannot see the future.
In-sample fit is necessary and nowhere near sufficient: a model with enough knots fits anything. So the scorecard is built to attack the fit rather than to flatter it. The checks that matter most are the three that refit the model on deliberately corrupted inputs and ask whether the answer survives.
The adversarial checks
Placebo.The largest channel’s spend is shuffled through time and the model refitted. The timing relationship is destroyed while the spend distribution is preserved exactly, so any effect the model still finds is an artifact of the design rather than of the data.
The spec asked that the shuffled channel’s ROI interval cover zero. That is impossible under this parameterization — chapter 03 — because a LogNormal has no mass at zero. Rather than quietly dropping the check or declaring it passed, it is restated as the same claim in a representable form: the shuffled channel’s 95th percentile ROI must fall below the real fit’s median.
Perturbation. Spend is jittered with 5% lognormal noise and the model refitted. The naive version of this test fails on every dataset, because refitting with a different seed alone moves ROI by around 6% — so a 5% threshold measures sampler noise, not sensitivity to the data. The check therefore runs a reseed control first and reports the excess movement above that floor.
excess = movement(5% spend noise) - movement(reseed only)
Refresh stability. The final period is masked and the model refitted. Masking rather than truncating keeps the spline basis and the scalers identical, so the test isolates the effect of the data rather than of a changed parameterization. A model whose answer swings on one week will contradict itself at every monthly refresh, and the client will notice.
source · engine/mmm/validate.py
Grading a holdout honestly
Out-of-sample R² is the usual holdout metric and it cheats: R² is computed against the holdout window’s own mean, which is not knowable at forecast time. The scorecard uses MASE instead, scaling absolute error against a seasonal-naive benchmark built from training data alone. A MASE of 1.0 means the model matched a forecast that requires no model at all.
| Check | Measures | Trips at | Reading |
|---|---|---|---|
| Holdout MAPE | Absolute percentage error on held-out cells | > 15% · > 30% | Accuracy in the units a stakeholder will ask about. |
| Holdout MASE | Error scaled by a seasonal-naive benchmark | > 1.0 · > 1.5 | Above 1.0 the model is losing to a forecast that uses no model. |
| Posterior predictive | Bayesian p-value on a chi-square discrepancy | outside 0.05–0.95 | Low means residuals exceed the estimated noise — a missing driver. High means the noise term is larger than the residuals, so the model is not being held accountable by the data. |
| Baseline sign | P(baseline < 0) | > 0.2 · > 0.5 | The classic symptom of over-attribution: media has been credited with more than the business would sell with no marketing at all. |
| Rolling origin | MAPE and bias per fold, one quarter ahead | reported, not scored | No accuracy threshold is right for every KPI, and inventing one makes the scorecard fail for reasons unrelated to the model. |
Did the data teach the model anything?
The failure mode chapter 04 describes — the model returning the prior with a credible interval — has to be detected after the fit as well as predicted before it. A channel is flagged as uninformed only when two independent signals agree: a Kolmogorov–Smirnov test finds no distributional shift from prior to posterior and posterior variance has contracted by less than 20%.
contraction = 1 - (posterior_sd ** 2 / prior_sd ** 2)
A separate check looks for the opposite problem: a posterior mean outside the prior’s 1st-to-99th percentile range means either the prior was wrong or the data are being over-read, and the fit is a compromise between two claims that disagree.
Coverage on one dataset is not calibration